Skip to content

fix(index): accept a single file as a source path - #130

Open
Sylvere-bamenou wants to merge 1 commit into
StarTrail-org:mainfrom
Sylvere-bamenou:fix/source-single-file-path
Open

fix(index): accept a single file as a source path#130
Sylvere-bamenou wants to merge 1 commit into
StarTrail-org:mainfrom
Sylvere-bamenou:fix/source-single-file-path

Conversation

@Sylvere-bamenou

Copy link
Copy Markdown

Problem

Following the README's own PDF quickstart end to end fails:

source:
  type: local
  path: ./paper.pdf

LocalSource discovers files with rglob("*") and PDFSource with glob("**/*.pdf"). Both return nothing when the path points at a file rather than a directory, so the source yields zero documents. Nothing errors at that point — the run continues through render, chunk and embed as no-ops and dies four stages later:

Stage 1/4: Rendering 0 documents to tiles...
Stage 4/4: Building FAISS index (0 vectors, nlist=1)...
No shard files found!
subprocess.CalledProcessError: ... returned non-zero exit status 1

The failure message points at the index builder, several stages away from the actual cause.

Reproducer against the repo's own asset:

>>> len(PDFSource("assets/pixelrag-paper.pdf")._files)
0
>>> len(PDFSource("assets")._files)
1

Fix

Branch on is_file() in both adapters, so a path pointing straight at a document is a single-item source. Directory traversal is untouched, and the single-file path keeps the same extension filtering the directory path applies.

Also

The PDF quickstart installs pixelrag[index], which cannot render a PDF — pdf2image lives in the pdf extra and needs poppler. Added to the install line and a short note.

Tests

Four tests in tests/test_source_single_file.py, including one pinning the existing directory-traversal behaviour (recursive, both adapters) so this stays a widening and not a change.

Verification

On macOS (Apple Silicon, MPS), pixelrag index build against assets/pixelrag-paper.pdf:

Source path Before After
assets/ (directory) 35 vectors 35 vectors
assets/pixelrag-paper.pdf (file) 0 documents, crash 35 vectors

Both indexes serve and return the expected tile — querying "Overview of PixelRAG and the diagram" returns page 2 (the Figure 1 overview), matching the README's stated expectation.

pytest tests/ → 86 passed, 6 skipped. ruff check and ruff format --check clean.

Note for reviewers

LocalSource also feeds the web/kiwix paths downstream. The existing suite only covered directory inputs, so I could not verify against a usage that relies on a file path yielding zero documents — worth a look if any exists.

`LocalSource` walked the path with `rglob("*")` and `PDFSource` with
`glob("**/*.pdf")`. Both return nothing when the path points at a file
rather than a directory, so a single-document source silently yielded
zero documents and the run failed four stages later in `build-index`
with "No shard files found!".

This is the form the README documents:

    source:
      type: local
      path: ./paper.pdf

Branch on `is_file()` in both adapters so a path pointing straight at a
document is a single-item source. Directory traversal is unchanged, and
the single-file path keeps the same extension filtering as the directory
one.

Also note the `pdf` extra and poppler in the PDF quickstart — the
documented `pixelrag[index]` alone cannot render a PDF.

Verified on macOS (Apple Silicon, MPS): `pixelrag index build` against
assets/pixelrag-paper.pdf now builds a 35-vector index from either a file
or a directory path; before, the file form produced zero documents.
@vercel

vercel Bot commented Jul 30, 2026

Copy link
Copy Markdown

@Sylvere-bamenou is attempting to deploy a commit to the andylizf's projects Team on Vercel.

A member of the Team first needs to authorize it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant