fix(index): accept a single file as a source path - #130
Open
Sylvere-bamenou wants to merge 1 commit into
Open
Conversation
`LocalSource` walked the path with `rglob("*")` and `PDFSource` with
`glob("**/*.pdf")`. Both return nothing when the path points at a file
rather than a directory, so a single-document source silently yielded
zero documents and the run failed four stages later in `build-index`
with "No shard files found!".
This is the form the README documents:
source:
type: local
path: ./paper.pdf
Branch on `is_file()` in both adapters so a path pointing straight at a
document is a single-item source. Directory traversal is unchanged, and
the single-file path keeps the same extension filtering as the directory
one.
Also note the `pdf` extra and poppler in the PDF quickstart — the
documented `pixelrag[index]` alone cannot render a PDF.
Verified on macOS (Apple Silicon, MPS): `pixelrag index build` against
assets/pixelrag-paper.pdf now builds a 35-vector index from either a file
or a directory path; before, the file form produced zero documents.
|
@Sylvere-bamenou is attempting to deploy a commit to the andylizf's projects Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Following the README's own PDF quickstart end to end fails:
LocalSourcediscovers files withrglob("*")andPDFSourcewithglob("**/*.pdf"). Both return nothing when the path points at a file rather than a directory, so the source yields zero documents. Nothing errors at that point — the run continues through render, chunk and embed as no-ops and dies four stages later:The failure message points at the index builder, several stages away from the actual cause.
Reproducer against the repo's own asset:
Fix
Branch on
is_file()in both adapters, so a path pointing straight at a document is a single-item source. Directory traversal is untouched, and the single-file path keeps the same extension filtering the directory path applies.Also
The PDF quickstart installs
pixelrag[index], which cannot render a PDF —pdf2imagelives in thepdfextra and needs poppler. Added to the install line and a short note.Tests
Four tests in
tests/test_source_single_file.py, including one pinning the existing directory-traversal behaviour (recursive, both adapters) so this stays a widening and not a change.Verification
On macOS (Apple Silicon, MPS),
pixelrag index buildagainstassets/pixelrag-paper.pdf:assets/(directory)assets/pixelrag-paper.pdf(file)Both indexes serve and return the expected tile — querying "Overview of PixelRAG and the diagram" returns page 2 (the Figure 1 overview), matching the README's stated expectation.
pytest tests/→ 86 passed, 6 skipped.ruff checkandruff format --checkclean.Note for reviewers
LocalSourcealso feeds theweb/kiwixpaths downstream. The existing suite only covered directory inputs, so I could not verify against a usage that relies on a file path yielding zero documents — worth a look if any exists.