Add turbovec-backed retrieval for context pulling and semantic search#38
Draft
slavazeph-coder wants to merge 2 commits into
Draft
Add turbovec-backed retrieval for context pulling and semantic search#38slavazeph-coder wants to merge 2 commits into
slavazeph-coder wants to merge 2 commits into
Conversation
Crumb LLM's spectral signatures are vectors, so they can be indexed in turbovec (a local TurboQuant vector store) instead of scanned linearly. - crumb_wavelm/turbovec_index.py: thin IdMapIndex wrapper. L2-normalizes signatures (inner product -> cosine) and zero-pads to turbovec's multiple-of-8 dim requirement (lossless for cosine). numpy-only deps, graceful absence handling. - context_pull.py: CrumbIndex.ensure_backend() lazily builds the vector index; score_sections uses it when present (over-fetching candidates so the priority boost can still re-rank), else the pure-Python cosine scan unchanged. - crumb llm index now reports the active backend. - crumb search --method semantic: spectral vector search over .crumb sections, backed by turbovec when available, behind the [llm] extra. - turbovec added to the [llm] optional extra; core install untouched. - Tests: recall@5 parity with the cosine scan + padding behavior. https://claude.ai/code/session_01TJ94h1zxbtm9g5jVVnYUSW
Contributor
✅ Shadow AI ScanNo Shadow AI findings detected at or above medium risk. |
Contributor
CRUMB Bench Results
Powered by crumb-format — two-stage context compression |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Integrates turbovec (a local TurboQuant vector index) as the retrieval backend for Crumb LLM context pulling, and adds a semantic mode to
crumb search.The key insight: Crumb LLM already turns each section into a spectral signature (
|rFFT(scatter(tokens))|) — a fixed-length float vector. Those signatures are embeddings, so they can be indexed in turbovec instead of scanned linearly, with no external embedding service (keeping the "runs entirely locally" property intact).Changes
crumb_wavelm/turbovec_index.py(new) — thin wrapper overturbovec.IdMapIndex. L2-normalizes signatures so turbovec's inner-product score equals cosine, and zero-pads vectors to turbovec's multiple-of-8 dim requirement (lossless for cosine). numpy-only dependency surface;turbovec_available()gates everything.context_pull.py—CrumbIndex.ensure_backend()lazily builds the vector index;score_sectionssearches it when present (over-fetching candidates so the@priorityboost can still re-rank), and falls back to the unchanged pure-Python cosine scan when turbovec is absent.crumb llm indexnow reports the active backend (turbovecvsnumpy cosine scan).crumb search --method semantic— ranks.crumbsections by wave-field spectral relevance instead of keyword/TF-IDF, backed by turbovec when available. Behind the[llm]extra; prints an install hint otherwise.pyproject.toml—turbovec>=0.7added to the[llm]optional extra. Corecrumb-formatinstall is untouched and install-free.Notes
Tests
tests/test_crumb_llm_context_pull.pygains recall-parity and zero-padding cases. Full suite: 807 passed, 1 skipped.https://claude.ai/code/session_01TJ94h1zxbtm9g5jVVnYUSW
Generated by Claude Code