[Question]: Question about the hard limit of 10000 in doc_metadata_service.py for document metadata

[liang0112-ne]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (Language Policy).
• Non-english title submitions will be closed directly ( 非英文标题的提交将会被直接关闭 ) (Language Policy).
• Please do not modify this template :) and fill in all the required fields.

Describe your problem

There is a hard-coded limit=10000 in doc_metadata_service.py, which applies to both metadata updates and document metadata display in knowledge bases. I would like to ask why this limit was introduced.
For knowledge bases with more than 10,000 documents, metadata can still be updated via missing_doc, which makes this limit seem redundant for the update scenario. However, the display function cannot bypass the 10000 retrieval cap.
Could you consider adopting other Elasticsearch APIs to remove this restriction? Also, is there any official plan to optimize or adjust this limit in future releases?

[dosubot]

Hi @liang880912! I'm Dosu and I'm helping the InfiniFlow team.

The limit=10000 in get_flatted_meta_by_kbs() appears to be an undocumented safety/performance cap rather than a deliberate architectural decision — there are no code comments explaining why it was chosen [1]. Notably, the same file's _search_metadata() function already implements full pagination (page size 1000, looping until all results are fetched), demonstrating that retrieving beyond 10k is technically feasible [2].

Why it matters: For datasets with >10k documents, get_flatted_meta_by_kbs() silently truncates results (HTTP 200), which breaks:

1. Retrieval with metadata_condition (empty results)
2. Filter statistics in the UI (remaining docs misreported as empty_metadata)
3. Batch metadata CRUD selectors

As you noted, metadata updates can work via missing_doc paths, but the read/display/filter path cannot bypass this cap.

Current fix status:

PR	Status	What it does
#14109	Merged	Added warning log at the 10k boundary, but no actual fix
#14576	Merged	Push-down filtering to ES, but kept 10k cap
#16095	Open	Proposes paginating get_flatted_meta_by_kbs() with a 1000-record loop [3]
#16041	Open	Similar pagination fix + track_total_hits=True for accurate counts [4]

There's no official timeline for merging these, but the maintainers have acknowledged the issue and the fix direction is clear: refactor get_flatted_meta_by_kbs() to paginate via _search_metadata() and fix the stop condition to use len(page_docs) < page_size instead of relying on ES's potentially capped total [5]. You may want to follow PRs #16095 and #16041 for progress.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[zeweihan]

The dangerous part of this cap isn't that it exists — it's the read/write asymmetry it creates. Writes still succeed through the missing_doc path (no cap), but reads silently truncate to 10k. So an operator indexing document #10,501 gets a successful write response, and the very next metadata-conditioned retrieval against that document returns empty results with HTTP 200. There's no signal anywhere in the response that the document exists but is invisible. That's the classic silent-corruption signature: a plausible-looking-empty result beats an explicit error every time, because nothing tells the caller to page, retry, or warn the user.

Two angles worth adding to dosu's summary:

1. Metadata rows scale with chunks, not documents — the cap hits much sooner than "10k docs" implies. For text-heavy corpora (legal agreements, regulatory filings, clinical protocols), a single source document routinely fans out into 50–200 chunks, each carrying its own {doc_id, source_page, box, effective_date, jurisdiction, clause_anchor, version} provenance row. "10k metadata rows" is reached at roughly 50–200 actual documents, not 10,000. The operator's mental model ("I have 800 documents, I'm nowhere near the limit") is wrong by 1–2 orders of magnitude, and the failure only surfaces as retrieval silently returning nothing for filters that should match.

2. The fix direction in #16095 / #16041 is right, but push it one step further — surface truncation to the API consumer, not just the operator. #14109's warning log helps server-side debugging, but the application calling get_flatted_meta_by_kbs() still sees a clean 200 with a short list and no way to distinguish "10k is everything" from "10k is a truncated slice of 47k." If track_total_hits=True (already in #16041) gives an accurate total, exposing {results, truncated, total_available} on the read-path response turns silent truncation into informed truncation — the downstream app can page, warn, or fail loudly. In a retrieval pipeline that distinction is load-bearing: "degraded but flagged" is safe to ship; "plausible but incomplete with no signal" is exactly the failure mode that sends a regulator or a partner the wrong answer with full confidence.

Concrete suggestion: as long as the cap stays in place, return total_available (via track_total_hits) alongside the truncated list so consumers can detect len(results) < total_available without a second round-trip — that single field converts this from a silent-data-loss bug into a known, handleable limit.