Fix gene coloring 400 with pandas 3.0 string dtype var index - #2794
Open
colganwi wants to merge 1 commit into
Open
Fix gene coloring 400 with pandas 3.0 string dtype var index#2794colganwi wants to merge 1 commit into
colganwi wants to merge 1 commit into
Conversation
Under pandas 3.0 (and pandas 2.x with future.infer_string), text columns
default to a string dtype instead of object: the NumPy-backed "str" dtype,
or StringDtype ("string"/"string[pyarrow]"). AnnData files written/read in
that mode have a string-dtype var index (gene names) and obs label columns.
_annotation_filter_to_mask only treated ["boolean", "category", "object"]
as discrete/value-filtered; a string-dtype column fell through to the
numeric min/max branch, which does nothing for a value filter. The mask
stayed all-True, so a single-gene GET /data/var selected every column and
tripped column_request_max -> ExceedsLimitError -> HTTP 400 when coloring
by a gene.
Use pandas.api.types.is_string_dtype so all string variants (object,
"string", "str", pyarrow-backed) take the value-equality path. Adds
regression tests for StringDtype and the pandas 3.0 "str" dtype.
Fixes chanzuckerberg#2793
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Fixes #2793.
Coloring by a gene returns HTTP 400 for AnnData files whose
varindex (gene names) uses a pandas string dtype instead ofobject— the default for text columns under pandas 3.0, and under pandas 2.x withfuture.infer_string:Root cause
DataAdaptor._annotation_filter_to_maskchooses between an exact-value filter and a numeric min/max filter by dtype name:A pandas string dtype (
"string","string[pyarrow]", or the pandas 3.0 NumPy-backed"str") is not in the allow-list, so a gene-name value filter falls into the numeric branch and does nothing. The mask stays all-True, so every var column is selected;data_frame_to_fbs_matrixthen exceedscolumn_request_maxand raisesExceedsLimitError→ 400. The same applies to obs categorical value filters stored as a string dtype.Fix
Route all pandas string dtypes through the value-equality branch via
pandas.api.types.is_string_dtype:Tests
Adds
StringDtypeFilterMaskTestcovering bothStringDtype("string") and the pandas 3.0"str"dtype (viafuture.infer_string), asserting the value filter selects exactly the matching row.Reproduction