Skip to content

Fix gene coloring 400 with pandas 3.0 string dtype var index - #2794

Open
colganwi wants to merge 1 commit into
chanzuckerberg:mainfrom
colganwi:fix/pandas3-string-dtype-var-filter-upstream
Open

Fix gene coloring 400 with pandas 3.0 string dtype var index#2794
colganwi wants to merge 1 commit into
chanzuckerberg:mainfrom
colganwi:fix/pandas3-string-dtype-var-filter-upstream

Conversation

@colganwi

Copy link
Copy Markdown

Problem

Fixes #2793.

Coloring by a gene returns HTTP 400 for AnnData files whose var index (gene names) uses a pandas string dtype instead of object — the default for text columns under pandas 3.0, and under pandas 2.x with future.infer_string:

GET /api/v0.2/data/var?var:name_0=<gene>  ->  400

Root cause

DataAdaptor._annotation_filter_to_mask chooses between an exact-value filter and a numeric min/max filter by dtype name:

if anno_data.dtype.name in ["boolean", "category", "object"]:
    key_idx = np.in1d(anno_data, values)   # value filter
    ...
else:
    # numeric min/max — a no-op when only "values" is provided

A pandas string dtype ("string", "string[pyarrow]", or the pandas 3.0 NumPy-backed "str") is not in the allow-list, so a gene-name value filter falls into the numeric branch and does nothing. The mask stays all-True, so every var column is selected; data_frame_to_fbs_matrix then exceeds column_request_max and raises ExceedsLimitError → 400. The same applies to obs categorical value filters stored as a string dtype.

Fix

Route all pandas string dtypes through the value-equality branch via pandas.api.types.is_string_dtype:

if anno_data.dtype.name in ["boolean", "category", "object"] or pd.api.types.is_string_dtype(anno_data):

Tests

Adds StringDtypeFilterMaskTest covering both StringDtype ("string") and the pandas 3.0 "str" dtype (via future.infer_string), asserting the value filter selects exactly the matching row.

Reproduction

import anndata, numpy as np, pandas as pd
pd.set_option("future.infer_string", True)  # pandas 3.0 default
adata = anndata.AnnData(
    X=np.random.default_rng(0).random((10, 2000), dtype=np.float32),
    var=pd.DataFrame(index=[f"gene{i}" for i in range(2000)]),
)
adata.write_h5ad("repro.h5ad")
# cellxgene launch repro.h5ad -> color by a gene -> 400 (before this fix)

Under pandas 3.0 (and pandas 2.x with future.infer_string), text columns
default to a string dtype instead of object: the NumPy-backed "str" dtype,
or StringDtype ("string"/"string[pyarrow]"). AnnData files written/read in
that mode have a string-dtype var index (gene names) and obs label columns.

_annotation_filter_to_mask only treated ["boolean", "category", "object"]
as discrete/value-filtered; a string-dtype column fell through to the
numeric min/max branch, which does nothing for a value filter. The mask
stayed all-True, so a single-gene GET /data/var selected every column and
tripped column_request_max -> ExceedsLimitError -> HTTP 400 when coloring
by a gene.

Use pandas.api.types.is_string_dtype so all string variants (object,
"string", "str", pyarrow-backed) take the value-equality path. Adds
regression tests for StringDtype and the pandas 3.0 "str" dtype.

Fixes chanzuckerberg#2793
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Gene coloring returns HTTP 400 with pandas 3.0 string dtype var index

1 participant