Skip to content

fix: misc bug fixes and bump fsspec for pyarrow>=17 compat - #1020

Open
cmgzn wants to merge 1 commit into
datajuicer:mainfrom
cmgzn:fix/misc-fixes-and-fsspec-bump
Open

fix: misc bug fixes and bump fsspec for pyarrow>=17 compat#1020
cmgzn wants to merge 1 commit into
datajuicer:mainfrom
cmgzn:fix/misc-fixes-and-fsspec-bump

Conversation

@cmgzn

@cmgzn cmgzn commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Summary

A collection of small bug fixes and a dependency bump.

Changes

  1. fix(image_diffusion_mapper): handle missing or null captions gracefully — prevents crash when samples have missing or empty caption fields. (Related: [Pulp] Handle missing or null captions in image_diffusion_mapper cmgzn/data-juicer#60)

  2. fix: warn when process operator list is empty — adds a warning in both DefaultExecutor and RayExecutor when the user provides an empty process operator list, preventing silent no-op execution. (Related: [Pulp] Empty process list produces no log initialization cmgzn/data-juicer#69)

  3. fix(vggt_mapper): guard and initialize __dj__meta__ before access — ensures __dj__meta__ is initialized before VggtMapper accesses it, preventing KeyError on samples without prior metadata. (Related: [Pulp] Guard and initialize __dj__meta__ in VggtMapper cmgzn/data-juicer#93)

  4. deps: bump fsspec from ==2023.5.0 to >=2026.1.0 — the old pin is incompatible with pyarrow>=17. Root cause: fsspec<=2025.9.0 mirrors ArrowFile.size from the pyarrow stream via @mirror_from (treats it as an attribute), but pyarrow>=17 changed NativeFile.size to a method. This causes TypeError: unsupported operand type(s) for +: 'method' and 'float' in tqdm during HDFS downloads through ArrowFSWrapper. fsspec>=2026.1.0 replaces the mirror with an explicit @property that calls stream.size(), returning the correct int. Verified end-to-end with fsspec==2026.4.0 + pyarrow==24.0.0 on a Docker HDFS setup (load → filter → export).

    Note: datasets==5.0.0 declares fsspec<=2026.4.0, so the effective resolved range is 2026.1.0–2026.4.0. s3fs versions are co-released with matching fsspec versions (e.g. s3fs==2026.4.0 requires fsspec==2026.4.0), so no resolver conflicts.

Comment thread data_juicer/ops/mapper/image_diffusion_mapper.py Outdated
- fix(image_diffusion_mapper): handle missing or null captions gracefully
- fix: warn when process operator list is empty
- fix(vggt_mapper): guard and initialize __dj__meta__ before access
- deps: bump fsspec from ==2023.5.0 to >=2026.1.0 (ArrowFile.size
  changed from mirror_from attribute to @Property in 2026.1.0,
  required for pyarrow>=17 where NativeFile.size is a method)
@cmgzn
cmgzn force-pushed the fix/misc-fixes-and-fsspec-bump branch from 3a6b16a to e7f003a Compare July 29, 2026 06:41
@cmgzn
cmgzn requested a review from fengrui-z July 29, 2026 07:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants