fix: misc bug fixes and bump fsspec for pyarrow>=17 compat - #1020
Open
cmgzn wants to merge 1 commit into
Open
Conversation
fengrui-z
reviewed
Jul 29, 2026
- fix(image_diffusion_mapper): handle missing or null captions gracefully - fix: warn when process operator list is empty - fix(vggt_mapper): guard and initialize __dj__meta__ before access - deps: bump fsspec from ==2023.5.0 to >=2026.1.0 (ArrowFile.size changed from mirror_from attribute to @Property in 2026.1.0, required for pyarrow>=17 where NativeFile.size is a method)
cmgzn
force-pushed
the
fix/misc-fixes-and-fsspec-bump
branch
from
July 29, 2026 06:41
3a6b16a to
e7f003a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A collection of small bug fixes and a dependency bump.
Changes
fix(image_diffusion_mapper): handle missing or null captions gracefully — prevents crash when samples have missing or empty caption fields. (Related: [Pulp] Handle missing or null captions in image_diffusion_mapper cmgzn/data-juicer#60)
fix: warn when process operator list is empty — adds a warning in both
DefaultExecutorandRayExecutorwhen the user provides an empty process operator list, preventing silent no-op execution. (Related: [Pulp] Empty process list produces no log initialization cmgzn/data-juicer#69)fix(vggt_mapper): guard and initialize
__dj__meta__before access — ensures__dj__meta__is initialized beforeVggtMapperaccesses it, preventingKeyErroron samples without prior metadata. (Related: [Pulp] Guard and initialize __dj__meta__ in VggtMapper cmgzn/data-juicer#93)deps: bump fsspec from
==2023.5.0to>=2026.1.0— the old pin is incompatible withpyarrow>=17. Root cause:fsspec<=2025.9.0mirrorsArrowFile.sizefrom the pyarrow stream via@mirror_from(treats it as an attribute), butpyarrow>=17changedNativeFile.sizeto a method. This causesTypeError: unsupported operand type(s) for +: 'method' and 'float'in tqdm during HDFS downloads throughArrowFSWrapper.fsspec>=2026.1.0replaces the mirror with an explicit@propertythat callsstream.size(), returning the correct int. Verified end-to-end withfsspec==2026.4.0+pyarrow==24.0.0on a Docker HDFS setup (load → filter → export).Note:
datasets==5.0.0declaresfsspec<=2026.4.0, so the effective resolved range is2026.1.0–2026.4.0.s3fsversions are co-released with matching fsspec versions (e.g.s3fs==2026.4.0requiresfsspec==2026.4.0), so no resolver conflicts.