Skip to content

perf: Reduce eager Ray dataset actions - #1025

Open
fengrui-z wants to merge 1 commit into
datajuicer:mainfrom
fengrui-z:codex/reduce-ray-eager-actions
Open

perf: Reduce eager Ray dataset actions#1025
fengrui-z wants to merge 1 commit into
datajuicer:mainfrom
fengrui-z:codex/reduce-ray-eager-actions

Conversation

@fengrui-z

Copy link
Copy Markdown
Collaborator

Summary

Reduce unnecessary eager Ray Dataset actions and fix row retrieval truncation.

Changes

  • Remove the full count() call before building an operator pipeline.
  • Use take(k) directly instead of count(), limit(k), and take().
  • Use take_all() when get_column() is expected to return every row.
  • Avoid repeated columns() and schema() calls.
  • Add regression coverage with datasets containing more than 20 rows.

Motivation

Ray Datasets are lazily evaluated, but count() triggers execution. Calling it before constructing every processing pipeline adds unnecessary work, especially when the partitioned executor invokes process() for each partition or checkpoint group.

Additionally, Ray’s take() defaults to 20 rows. The previous implementation could therefore silently truncate results from get() and get_column() even when more rows were requested.

Validation

  • 20 related tests passed.
  • Verified that 25 requested rows are returned without truncation.
  • Verified that process() no longer calls count() before building the execution plan.
  • All pre-commit checks passed.

Note

get_column(k=None) now follows its documented behavior and returns all rows using take_all(). As before, callers should avoid using it with datasets too large to fit in driver memory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant