feat: enable client-side bulk update/delete operations with SearchIndex (RAAE-1326)

[vishal-bala]

Motivation

Redis has no "delete/update by query." To remove or change many documents today, you scan the index for matching keys and issue the writes yourself, batch by batch. It works, but every team rewrites the same loop — and gets the sharp edges (batching, cluster hash-slots, pagination limits) wrong in slightly different ways.

This PR adds two filter-driven bulk operations to SearchIndex and AsyncSearchIndex, so a common request becomes one call:

from redisvl.query.filter import Tag, Num

# Remove every archived document from before 2020
index.drop_by_filter((Tag("status") == "archived") & (Num("year") < 2020))

# Mark all drafts as published, leaving every other field untouched
index.update_by_filter(Tag("status") == "draft", {"status": "published"})

Both return a BulkResult (matched, processed, completed, dry_run) and accept dry_run=True for a no-op count preview, an on_progress(processed, matched) callback, and a tunable batch_size.

Implementation

Redis still can't mutate-by-query, so the work is client-side; the value is doing it correctly behind one call.

• drop_by_filter resolves matching keys and removes them with non-blocking UNLINK, re-querying from offset 0 each round. Because deleted docs leave the result set, the set shrinks to empty on its own — the same trick clear() uses — which sidesteps RediSearch's MAXSEARCHRESULTS paging cap. A count-based backstop guards against a runaway loop under heavy concurrent inserts; if it trips, the result reports completed=False and logs a warning rather than silently under-reporting.

• update_by_filter applies a partial update: HSET for hash indexes, JSON.MERGE (RFC 7396) at the document root for JSON. An update doesn't remove a doc from the match set, so the shrink-to-empty trick can't work. Instead it resolves the full key set via FT.AGGREGATE ... WITHCURSOR (unbounded, unlike FT.SEARCH), then writes in pipelined batches. The two phases are strictly separated because reading an aggregation cursor while the same index is being written hangs the cursor (verified against Redis 8). The cursor is always released (FT.CURSOR DEL) on exit.

• Naming follows RedisVL's existing convention — drop_* removes documents (drop_keys, drop_documents), while delete() removes the index. drop_by_filter slots into the first group; delete() is left untouched.

• drop_keys / drop_documents now chunk large inputs and stay cluster-safe (per-key UNLINK across hash slots).

Caveats

• Not atomic across the match set. Each document is deleted/updated atomically, but batches apply incrementally with no rollback — a crash mid-run leaves some documents changed and the rest untouched. Both operations are idempotent, so re-running the same call converges on the intended state; that is the recovery model (there is no checkpoint/resume).
• update_by_filter memory scales with the match count (all keys are resolved before writing). For very large match sets, partition the filter and run in waves.
• Concurrency: if a matching document is deleted by someone else between key resolution and the write, update_by_filter recreates it as a partial document.
• JSON updates: values merge at the document root, so keys must match the document's JSON layout, not the schema field name (a field indexed at $.metadata.status needs {"metadata": {"status": ...}}). Values are written without schema validation; only static values are supported.
• Redis Cluster: writes are issued per key (no cross-slot pipelining), so large jobs are slower on Cluster.
• Filters: prefer the escaping builders (Tag, Num, ...) — never string-concatenate untrusted input into a filter, since here an injected predicate mutates data.

Future steps

• Resumable/checkpointed execution for corpus-scale jobs.
• Callable/expression transforms for update_by_filter (e.g. price *= 1.1).
• Optional partial-field schema validation.
• Per-slot write pipelining on Cluster.

Testing

tests/integration/test_bulk_operations.py — 18 tests covering sync/async, hash/JSON, the match-all guard, dry_run, multi-batch cursor iteration, JSON nested-merge and None-deletes-path, and the batched drop_* helpers. make format and make check-types (mypy) pass.

Jira: RAAE-1326

---

Note

Medium Risk
Bulk filter APIs can delete or mutate large document sets with non-atomic batch semantics; filter injection or overly broad filters are destructive, though match-all guards and existence-guarded updates reduce some failure modes.

Overview
Adds client-side bulk mutation on SearchIndex and AsyncSearchIndex because Redis has no delete/update-by-query.

drop_by_filter matches documents via a filter, then removes them in batches with UNLINK, re-querying from offset 0 (like clear) so paging limits do not block large deletes. It returns BulkResult (matched, processed, completed, dry_run), supports dry_run, allow_all for match-all guards, on_progress, and a runaway backstop that can set completed=False.

update_by_filter does partial updates: HSET on hash indexes and JSON.MERGE at $ on JSON. It first resolves all matching keys with FT.AGGREGATE + cursor, then writes in batches (read/write phases are separated because cursors cannot be read while mutating). Updates use Lua scripts that only write if the key still exists, so concurrent deletes are skipped rather than resurrected as partial docs.

drop_keys and drop_documents now accept batch_size (default 500), chunk large lists, and use per-key UNLINK on cluster to avoid CROSSSLOT.

Documentation in docs/concepts/search-and-indexing.md covers semantics, JSON path caveats, durability, and scale guidance. Integration tests in tests/integration/test_bulk_operations.py cover hash/JSON, async, guards, dry-run, concurrency skips, and batched drops.

^{Reviewed by Cursor Bugbot for commit cacde65. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit e922cf5. Configure here.}

[rbs333]

Love this! At one point in history we had talked about adding something like this but it felt to the wayside. Only thing I would check is to build the docs and make sure the updates are linked properly!