Ilongin/12872 dataset soft delete by ilongin · Pull Request #1770 · datachain-ai/datachain

ilongin · 2026-05-14T08:58:01Z

Closes datachain-ai/studio#12872.

catalog.remove_dataset_version no longer hard-deletes a COMPLETE user dataset version by default. Instead it:

drops the warehouse rows table,
sets status = REMOVED + removed_at = now(),
keeps the version row and all its dataset_dependencies so dependents can still render lineage,
permanently reserves the semver — saving the same name again auto-bumps past the removed version (no slot reuse).

Non-COMPLETE versions (CREATED/FAILED/STALE leftovers from the GC path) and internal datasets (lst__*, session_*) continue to fully delete — no benefit to keeping their metadata.

`keep_metadata` flag

A new keep_metadata: bool = True parameter on catalog.remove_dataset_version, catalog.remove_dataset, and dc.delete_dataset controls the behavior:

Default (keep_metadata=True): the behavior above — keeps the REMOVED record.
keep_metadata=False: fully wipes the version (rows, dependencies, version row, and the dataset row if it was the last). Reserved for cases like GDPR/PII removal or cleaning up garbage versions.

Exposed via CLI as datachain datasets rm <name> --no-keep-metadata.

State machine

Three new/repurposed dataset statuses:

REMOVING = 7 (repurposed) — keep-metadata removal in progress; GC resumes to REMOVED
REMOVED = 8 (new) — terminal state for keep-metadata path; semver permanently reserved
REMOVING_DROP_METADATA = 9 (new) — wipe in progress; GC resumes to row deletion

remove_dataset_version routes resumption purely off current status, so a GC retry (which doesn't have access to the caller's original intent) lands the row in the correct end state for both paths.

Other surface changes

DatasetRecord.live_versions returns versions excluding REMOVED ones. latest_version / latest_major_version / latest_compatible_version / DatasetListRecord.latest_version skip REMOVED.
_max_version / _max_version_value (private) consider all versions including REMOVED, so auto-bump never reclaims a reserved semver.
DatasetVersion.removed_at field added (timestamp of removal).
Checkpoint and delta paths detect REMOVED and fall back to recreate / rebuild instead of trying to read a stub.

Schema

Column-only: removed_at (nullable timestamp). OSS handles it via the existing auto-migration in _migrate_table_schema; Studio companion PR adds the corresponding Django migration.

codecov · 2026-05-14T09:06:36Z

Codecov Report

❌ Patch coverage is 93.63636% with 7 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/datachain/catalog/catalog.py	84.78%	5 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

cloudflare-workers-and-pages · 2026-05-15T04:06:21Z

Deploying datachain with Cloudflare Pages

Latest commit:	`fe1a214`
Status:	✅ Deploy successful!
Preview URL:	https://b09cc851.datachain-2g6.pages.dev
Branch Preview URL:	https://ilongin-12872-dataset-soft-d.datachain-2g6.pages.dev

View logs

amritghimire

I think more general approach on how soft delete are used is, adding a new column deleted_at, so we can clear items in trash for long time if needed by filtering with time. Also, a simple filter on selection to deleted_at to null would do the work and so on,

ilongin · 2026-05-15T07:01:27Z

I think more general approach on how soft delete are used is, adding a new column deleted_at, so we can clear items in trash for long time if needed by filtering with time. Also, a simple filter on selection to deleted_at to null would do the work and so on,

I did add removed_at column.

…ain-ai/datachain into ilongin/12872-dataset-soft-delete

amritghimire · 2026-05-18T15:05:16Z

I think more general approach on how soft delete are used is, adding a new column deleted_at, so we can clear items in trash for long time if needed by filtering with time. Also, a simple filter on selection to deleted_at to null would do the work and so on,

I did add removed_at column.

If we have removed_at column, why add another status? We can preserve the status as it was before removed imo.

ilongin · 2026-05-19T06:45:14Z

I think more general approach on how soft delete are used is, adding a new column deleted_at, so we can clear items in trash for long time if needed by filtering with time. Also, a simple filter on selection to deleted_at to null would do the work and so on,

I did add removed_at column.

If we have removed_at column, why add another status? We can preserve the status as it was before removed imo.

It's strange to me to leave status field to COMPLETE or similar but dataset is removed. We also have created_at and CREATED status. I would rather have little bit of duplicate than risk of reading wrong information.

shcheklein · 2026-06-10T18:09:21Z

-            DatasetStatus.REMOVING,
-            DatasetStatus.REMOVED,
-        ):
+        if keep_metadata and not v.is_soft_deletable:


can we call it is_internal? or even is_system?

is soft deletable again is not reusable - we are just leaking removal business logic outside

was it addressed?

I completely removed this method as it's not really needed so no need for figuring out naming. I agree that soft delete should not be used anywhere but not sure what is substitute for that to be honest ..

dreadatour

I would prefer to be able to soft delete listing datasets (to keep meta) as it can be a dependency for all other datasets.

It also can be useful for Knowledge Base.

dreadatour · 2026-06-18T12:53:41Z

+            )
+        if (
+            not keep_metadata
+            and v.status == DatasetStatus.REMOVING


What about other statuses? REMOVING_TOTAL, REMOVED?

This is about checking if someone wants to fully remove / wipe dataset version but in the same time default removing with keeping metadata is present...in that way we should raise.

so REMOVING is ongoing removal where we keep metadata. REMOVING_TOTAL is ongoing removal status which ends up with removing metadata and actual data table

dreadatour · 2026-06-18T13:08:39Z

State machine

Three new/repurposed dataset statuses:

REMOVING = 7 (repurposed) — keep-metadata removal in progress; GC resumes to REMOVED

REMOVED = 8 (new) — terminal state for keep-metadata path; semver permanently reserved

REMOVING_DROP_METADATA = 9 (new) — wipe in progress; GC resumes to row deletion

Why do we need these statuses (REMOVING, REMOVING_DROP_METADATA)? What if "removing" will fail, will dataset (version) ends up with broken state (REMOVING) forever?

Will it be easier to basically mark datasets (versions) as REMOVED and remove actual rows tables in garbage collector? Then next GC run will remove all stale rows tables, no matter if previous was failed or not. Same in SaaS. We can also run GC (remove rows tables) in SaaS only if/when ClickHouse is active.

I see this as more robust and easier logic, rather than to have state machine and all the logic around to process edge cases (failed removal, locks, etc).

shcheklein · 2026-06-18T13:52:51Z


-            if dataset.versions and len(dataset.versions) == 1:
-                # had only one version, fully deleting dataset
+            # Count in DB, not in the in-memory record: GC-shaped paths


this is a very bad comment:

what is GC-shaped paths?

it is referring to now non-existent line len(dataset.versions) == 1 - it will be impossible to understand w/o PR

can be simpler ... a lot simpler

Please review everything AI generates, clean it up

shcheklein · 2026-06-18T13:55:13Z

-            self.update_dataset_version(dataset, version, **update_data)
+        with self.db.transaction():
+            if version:
+                # Update the version row first. If a status guard was requested


What is updated "second"?

It was about updating Dataset object below. I updated comment to make it more clear

ilongin · 2026-06-19T07:48:36Z

State machine

Three new/repurposed dataset statuses:

REMOVING = 7 (repurposed) — keep-metadata removal in progress; GC resumes to REMOVED

REMOVED = 8 (new) — terminal state for keep-metadata path; semver permanently reserved

REMOVING_DROP_METADATA = 9 (new) — wipe in progress; GC resumes to row deletion

Why do we need these statuses (REMOVING, REMOVING_DROP_METADATA)? What if "removing" will fail, will dataset (version) ends up with broken state (REMOVING) forever?

Will it be easier to basically mark datasets (versions) as REMOVED and remove actual rows tables in garbage collector? Then next GC run will remove all stale rows tables, no matter if previous was failed or not. Same in SaaS. We can also run GC (remove rows tables) in SaaS only if/when ClickHouse is active.

I see this as more robust and easier logic, rather than to have state machine and all the logic around to process edge cases (failed removal, locks, etc).

We need those statuses because we have 2 different types of delete now:

Completely remove dataset metadata and warehouse data from DB - current way
Keep metadata, mark it as REMOVED, but remove warehouse data

If any of the process fails GC needs to continue removing but we need to save information of what kind of delete was in action. If it get's stuck in REMOVING, GC will know how to continue. I don't like depending completely on GC for removing warehouse data as it's little bit strange and it won't work on local where user needs to run GC explicitly.

ilongin added 2 commits May 13, 2026 23:08

working on dataset soft delete

73e3c77

adding logic and fixing tests

dc34aa7

ilongin marked this pull request as draft May 14, 2026 08:58

ilongin commented May 14, 2026

View reviewed changes

Comment thread src/datachain/catalog/catalog.py Outdated

Merge branch 'main' into ilongin/12872-dataset-soft-delete

b0dfe90

amritghimire reviewed May 15, 2026

View reviewed changes

ilongin requested a review from amritghimire May 15, 2026 07:03

ilongin added 3 commits May 15, 2026 13:47

Merge branch 'main' into ilongin/12872-dataset-soft-delete

d88917b

Merge branch 'ilongin/12872-dataset-soft-delete' of github.com:datach…

a7f0b15

…ain-ai/datachain into ilongin/12872-dataset-soft-delete

fixing test

236af57

shcheklein reviewed May 15, 2026

View reviewed changes

Comment thread src/datachain/catalog/catalog.py Outdated

shcheklein reviewed May 15, 2026

View reviewed changes

Comment thread src/datachain/data_storage/metastore.py Outdated

shcheklein reviewed May 15, 2026

View reviewed changes

Comment thread src/datachain/dataset.py Outdated

shcheklein reviewed May 15, 2026

View reviewed changes

Comment thread src/datachain/delta.py Outdated

shcheklein reviewed May 15, 2026

View reviewed changes

Comment thread src/datachain/delta.py Outdated

ilongin added 3 commits May 18, 2026 09:18

Merge branch 'main' into ilongin/12872-dataset-soft-delete

7737683

changing logic of solf delete to reserve removed datasets

425ac01

adding purge option

e71555e

ilongin requested a review from shcheklein May 18, 2026 13:28

ilongin marked this pull request as ready for review May 18, 2026 13:56

shcheklein reviewed May 20, 2026

View reviewed changes

Comment thread src/datachain/catalog/catalog.py Outdated

shcheklein reviewed May 20, 2026

View reviewed changes

Comment thread src/datachain/catalog/catalog.py Outdated

shcheklein reviewed May 20, 2026

View reviewed changes

Comment thread src/datachain/catalog/catalog.py Outdated

shcheklein reviewed May 20, 2026

View reviewed changes

Comment thread src/datachain/catalog/catalog.py Outdated

shcheklein requested a review from Copilot May 20, 2026 00:46

shcheklein reviewed Jun 9, 2026

View reviewed changes

Comment thread src/datachain/catalog/catalog.py Outdated

shcheklein reviewed Jun 9, 2026

View reviewed changes

Comment thread src/datachain/data_storage/metastore.py Outdated

ilongin added 3 commits June 10, 2026 11:49

refactoring

bccfeed

refactoring

301810a

refactoring

d19f0fd

shcheklein reviewed Jun 10, 2026

View reviewed changes

ilongin requested a review from shcheklein June 10, 2026 21:38

ilongin added 15 commits June 12, 2026 15:14

adde missing flag

845cdb6

refactoring

822328f

refactoring

52d557e

fix race condition

5c9461d

refactoring

cdace00

added transaction

c991878

fixing export

4874900

refactoring

2d4b9e6

refactoring dependencies

99833bd

removed purge option on interface

40e6152

added 2 more tests

664d9ab

Merge branch 'main' into ilongin/12872-dataset-soft-delete

fac7aee

Merge branch 'main' into ilongin/12872-dataset-soft-delete

f22bfc3

simplifying constants

352822e

Merge branch 'main' into ilongin/12872-dataset-soft-delete

5cbe55b

dreadatour reviewed Jun 18, 2026

View reviewed changes

shcheklein reviewed Jun 18, 2026

View reviewed changes

fixing PR comments

fe1a214

ilongin requested review from dreadatour and shcheklein June 19, 2026 13:19

Conversation

ilongin commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

keep_metadata flag

State machine

Other surface changes

Schema

Uh oh!

codecov Bot commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

cloudflare-workers-and-pages Bot commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying datachain with Cloudflare Pages

Uh oh!

amritghimire left a comment

Choose a reason for hiding this comment

Uh oh!

ilongin commented May 15, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

amritghimire commented May 18, 2026

Uh oh!

ilongin commented May 19, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dreadatour left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dreadatour commented Jun 18, 2026

State machine

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ilongin commented Jun 19, 2026

State machine

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ilongin commented May 14, 2026 •

edited

Loading

`keep_metadata` flag

codecov Bot commented May 14, 2026 •

edited

Loading

cloudflare-workers-and-pages Bot commented May 15, 2026 •

edited

Loading