Add citus_internal.distribute_object repair UDF#8621
Open
onurctirtir wants to merge 16 commits into
Open
Conversation
Adds a superuser-only UDF citus_internal.distribute_object(classid, objid) that recreates a single object on all worker nodes and records it in pg_dist_object on the coordinator and all workers, in an idempotent manner. This is a repair tool for objects that should have been distributed but were not (e.g. due to a bug). It reuses GetDependencyCreateDDLCommands to build the DDL for the object itself and deliberately ignores the object's dependencies; the caller is responsible for those. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #8621 +/- ##
==========================================
- Coverage 88.73% 88.73% -0.01%
==========================================
Files 288 288
Lines 64384 64464 +80
Branches 8108 8114 +6
==========================================
+ Hits 57133 57202 +69
- Misses 4909 4920 +11
Partials 2342 2342 🚀 New features to boost your workflow:
|
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Instead of broadcasting create-or-replace style DDL to every worker, probe
each node for the object by name via pg_get_object_address() and only open a
connection and run the create DDL on the nodes that are missing it.
This avoids worker_create_or_replace_object()'s rename-on-conflict behavior
on nodes that already have the object. We accept the small TOCTOU window
between the existence check and the create: the worst case is a loud failure,
which is fine for a superuser-only repair UDF.
Also align the local-existence error message with the regression test's
expected output ("does not exist on the local node").
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
onurctirtir
force-pushed
the
onurctirtir-cuddly-waffle
branch
from
fce694f to
55cbf18
Compare
Adds a force_recreate boolean (default false) to citus_internal.distribute_object. When false (the default) we keep skip-missing-only: a node is skipped if it already has the object, avoiding worker_create_or_replace_object's rename-on-conflict. When true, the per-node existence probe is skipped and the create DDL is re-applied on every node, which syncs drift such as role grants/options that were never propagated to nodes that already have the role. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Cover function, schema, collation, type, ts dict/config, sequence, view, publication, role, extension and foreign server with: no-force create, force on existing, and force-resync after coordinator-only drift. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ction Previously the per-node DDL was committed in its own session via SendCommandListToWorkerOutsideTransactionWithConnection so that the created object would be visible to the separate metadata connection used by MarkObjectDistributedViaSuperUser. That left object creation and the pg_dist_object records non-atomic. Send the DDL over the same REQUIRE_METADATA_CONNECTION that MarkObjectDistributedViaSuperUser reuses, inside a coordinated 2PC transaction, so creation and metadata records commit or roll back together. The existence probe now runs inside a savepoint so a name-resolution error does not abort the surrounding remote transaction. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The per-node existence probe called pg_get_object_address() directly, which raises when the object is absent. To keep that error from aborting our coordinated 2PC remote transaction, the probe was wrapped in a SAVEPOINT/ROLLBACK TO SAVEPOINT dance in C. Add a small citus_internal.object_exists(text, text[], text[]) PL/pgSQL helper that traps the lookup error and returns a boolean. The probe now runs as a plain SELECT that never errors, so RemoteObjectExists() drops all savepoint handling and just reads the boolean result. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Recreating the object over a coordinated 2PC metadata connection required probing existence inside the same remote transaction, which in turn needed either a SAVEPOINT/ROLLBACK dance or a PL/pgSQL EXCEPTION helper to keep pg_get_object_address()'s not-found error from aborting the transaction. There is no fully generic non-erroring existence check in core SQL (the to_reg* family does not cover text search, publication, extension or foreign server objects), so both options were undesirable. Go back to the original approach: send the create DDL over a separate FORCE_NEW_CONNECTION outside any coordinated transaction. The existence probe's not-found error is then naturally contained on that standalone connection, so no savepoint or exception handling is needed. The trade-off is that recreation is no longer atomic across nodes, which is acceptable for a re-runnable superuser repair UDF. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ommandReturnsRow Cover scenarios iv (object present on coordinator + one worker, repaired on the missing workers) and v (object already present everywhere, not recreated) for every supported object type. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
For the role (the motivating case) and the function (representative type), assert that the object is absent from pg_dist_object on the coordinator and every worker before distribute_object(force_recreate := false) and present on all of them afterwards. The pg_dist_object probe joins the catalog by name instead of casting to a reg* type so it does not error on workers where the object does not exist yet. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
After a force_recreate := true call on an object that already exists everywhere, assert that pg_dist_object still holds exactly one row on the coordinator and every worker for the function and the role, proving the distributed mark is not duplicated. This also adds an explicit ii-a block to the role section. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…hots - dependencies.c: add the second blank line citus_indent requires between top-level definitions (check-style failure). - multi_extension.out and upgrade_list_citus_objects.out: update the citus_internal.distribute_object signature to (oid,oid,boolean) to reflect the force_recreate parameter (multi_extension, check-multi-mx and check-pg-upgrade failures). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
RemoteCommandOnNodeReturnsRow re-raised statement-level errors (e.g. the existence probe's "function ... does not exist" on a worker missing the object), which broke the missing-object code path and failed check-multi-1-create-citus. Restore the original 3-way handling: connection-level failure raises, while a statement-level error is treated as "no rows returned" so the probe reports the object as absent and the caller creates it. Keeps the OUTSIDE_TRANSACTION self-owned connection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
onurctirtir
commented
Jun 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DESCRIPTION: Adds citus_internal.distribute_object() tool that can be used to manually propagate an object to workers.
This is a super-user only UDF that can be used by admins to
re-create an object as distributed in cluster-wide.