feat: allow disabling/filtering Claude SDK built-in tools on `ai.action`

[eviltik]

Update — scope clarification (post-investigation)

Same as for #2654 (cross-link): exposing allowed_tools on the ai.action form requires extending AgentActionArgs in packages/tracecat-ee/tracecat_ee/agent/schemas.py, which is in the EE module. The custom-source-level part is fully OSS.

We don't intend to ship code into the EE module ourselves. The OSS PR will include a defensive getattr so the cascade is forward-compatible — when EE exposes the field, the action-level override starts taking effect automatically with no further OSS change. Maintainers can apply the EE one-liner at merge time, or guide us towards a different layout before the PR opens.

---

TL;DR

ai.action always inlines the JSON schemas of the full Claude Code built-in toolset (Bash, Read, Write, Edit, Grep, Glob, MultiEdit, etc.) in the system prompt — even though the action is documented as a "simple LLM call, no tools" and runs with max_tool_calls=0.

That's ~3 940 input tokens of pure overhead per call (96 % of the system context), measured below.

This issue proposes exposing one optional field (allowed_tools: list[str] | None) at both the custom-source and action levels, mapping directly onto ClaudeAgentOptions.tools which already supports the full range (None = SDK default, [] = disable all built-ins, ["Bash","Read"] = whitelist).

Default None everywhere → zero behaviour change for existing workflows.

Measured impact

Tested directly against the bundled CLI 2.1.85 inside sysdio2_tracecat_agent_executor, ANTHROPIC_BASE_URL pointed at a remote Ollama running qwen3.5:9b. User prompt: "Reply: hi" (3 tokens). The CLI's --output-format json returns a usage.input_tokens field that captures system prompt + tool definitions + user prompt cumulatively.

#	Configuration	input_tokens	Delta vs A
A	CLI defaults (no flags)	4 096	reference
B	--system-prompt "You are Qwen, by Alibaba."	4 096	0
C	--system-prompt "..." --tools ""	158	−96 %
D	--bare --system-prompt "..." --tools ""	157	−96 %

So roughly 96 % of the per-call overhead comes from the inlined JSON tool definitions. For an ai.action that's just doing a one-shot LLM completion (classification, extraction, summarization), this is pure waste — and on top of that, smaller models often hallucinate around tools they "see" but cannot actually call.

Capability already in the SDK

ClaudeAgentOptions.tools natively supports:

• tools=None → no --tools flag → CLI applies its default (full toolset).
• tools=[] → --tools "" → all built-ins disabled.
• tools=["Bash","Read"] → --tools "Bash,Read" → whitelist.

The proposal is to surface this in the Tracecat schema.

Proposed change

Custom source (default)

Field	Type	Default
allowed_tools	list[str] | None	None

ai.action schema (per-call override)

Same field. When set, takes precedence over the source-level value.

Cascade rule

if action.allowed_tools is not None:
    resolved = action.allowed_tools          # action overrides source (even if [])
elif source.allowed_tools is not None:
    resolved = source.allowed_tools          # source default if no action value
else:
    resolved = None                           # SDK default (full toolset)

The runtime maps resolved directly onto ClaudeAgentOptions.tools.

Backward compatibility

• New field defaults to None → zero behaviour change for existing workflows or sources.
• disallowed_tools and existing built-in filtering logic in Tracecat are untouched.
• MCP servers (tracecat-registry) are not affected by this PR.

Note on pydantic_ai runtime

The pydantic-ai harness doesn't have the same notion of "implicit built-in tools" — it only sees the tools you give it. So allowed_tools would be a no-op there. We'd log a WARNING if it's set on a workflow that resolves to the pydantic-ai runtime, and document this clearly.

Scope of implementation

• tracecat/db/models.py: allowed_tools JSONB NULL on agent_custom_provider.
• Alembic migration.
• tracecat/agent/provider/schemas.py: extend Create/Update/Read.
• Registry action schema for ai.action: add allowed_tools.
• tracecat/dsl/action.py:build_agent_args_activity: cascade.
• tracecat/agent/runtime/claude_code/runtime.py: pass tools=resolved to ClaudeAgentOptions.
• tracecat/agent/runtime/pydantic_ai/runtime.py: warn if set, ignore.
• Frontend: types regenerated via just gen-client-ci. Custom source modal: add the field under "Agent settings". Action form picks it up automatically through schema generation.
• Tests:
  • Unit: verify the cmd array produced by subprocess_cli._build_command includes --tools "" when tools=[], --tools "Bash,Read" when whitelist, and no --tools flag when None.
  • Cascade: matrix of source × action × None.
  • Token-overhead regression: integration test asserting input_tokens for tools=[] is well below the default.
• Docs: update docs/agents/ai-action.mdx with a "Reducing token overhead" section.

Open question for maintainers

Two design alternatives we considered but rejected:

1. Auto-disable tools when model_provider == "custom-model-provider". Pro: zero UX. Con: silent behaviour change for existing custom-source users — some might rely on tools being there.
2. Auto-disable tools when max_tool_calls == 0. Same trade-off, with the bonus of fitting the ai.action semantics. Still a silent behaviour change though.

The opt-in design proposed here is more conservative and matches the spirit of Passthrough mode (you choose when to skip default behaviour). Happy to discuss if maintainers prefer one of the auto-disable variants.

Happy to send the PR. Splitting source-level and action-level fields into two PRs is doable.

[topher-lo]

This is a legit issue. @eviltik yeah if you have an idea on how to tackle this cleanly with minimal change (I think it's possible to be very surgical with this). Happy to review

[topher-lo]

@eviltik also let's chat async in our Discord #contributors channel? https://discord.gg/H4XZwsYzY4 I think the ideas you're pushing towards but we have really high quality bar for contributors 🫡

[eviltik]

Status update (2026-05-08)

Cross-referencing #2654: the per-source / DB-column approach (which would have been the natural pattern to expose allowed_tools here too) was closed by maintainers as too much surface area. They suggested an env var instead.

For allowed_tools, the env-var path would be TRACECAT__AGENT_DEFAULT_ALLOWED_TOOLS= (CSV) — a global override that disables/whitelists tools at process start.

Pausing this issue while we decide whether to send the slim env-var PR. The token-overhead measurements (~96 % savings on ai.action with tools=[]) still stand and remain the main motivation. Open to maintainer feedback on whether a slim version of this would be welcome upstream, or whether this should also stay on a fork.

[eviltik]

Quick meta-comment to thread #2654, #2655, #2656, and #2652 — they look like four disconnected requests but they're four faces of the same shape: deployability for self-hosted operators running Tracecat alongside existing infrastructure.

The pattern they share:

1. Network shape (Add basePath support so Tracecat can be served under a sub-path #2652): one hostname per app domain, no sub-domain delegation. → basepath support.
2. Multi-backend identity (feat: expose system prompt overrides on ai.action and custom sources #2654): different LLM backends should present themselves differently per workspace. → per-source system prompt.
3. Token economics (feat: allow disabling/filtering Claude SDK built-in tools on ai.action #2655): high-volume ai.action workloads pay a ~96 % per-call overhead from the full SDK toolset, even when the upstream model can't execute those tools. → per-source tools control.
4. Diagnosability (feat: surface runtime_resolution metadata on agent action results for troubleshooting #2656): with overrides at multiple layers, operators need a single place to see what was actually applied on a given run.

Each one alone reads "niche". Stacked, they're table-stakes for the class of self-hosted deployments where Tracecat will be compared against established workflow-automation platforms — those platforms already check these boxes. Operators who hit the gap rarely file an issue; they conclude "doesn't fit, moving on".

A note on the closures so far: nothing in the maintainer feedback on #2651 / #2657 challenged the functional case for either feature — the reasoning was about surface area and maintenance cost, both legitimate. Functionally, the patches were verified end-to-end (Keycloak SSO, full UI click-through, Jest suites, token-overhead measurements) and passed Cubic / semgrep / lint / typecheck.

That distinction matters because it leaves room to converge: if the blocker is surface area, we can ship narrower / opt-in versions; if it's code quality / tests / docs / conventions, we'll happily iterate to whatever bar you set — that's a fair ask of any contributor.

Concrete next steps from our side:

• feat: allow disabling/filtering Claude SDK built-in tools on ai.action #2655: per your "happy to review if surgical" — we can ship a minimal, opt-in form (no DB column, no UI surface) exposing the SDK-native tools=[] capability via a single ai.action argument. ~30 lines. Lowest-risk merge, highest-value given the token math.
• feat: expose system prompt overrides on ai.action and custom sources #2654: looks blocked on the EE schema discussion you raised with @daryllimyt. Once that lands, the cascade collapses to a similar surgical patch.
• feat: surface runtime_resolution metadata on agent action results for troubleshooting #2656: a useful follow-up once feat: expose system prompt overrides on ai.action and custom sources #2654 / feat: allow disabling/filtering Claude SDK built-in tools on ai.action #2655 are merged — the observability counterpart to the cascade. Happy to align with @jordan-umusu's broader error-handling work.
• Add basePath support so Tracecat can be served under a sub-path #2652: kept on the fork as suggested. Mentioned here only because it belongs to the same thesis.

Not asking you to revisit any decision today. Putting the framing on record, and signalling we're available to iterate on the technical bar whenever the merge calculus shifts.

[topher-lo]

See #2658 (comment)

We'll take this on 🫡