Optimize q filter to use pk__in subqueries instead of set operations

[acheng-01]

The q complex filter composed its NOT/AND/OR operands with QuerySet .difference()/.intersection()/.union(), which compile to SQL EXCEPT/INTERSECT/UNION. For NOT in particular, the left operand is the full unfiltered queryset, so evaluating an expression like repository_version=X AND NOT repository_version=Y seq-scans the entire content table and sorts/hashes every leg, spilling to disk on large repositories.

Rewrite the three operand evaluators to compose with pk__in subqueries:

• _NotAction: qs.exclude(pk__in=expr.evaluate(qs).values("pk"))
• _AndAction: chained qs.filter(pk__in=...) semi-joins
• _OrAction: OR of Q(pk__in=...) subqueries

This collapses the plan to indexed anti-/semi-joins with no full-table scan, no SetOp, and no disk-spilling sorts. Results are unchanged (each operand still filters the same base queryset on a unique pk, so the implicit de-duplication of the set operations is preserved); the existing q filter functional tests pass without modification.

Benchmarked on a 200k-unit repository computing a 1000-unit diff between two repository versions (steady-state, work_mem=4MB):

Version sizes	Before (set-ops)	After (pk__in)
199k units	~1400 ms	~680 ms
50k units	~450 ms	~150 ms

Planning time also drops (e.g. ~1.4 ms -> ~0.5 ms) as the large inlined operand lists are replaced by parameterized subqueries.

Closes #7831

Assisted by: Claude Opus 4.8

📜 Checklist

• Commits are cleanly separated with meaningful messages (simple features and bug fixes should be squashed to one commit)
• A changelog entry or entries has been added for any significant changes
• Follows the Pulp policy on AI Usage
• (For new features) - User documentation and test coverage has been added

See: Pull Request Walkthrough

[mdellweg]

FYI, the complexity is meant to prevent easy dos attacks by generating arbitrarily complex queries. I guess you can already get quite far with the current setting.

[mdellweg]

I would be willing to go this way.
It sounds like it can be a performance boost for all uses of q.

Do I understand correctly that your concern is manly how the performance is impacted by deeply nesting constructs?

[mdellweg]

This is supposed to be markdown, not ReST.
Single backticks work better, as far as I recall.

[acheng-01]

I would be willing to go this way. It sounds like it can be a performance boost for all uses of q.

Do I understand correctly that your concern is manly how the performance is impacted by deeply nesting constructs?

Kind of. The benchmark copilot ran yesterday were done from more theoretical flattened SQL queries that q= can not actually achieve and thus produced misleading improvement figures. Because each sub-filter now returns qs.filter(...) and the operators compose by wrapping, every operand expands into several stacked core_content self-joins. For example, repository_version=X turns into:

content WHERE pk IN (SELECT pk FROM content WHERE pk IN (
    SELECT pk FROM content WHERE pk IN (SELECT unnest(content_ids) FROM version)))

Switching from set-ops to pk__in subqueries removes the EXCEPT / INTERSECT disk-spill sorts, which is an improvement for all of q usage, but it trades in one more wrapping layer, so the net gain through the actual filter is more modest than a flattened query (going the base_version filter route) would suggest (~15% rather than ~100%). I just want to be upfront that it optimizes the shape rather than eliminating the underlying per-operand nesting.

[acheng-01]

Following up on my own comment — I pushed a commit that I think actually addresses the performance issue.

• Before: each operand of an AND / OR / NOT got its own pk__in subquery, so nesting just stacked up more layers of SQL.
• Now: each operand can report an as_q() — if it's a plain field filter (name, dates, numbers, __in, etc.) it returns a Q object, and the operators just combine those with & / | / ~. A whole expression of standard fields therefore folds into a single flat SQL WHERE, no matter how deep it's nested.
• Operands that genuinely can't be reduced — labels, repository_version, href/prn/id lookups — return no Q and fall back to a pk__in subquery like before.
• Since nearly every filter on every endpoint is a plain field, this helps the common case across the board.
• repository_version on its own is unchanged (~2.4s on 200k units) since it stays a subquery, it just doesn't get the flattening benefit.

The existing q filter tests all pass without changes.

[acheng-01]

Some more benchmarks run on actual differences in code:

100k rows (200k for repository_version), best of 4.
The number next to the dot is the number of SELECTs made by the query.

ContentGuard (name)

Expression	set-ops	subquery	hybrid
A AND B	68 ms · 2	41 ms · 3	14 ms · 1
A OR B (tiny result)	11 ms · 2	23 ms · 3	13 ms · 1
(A AND B) OR C	128 ms · 3	101 ms · 5	17 ms · 1
NOT A	124 ms · 2	31 ms · 2	12 ms · 1

Repository (name / number)

Expression	set-ops	subquery	hybrid
name AND name	164 ms · 2	43 ms · 3	15 ms · 1
(A AND B) OR (C AND D)	170 ms · 4	98 ms · 7	20 ms · 1
(A AND (B OR C)) AND NOT D (complexity 8)	559 ms · 5	166 ms · 8	22 ms · 1
name AND retain>=x AND retain<y	251 ms · 3	52 ms · 4	15 ms · 1
(num range) OR num=0	263 ms · 3	117 ms · 5	13 ms · 1

Task

Expression	set-ops	subquery	hybrid
name AND NOT name	303 ms · 3	69 ms · 4	17 ms · 1
name AND (NOT state OR state)	478 ms · 4	105 ms · 6	97 ms · 5

Content — repository_version (opaque, 200k units)

Expression	set-ops	subquery	hybrid
199k AND NOT 199k	2662 ms · 7	2307 ms · 8	2234 ms · 8
50k AND NOT 50k	1900 ms · 5	1458 ms · 6	1558 ms · 6

As the results show, the changes made in the new commit will in fact improve q= performance overall for all queries. However, for operands that can't be reduced/flattened, performance doesn't have as dramatic of an improvement. So in our case with repository_version = X AND NOT repository_version = Y, we would get marginal benefits from this vs the other base_version filter change. We could still look into that route and/or we should do something about the double wrapper on ContentRepositoryVersionFilter that might not need to be there. Happy to hear any thoughts.