Skip to content

relocate memory backpressure throttle from ScanIngress consumer to emit_event producer side #3135

@liquidsec

Description

@liquidsec

Context

#3134 fixes the immediate user-visible failure mode of the memory-pressure ingress throttle (drain-mode feedback loop). The fix is surgical: skip the throttle when no module has work in flight. But the throttle is in the structurally wrong place, and the drain-mode bypass is a workaround for that.

The structural problem

The throttle lives in ScanIngress.get_incoming_event — the consumer side of the pipeline. That's backwards:

  • Queue-buffered memory pressure is caused by emit_event (producers adding to queues).
  • Queue-buffered memory pressure is relieved by get_incoming_event (the consumer removing from queues).
  • Backpressure belongs on the increaser, not the decreaser.

Throttling the consumer when queues are full means the only thing that frees memory is the only thing being slowed down. In the pathological case (drain-only mode, modules killed), this is a hard deadlock that requires the special-case bypass added in #3134.

Proposal

Move the throttle into BaseModule.emit_event. Every producer slows down proportionally under memory pressure; consumers stay unblocked and keep draining. Drain mode is self-handling: no producers running, nothing to throttle, ingress runs at full speed without needing a special-case predicate.

Why this is a real refactor, not a quick relocation

  • Every emit path needs to honor the delay: BaseModule.emit_event, _emit_event, intercept-module emit paths (ScanIngress.queue_event, ScanEgress.forward_event), output module emits.
  • Intercept modules emit too — throttling ScanIngress/ScanEgress reintroduces the drain-mode deadlock from the other end. They need to be exempt.
  • Async semantics differ: sleeping in emit_event blocks the caller's task, not a centralized consumer. Modules will compete for cycles differently than they do today.
  • Interaction with existing backpressure: per-module _qsize limits, the init_events seed-queue throttle (manager.py:67). Need to think through how the layers compose.
  • Once this lands, the drain-mode bypass in bypass ingress throttle in drain mode #3134 can be removed.

References

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions