Skip to content

DESIGN.md

Latest commit

 

History

History
165 lines (150 loc) · 9.48 KB

DESIGN.md

File metadata and controls

165 lines (150 loc) · 9.48 KB
Raw

loadtest — LLM gateway stress tester (design)

Purpose-built load generator for LLM gateways — any OpenAI-/Anthropic-compatible endpoint. Decoupled from the system under test (zero DB dependency). Closed- and open-loop load models, target tens-of-thousands concurrency.

Why not a generic HTTP tool

LLM traffic has semantics a generic tool ignores: multi-turn conversations (growing context), streaming with time-to-first-token, token throughput, cost, and response/prompt caching that silently distorts results. This tool models those first-class.

Layers (each one purpose, testable in isolation)

Engine (staged ramp: ramp → steady hold, warmup excluded)
 └ VU pool per stage (total concurrency split across scenarios by weight)
     └ each VU loops: pick scenario by weight → run one Conversation
         └ Conversation: inject conv-UUID (front of first user msg + header)
             └ 1..turns Turns: build normalized msgs (prior assistant fed back)
                 → Protocol.BuildBody → HTTP/SSE → measure total / TTFT / tokens
 ├ Sink: JSONL, large buffer, batched, crash-safe (decoupled from metrics)
 └ Reporter: per-scenario + aggregate → report.txt + summary.json

The metrics aggregator is in-memory and always complete (feeds the report); the JSONL sink is best-effort under extreme rate (overflow counted, never blocks a VU — blocking would distort latency). At LLM rates the sink never drops.

Protocol adapters (the only place wire-format lives)

The engine is 100% protocol-agnostic; it works on a normalized conversation. A protocol is ONE self-registering file. Adding/changing a protocol touches only that file — never the engine, sink, or reporter.

type Msg struct { Role, Content string }
type Conversation struct { Model, System string; Msgs []Msg; MaxTokens int; Stream bool }
type Turn struct { Content string; PromptTokens, CompletionTokens int }

type Protocol interface {
    Name() string
    Path() string                          // default endpoint path
    BuildBody(Conversation) ([]byte, error)
    ParseNonStream([]byte) (Turn, error)
    ParseStream(io.Reader) (Turn, error)   // owns its SSE format
}
// adapters self-register: func init(){ Register("anthropic", func() Protocol {...}) }

The only stream/non-stream branch in the engine is protocol-agnostic: if c.Stream { p.ParseStream(body) } else { p.ParseNonStream(bytes) }. TTFT is measured by the transport (httptrace), independent of protocol and streaming.

Ship now: openai-chat (/v1/chat/completions), anthropic (/v1/messages). Future (one file each, one register line): openai-responses, gemini, … Protocol-specific headers (e.g. anthropic-version) go in the scenario's headers config — not baked into code.

Config (declarative JSON profile)

{
  "defaults": { "protocol":"openai-chat", "target":"https://.../v1/chat/completions",
                "headers": {"Authorization":"Bearer nvk_..."}, "model":"claude-haiku-4-5",
                "max_tokens":64, "stream":false },
  "stages":  [ {"concurrency":100,"duration":"60s"},          // closed-loop (VU ramp)
               {"rate":4000,"duration":"60s"},               // open-loop constant arrival rate
               {"rate_from":1000,"rate_to":8000,"duration":"60s"} ], // open-loop ramp (rate sweep)
  "warmup":  "10s",
  "arrival": "uniform",            // open-loop inter-arrival: uniform (default) | poisson
  "live_interval": "10s",          // rolling RPS/p50/p99/err progress line ("0" disables)
  "reuse_body": true,              // marshal the sized body once (cache-OFF runs only)
  "openloop_max_inflight": 50000,  // open-loop in-flight bound (drives CO backpressure)
  "cache_mode": "bust",            // bust (default) | natural
  "correlation": { "uuid_in_prompt": true, "header": "x-request-id" },
  "thresholds": { "ttft_p95_ms":0, "p95_ms":0, "error_rate":0, "abort_on_fail":false },
  "scenarios": [
    { "name":"quick-qa", "weight":70, "turns":1, "stream":false, "max_tokens":64,
      "content": {"mode":"pool", "prompts":["...","..."]},
      "checks": {"contains":["chatcmpl-llm-mock"], "min_completion_tokens":1} },
    { "name":"chat", "weight":20, "turns":{"min":3,"max":6}, "stream":true,
      "content": {"mode":"scripted", "script":["Hi","tell me more","and then?"]} },
    { "name":"long-ctx", "weight":10, "turns":2, "stream":true, "max_tokens":256,
      "content": {"mode":"sized", "approx_input_tokens":2000} }
  ]
}

No scenarios block → a single implicit scenario from defaults (simple use). A scenario may override protocol/target → one run can mix OpenAI-ingress and Anthropic-ingress load against the same gateway.

LLM semantics

  • turns: fixed int or {min,max} (distribution). Multi-turn feeds the assistant reply back so context grows like a real session.
  • content: pool (random prompt), scripted (fixed dialogue), sized (generate ~N input tokens — controls input size for throughput tests).
  • cache_mode: bust default — a per-conversation UUID at the FRONT of the first user message forces a cache miss, threads all turns of the conversation, and is the join key to server-side records. Also sent as a header.
  • metrics: TTFT (first token), TPOT/inter-token = (total−TTFT)/completion_tokens, output tokens/s, prompt/completion tokens, total latency, error taxonomy.

Load models (per stage — set exactly one)

  • Closed-loop (concurrency): N VUs loop back-to-back → the max-sustained- throughput / ceiling view. weight = share of the VU pool (1000 VU → 700/200/100).
  • Open-loop (rate): fire on a fixed schedule regardless of in-flight — the honest latency-vs-offered-load view. Mirrors wrk2/vegeta. The in-flight bound (openloop_max_inflight) makes a saturated target backpressure the scheduler.
  • Open-loop ramp (rate_from+rate_to): the rate sweeps linearly across the stage (cumulative-arrivals inverse), walking straight through the latency knee in one stage instead of a hand-listed grid. k6 ramping-arrival-rate.
  • Arrival (arrival): uniform (evenly paced) or poisson (exponential gaps, a non-homogeneous Poisson process tracking any ramp) — Poisson reproduces real- traffic burstiness, so its tails are harsher and more honest.
  • Coordinated-omission correction (open-loop): latency is recorded as done − scheduled_send, NOT done − actual_send, so a backed-up target is charged the full queueing delay and p99/p99.9 stay honest (wrk2/vegeta semantics).
  • warmup window excluded from steady-state stats; startup raises RLIMIT_NOFILE.
  • Percentiles: bounded HDR-style log-linear histogram (metrics/, ~0.4% rel resolution) — accurate p50…p99.9 on a 300s multi-kRPS run with no unbounded sample slice. Count/min/max/mean are exact.
  • Generator health: client-side saturation errors (ephemeral-port/FD exhaustion) listed separately so a generator bottleneck is never misread as a server fault. Single-host ceiling (distributed generation intentionally omitted).

Validation + control

  • checks (per scenario): a 200 whose body fails an assertion (contains / not_contains substrings, min/max_completion_tokens) is counted as a failure (check_failed), catching wrong/truncated content behind a 200. k6 checks.
  • abort_on_fail (thresholds): stop the run after the first stage that breaks a threshold — an arrival-rate sweep halts as soon as a rate breaks the SLO.

Outputs

  • results-<ts>.jsonl — one line per turn (conv-UUID, scenario, turn, stream, start, CO latency + raw svc latency, ttft, dns/conn/tls/transfer breakdown, status, prompt/completion tokens, content_len, err).
  • report-<ts>.txt — per-scenario + aggregate (latency p50…p99.9 + TTFT percentiles, RPS, tokens/s, error breakdown, threshold pass/fail, generator health).
  • summary-<ts>.json — machine-readable (per-stage stageStat).
  • report-<ts>.csv — one row per stage for spreadsheets/notebooks.
  • metrics-<ts>.prom — Prometheus text exposition (textfile collector / Pushgateway).
  • regression compare: loadtest -compare old.json,new.json [-regress-pct 10] diffs two runs by stage, flags RPS drops / tail blow-ups, exits non-zero on a regression (CI gate). No load is generated.
  • join.sh — optional post-step template: if your gateway logs each request to a database, join the client JSONL against the server-side rows by conv-UUID into a combined client+server report. Edit it to match your schema; the load tester itself has no DB dependency.

Layout

Layered Go packages — the design's layers are compiler-enforced, and the engine is importable as a library:

cmd/loadtest/   CLI: flags, wiring, the crash-safe JSONL sink, signals (thin)
protocol/       Protocol interface + registry + openai-chat / anthropic adapters
                (the ONLY place a wire format lives) + SSE inter-token timing
metrics/        lock-free HDR histogram, Sampler, Stat/StageStat (the aggregator)
config/         declarative JSON profile: schema, defaults, validation, content gen
engine/         closed-loop (RunStage) + open-loop/ramp/poisson + CO (RunStageOpenLoop),
                the request transport, response checks, live progress
report/         text report, summary.json, CSV, Prometheus textfile, regression compare

Dependency graph (acyclic): protocol and metrics are leaves; config → {protocol, metrics}; engine → {config, protocol, metrics}; report → {config, metrics}; cmd/loadtest → {config, engine, report}. Single static binary; runs on any generator host.