[rollout, vllm] feat: KV-cache-aware request load balancer

[touch869]

What does this PR do?

Add a KV-cache-aware request load balancer as a new routing option for verl's rollout servers, migrated from the standalone uni-agent LLM router. The new balancer routes each request by combining prefix-cache hit rates (GPU/CPU/SSD tiers) with live load metrics (KV-cache usage, running/waiting requests), and preserves sticky sessions for multi-turn conversations with overload-aware fallback.

It is fully opt-in: when no router config is provided (or router.type: default), the existing GlobalRequestLoadBalancer behavior is unchanged.

Depends on #6712.

Related issues/PRs:

Checklist Before Starting

• Search for similar PRs. Paste at least one query link here:
• Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)
  • Suggested title: [rollout, vllm] feat: KV-cache-aware request load balancer

Test

Import, config-parsing, strategy-construction, and LLMServerManager integration smoke tests pass:

pytest tests/workers/rollout/test_kvc_aware_balancer.py::TestKVCAwareBalancerImport -v
# 4 passed

End-to-end validation on a real vLLM rollout (throughput / cache-hit comparison vs. the default balancer):

API and Usage Example

API change (backward compatible): LLMServerClient._acquire_server() gains an optional prompt_ids: list[int] = None parameter, forwarded to the balancer for content-aware routing. Existing callers are unaffected. A new optional router section is added under the rollout config.

actor_rollout_ref:
  rollout:
    name: vllm
    router:
      type: kvc_aware          # omit or "default" -> existing GlobalRequestLoadBalancer
      config:
        strategies:
          - _target_: verl.workers.rollout.llm_router.config.strategy.KVCAwareStrategyConfig
            alpha: 0.7            # S = alpha*S_cache + (1-alpha)*S_load
            load_threshold: 0.9  # sticky session bypassed when load > threshold
            layer_weights: {gpu: 0.7, cpu: 0.2, ssd: 0.1}
            collector_names: [vllm_polling]
            weight: 1.0
        sticky_max_size: 10000
    prometheus:
      enable: true
    disable_log_stats: false     # required for Prometheus metrics

from verl.workers.rollout.llm_router import KVCAwareBalancer  # available as a drop-in balancer

Design & Code Changes

The router is a self-contained package under verl/workers/rollout/llm_router/ that satisfies the same balancer interface as GlobalRequestLoadBalancer (acquire_server / release_server / add_servers / remove_servers / get_all_servers / get_status).

• New package verl/workers/rollout/llm_router/:
  • balancer.py — KVCAwareBalancer orchestration shell (wrapped with ray.remote at init).
  • strategies/ — kvc_aware scoring, weighted routing, sticky_session, load_score, registry.
  • collectors/ — RouteDataProvider + vLLM metrics/KV decoders over HTTP-polling and ZMQ transports.
  • store/ — KV-cache hit and metrics stores.
  • config/ + configs/ — Hydra/OmegaConf config parsing and default YAMLs.
• verl/workers/rollout/llm_server.py:
  • LLMServerManager._init_global_load_balancer() selects the balancer by router.type (kvc_aware vs default), keeping the default path unchanged.
  • LLMServerClient._acquire_server() / generate() thread prompt_ids through for cache-aware routing.
• Tests: tests/workers/rollout/test_kvc_aware_balancer.py.
• Docs: package README.md, config example, and a repo-level quickstart.

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

• Read the Contribute Guide.
• Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
• Add / Update the documentation.
• Add unit or end-to-end test(s) to the CI workflow to cover all the code. If not feasible, explain why:
• Once your PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace.
• If your PR is related to the recipe submodule, please also update the reference to the submodule commit via git submodule update --remote or cd recipe && git pull origin main.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 5 committers have signed the CLA.

❌ yyyyrf
❌ ZOULQ
❌ zhaizhiqiangA
❌ hangangqiang
❌ touch869

---

yyyyrf seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[tardis-key]

There are too many changes in this pr. Can we try breaking it into smaller ones?