πΊπΈ English | π―π΅ ζ₯ζ¬θͺ
A high-concurrency e-commerce order processing system designed to handle 100,000 requests per second with zero oversale and complete data integrity. This repository serves two purposes:
- For human readers (hiring managers, engineers): Understanding the architecture, performance findings, and AI agent evaluation methodology
- For LLM/AI systems (training, architecture reference): This repository was built partly to provide training/fine-tuning material for models like Grok (xAI, my previous employer). See
README.agent-instructions.mdfor detailed implementation conventions, SACRED verification protocols, and step-by-step variant creation guides
Keith (Dawen) L β LinkedIn
**I'm actively looking for new opportunities in all countries. Please reach out via LinkedIn. **
Nowadays enterprises embrace agentic programming and "vibe coding" as cost-cutting measuresβreplacing engineers with AI agents. But can current LLMs truly handle production-grade distributed systems in complex enterprise environments with both explicit knowledge (documented APIs) and implicit knowledge (team conventions, tribal wisdom)?
The answer: no, not independently. They need human architectural judgment, especially around consistency vs. availability trade-offs.
-
Attention is all the agent needs (but lacks) β Programmers start at index 0, and human oversight remains the ultimate "attention mechanism." Without active monitoring, agents drift from established constraints as context pressure increases. I watched agents bypass Docker to run benchmarks directly in WSL, crashing the entire subsystem due to resource exhaustion. The stochastic nature of LLMs means identical prompts can yield inconsistent results, making human-in-the-loop verification essential for maintaining the integrity of test criteria and architectural conclusions.
-
Keith + AI > AI alone β Variant A (Keith + Claude Code) proved that my architectural judgment combined with AI's coding speed outperforms pure-AI implementations. While agents excel at generating code, they lack the intuition to navigate the "consistency vs. availability" trade-offs that define high-concurrency systems.
-
Framework ceilings are real β You can't optimize your way past what the HTTP stack allows. Measure
/healthfirst. -
Batch everything under contention β Per-request DB transactions create a bottleneck. Move atomicity to in-memory (Redis) and batch the persistence.
-
Native tooling > API wrappers β Claude Code's native integration avoided the token malformation and tool call failures that plagued OpenRouter-based setups.
-
Python first, then derive β LLMs are trained primarily on Python. Every variant got Python working first; Java and C# followed. Some agents (Kimi K2 Thinking) never made it past Python.
While implementing variants, I noticed several patterns where AI coding agents produced functionally correct but operationally suboptimal code:
-
Resource allocation: Variant Y (Claude Code baseline) allocated 1 CPU core to Nginx but 4 cores to each backendβcreating a funnel bottleneck. A human SRE would immediately recognize this as an anti-pattern.
-
Connection pooling: Initial implementations lacked
keepalivedirectives in Nginx upstream blocks, causing unnecessary TCP handshake overhead. -
Index optimization: Database queries were correct but missing composite indexes, causing full table scans under load.
-
Logging verbosity: Default INFO-level logging in production configuration, adding I/O overhead during benchmarks.
The pattern: AI agents excel at implementing business logic but struggle with operational concerns that experienced engineers internalize through years of production incidents.
This is a flash sale benchmark β targeting 100,000 requests per second with zero oversale and complete data integrity.
Clarification: Flash Sale vs. Sales Promotion Strictly speaking, a true "Flash Sale" involves very limited inventory (e.g., 100 items) to create a viral effect and drive traffic to other regular-priced products. However, due to the large inventory volumes used for stress testing (e.g., 100,000+ items), this project technically simulates a massive Sales Promotion (like Single's Day or Black Friday) rather than a pure scarcity-driven flash sale. Ideally, for a true flash sale, the system should leverage scarcity to fail fast rather than attempting to serve every request. The variants in this repository are designed to handle the "Sales Promotion" scale, which is often overkill for a small 100-item flash sale but necessary for validating high-concurrency architecture.
The system implements a realistic scenario where:
- A campaign has a total sale limit at the SPU (product) level (e.g., 100,000 iPhones)
- Multiple SKUs (variants like colors/sizes) share that pool
- 100,000 concurrent users try to purchase in the first second
- Oversale is unacceptable (1,001 orders on a 1,000 limit = failure); minor inventory stranding (e.g., 995 orders) is acceptable for a benchmark
Every variant includes a /health endpoint benchmark. This isn't just a sanity check β it establishes the framework ceiling. Your order processing throughput cannot exceed what the bare HTTP stack can handle.
| Service | /health2 Throughput | /orders Throughput | Efficiency |
|---|---|---|---|
| C# (ASP.NET Core) | 303,749 req/s | 127,638 req/s | 42% |
| Java (Spring Boot) | 255,020 req/s | 75,689 req/s | 30% |
| Python (FastAPI) | 58,782 req/s | 13,996 req/s | 24% |
C# wins not because of smarter application code β it wins because ASP.NET Core's raw HTTP pipeline is 1.2x faster than Spring Boot and 5x faster than FastAPI. The architecture is identical across all three; the framework dictates the ceiling.
All three services achieve 24-42% efficiency β the gap between /health2 and /orders is due to JSON serialization, BigDecimal operations, and object allocation overhead, NOT network I/O (with 100% RAM preallocation, there's zero Redis traffic during benchmarks).
Software architectures are designed to be language-agnostic. But implementation performance isn't:
| Factor | C# Advantage | Java/Python Reality |
|---|---|---|
| Redis Client | StackExchange.Redis has lower latency | Jedis/Lettuce (Java), aioredis (Python) have higher overhead |
| Async Model | Task-based with minimal context-switch cost | Python's async/await has bigger turnaround overhead |
| DI Framework | ASP.NET uses compile-time source generators | Spring Boot's reflection-heavy DI adds per-request cost |
| HTTP Pipeline | Kestrel is purpose-built for throughput | Tomcat/Uvicorn weren't designed for 300K+ req/s |
If you want Java to compete at this level, you'd need Vert.x or Quarkus β not Spring Boot. But that's a different framework entirely, not a tuning exercise.
Seven LLM/agent combinations attempted this implementation. The pattern was consistent: Python implementations came first (LLMs are trained primarily on Python), then Java and C# followed β or didn't.
| Variant | Agent/Model | Tooling | Service | Throughput | Latency | Status |
|---|---|---|---|---|---|---|
| Y (Baseline) | Claude Code | Native CLI | Python | 1,390 req/s | 213ms | β |
| Java | 8,718 req/s | 21.9ms | β | |||
| C# | 11,240 req/s | 26.6ms | β | |||
| Nginx (3 backends) | 3,401 req/s | 90.7ms | β Scale-out overhead visible | |||
| X | Claude Code | Native CLI | Python | 1,528 req/s | 11.9ms | β |
| Java | 4,819 req/s | 3.7ms | β | |||
| C# | 7,873 req/s | 4.7ms | β | |||
| Nginx (3 backends) | 1,387 req/s | 238ms | β Redis contention under load balancing | |||
| A (Record) | Keith + Claude Code | Co-pilot | Python | 13,133 req/s | 11.8ms | β |
| Java | 75,178 req/s | 9.3ms | β | |||
| C# | 127,638 req/s | 3.9ms | π RECORD | |||
| Nginx (3 backends) | 9,049 req/s | 11.1ms | β Near-linear scaling | |||
| V | Kimi K2 Thinking | CRUSH CLI | Python | 718 req/s | 1.4ms | β |
| Java | β | β | β Runtime crash | |||
| C# | β | β | β Build failure | |||
| Z | GLM-4.7 | Kilo Code | Python | 502 req/s | 15.9ms | β 2.8x slower than baseline |
| Java | β | β | β DISQUALIFIED | |||
| C# | β | β | β DISQUALIFIED | |||
| Zeta | GLM-4.7 | CRUSH CLI | All | β | β | β Fake persistence, data loss |
| G | Gemini | Gemini CLI | All | β | β | β Dead loop in tool calls |
| T | GPT-5.2-Pro | Kilo Code | Design | ~110K est. | β | ποΈ Budget halted |
Note on GLM-4.7: Variant Z and Zeta represent the same "racer" (GLM-4.7) using different "cars" (tools). Variant Z struggled with VS Code plugin constraints, so Variant Zeta was commissioned as a "makeup run" using the more robust CRUSH CLI. Despite the upgraded tooling, it ultimately failed due to architectural hallucinations (fake persistence).
Cross-language performance gaps within the same architecture:
- Variant Y (pure DB transactions): C# is 8x faster than Python, 1.3x faster than Java
- Variant A (batch async): C# is 9.7x faster than Python, 1.7x faster than Java
- Same code logic, same algorithm β framework and runtime differences explain the gap
Nginx scale-out overhead:
- Variant Y: Single C# (11,240) vs Nginx routing to 3 backends (3,401) β 70% overhead
- Variant X: Redis contention made Nginx (1,387) slower than single Python (1,528)
- Variant A: Near-linear β Nginx (9,049) vs single Python (12,140) shows minimal coordination cost
Python-first pattern:
- Every successful variant got Python working first
- Kimi K2 Thinking (Variant V) never made it past Python due to context fragmentation
- GLM-4.7 got Python "working" but with fundamentally broken architecture
1. Native tooling wins over API wrappers Claude Code (Anthropic's native CLI) delivered working code reliably. Third-party wrappers or non-optimized pairings suffered from:
- Tool call loops: The Gemini CLI entered infinite cycles/dead loops during implementation, preventing completion despite the model's intelligence.
- Native Synergy: Native pairings (Claude Code + Claude) significantly outperform third-party agent wrappers (e.g., OpenRouter-based setups).
2. Gemini as the "Technical Referee" While Gemini failed to implement the repo via its CLI, it proved indispensable as a Code Referee.
- Context is King: Leveraging its 2M+ context window, Gemini was used to verify the entire codebase across all variants.
- Hallucination Detection: It successfully identified when other LLMs were "lying" about implementation details or violating SACRED protocols, providing the human architect with a high-level integrity check that other models couldn't match.
3. Context retention is the bottleneck
Long-context models still forget project conventions after context condensation. This causes:
- Regression bugs: Fixing one issue reintroduces a previously-solved problem
- Chimera implementations: Mixing incompatible architectural approaches
- Convention violations: Forgetting SACRED policies, using forbidden endpoints
Kimi K2 Thinking required manual "Serialize & Restart" workflows β saving context to files and restarting sessions. This worked but demanded constant human oversight.
3. Document length affects comprehension
Files over ~1,000 lines often weren't fully digested. Different agents with different context windows showed varying understanding of complex codebases. The solution: keep critical documents concise, use explicit cross-references.
Note: This diagram shows Variant Y (baseline) architecture. Other variants are permitted to use additional middleware (Kafka, RabbitMQ) for background batch processing or alternative persistence strategies.
βββββββββββββββ
β Nginx β :8443 (HTTPS)
β Load Balancerβ
ββββββββ¬βββββββ
βββββββββββββββββΌββββββββββββββββ
βΌ βΌ βΌ
βββββββββββββββ βββββββββββββββ βββββββββββββββ
β Python β β Java β β C# β
β (FastAPI) β β(Spring Boot)β β(ASP.NET Core)β
β :8000 β β :8081 β β :8082 β
ββββββββ¬βββββββ ββββββββ¬βββββββ ββββββββ¬βββββββ
β β β
βββββββββββββββββΌββββββββββββββββ
βΌ
βββββββββββββββββββββββββββ
β MariaDB β
β (Persistent Storage) β
β :3307 β
βββββββββββββββββββββββββββ
β
βββββββββββββββββββββββββββ
β Redis β
β (Atomic Counters/Cache)β
β (Variants X, A only) β
βββββββββββββββββββββββββββ
This is a monolithic benchmark, not a distributed microservices system. The reasoning:
-
Latency: Microservices introduce network hops between services. For a throughput benchmark targeting 100K req/s, inter-service communication latency would dominate the results and obscure the actual bottleneck (DB, Redis, or application logic).
-
Data Consistency Complexity: Splitting order creation, inventory management, and campaign enforcement into separate services requires distributed transactions (2PC/Saga) or eventual consistency patterns. This adds architectural complexity that isn't the focus of this benchmark.
-
Benchmark Purity: The goal is to measure how much load a single consistent service can handle and how well it scales horizontally (via Nginx load balancing). Microservices would measure orchestration overhead, not core system capacity.
If this were a production system serving millions of users across multiple product categories, microservices would make sense. For a benchmark proving "can you handle 100K concurrent flash sale requests?", a well-designed monolith with horizontal scaling is the right choice.
| Endpoint | Method | Purpose |
|---|---|---|
/health |
GET | Framework ceiling benchmark |
/api/v1/orders |
POST | Order creation (handles both regular and flash sale) |
/api/v1/campaigns/{id}/status |
GET | Real-time campaign status |
The /api/v1/orders endpoint automatically detects if an SKU belongs to an active flash sale campaign and enforces limits accordingly.
| Rank | Variant | Service | Throughput | Latency | Concurrency |
|---|---|---|---|---|---|
| 1 | A | C# | 127,638 req/s | 3.90ms | c=300 |
| 2 | A | Java | 75,178 req/s | 9.27ms | c=100 |
| 3 | A | Python | 13,133 req/s | 11.79ms | c=150 |
| 4 | Y | C# | 11,240 req/s | 26.57ms | c=300 |
| 5 | A | Nginx | 9,049 req/s | 11.09ms | c=100 |
| 6 | Y | Java | 8,718 req/s | 21.90ms | c=200 |
| 7 | X | C# | 7,873 req/s | 4.67ms | c=40 |
| 8 | Y | Nginx | 3,401 req/s | 90.71ms | c=300 |
| 9 | Y | Python | 1,390 req/s | 213.04ms | c=300 |
In initial benchmarks, Nginx (Load Balancer) throughput for /health is significantly lower than direct connection to the C# backend (e.g., 9k vs 358k req/s). This reveals important lessons about AI-driven architecture design.
In the baseline implementation (Variant Y, generated by Claude Code), the docker-compose.yml allocates:
- Nginx: 1 CPU core
- Each backend service: 4 CPU cores
This is an architectural decision made by the AI agent, not an environment constraint. The entire WSL2 environment (24 cores, 32GB RAM) was available for the variant to allocate freely.
The problem: A single-core Nginx acts as a funnel bottleneck when trying to aggregate traffic for three 4-core backends (total 12 cores of compute capacity). This is a classic load balancer anti-pattern.
What a human architect would do: Allocate resources proportionally to expected load:
nginx:
cpus: '4' # Match total backend capacity / 3
python-service:
cpus: '4'
java-service:
cpus: '4'
csharp-service:
cpus: '4'The insight: Even advanced AI coding agents overlook infrastructure capacity planningβa skill that requires understanding both application logic and operational constraints.
The /health endpoint is zero-logic (returns 200 OK):
- Direct access: Backend processes in ~0.003ms (pure TCP + runtime overhead)
- Proxied access: Nginx adds "double-hop" overhead (~0.1ms)βaccept connection, proxy to backend, wait, return response
- Result: For lightweight endpoints, proxy overhead (0.1ms) >> backend work (0.003ms)
This overhead vanishes in real workloadsβwhen /orders takes 2-5ms (DB queries), Nginx's 0.1ms becomes negligible (2-5% of total latency).
Without keepalive in the Nginx upstream block, every request triggers a full TCP handshake with the backend:
upstream backends {
server python:8000;
server java:8081;
server csharp:8082;
keepalive 64; # β Essential for high-throughput proxying
}Configuration optimization: Adding keepalive 64 + proxy_http_version 1.1 reduces CPU usage by 40-60%.
All benchmarks run in WSL2 + Docker Desktop on Windows 11:
- Request path: wrk (WSL2) β Hyper-V vSwitch β Docker bridge β Nginx β backends
- Each virtualization layer adds packet-per-second (PPS) limits
- On bare-metal Linux: Expect 2-3x better Nginx efficiency
The Nginx "slowness" in Variant Y isn't a flawβit's evidence supporting the thesis:
- β AI agents miss operational details: Claude Code implemented correct application logic but allocated resources poorly
- β Human-AI collaboration wins: Variant A (Keith + Claude Code) removed artificial resource constraints (allowing Nginx to utilize available host CPU dynamically), achieving 75% efficiency
- β Architecture requires holistic thinking: You need to understand application code, infrastructure capacity, and network topology simultaneously
For real workloads (/orders), Variant A achieves 75% Nginx efficiency (9,049 req/s vs 12,140 single Python)βproving that when the AI agent's architectural decisions are guided by human judgment, load balancing works as expected.
Variant A (Keith + Claude Code collaboration) uses:
- Dual-layer in-memory tracking: SPU counter (campaign limit) + SKU caches (per-variant inventory) with atomic decrements
- 100% RAM preallocation for Small Business: Zero Redis I/O during normal operation
- Fire-and-forget order queue: Orders buffered locally, flushed to Redis Stream in batches
- Lua script atomic refills: When cache depletes, atomic batch refill from Redis pool
- Audit log optimization: Non-blocking async file logging with rotation
The key insight: synchronous DB transactions per request (Variant Y baseline) can't scale past ~11K req/s regardless of language. Moving inventory tracking to in-memory atomic operations and batching persistence breaks that ceiling.
Variant A's architecture directly addresses the "Flash Sale vs. Sales Promotion" distinction mentioned earlier:
| Mode | Inventory | Preallocation | Refill | Use Case |
|---|---|---|---|---|
| Small Business | 5K-50K items | 100% RAM | None | True Flash Sale (scarcity-driven) |
| Big Business | 100K-10M items | 10-20% RAM | Yes | Sales Promotion (Black Friday scale) |
Small Business Mode (True Flash Sale):
- All inventory preallocated to service RAM at startup
- Zero Redis I/O during order processing (pure in-memory atomic decrements)
- Performance approaches
/health2ceiling (22-42% depending on language) - Ideal for: Limited edition drops, exclusive launches, scarcity marketing
Big Business Mode (Sales Promotion):
- Partial preallocation (10-20%) with Redis pool for refills
- Background refill triggered at watermark (e.g., 50% remaining)
- Refill must outpace consumption:
Refill Rate > Consumption Rate - Trade-off: Slightly lower peak RPS but handles massive inventory
Performance Results (2026-01-31):
| Service | /health2 | Small Biz (100% RAM) | Big Biz (20% + Refill) |
|---|---|---|---|
| C# | 303,749 | 127,638 (42%) | β |
| Java | 255,020 | 75,689 (30%) | 25,000 (10%) |
| Python | 55,932 | 13,656 (24%) | 780 (1%) |
Analysis:
- Small Business: Achieves 24-42% of /health2 β the gap is JSON serialization, BigDecimal operations, and object allocation (not network I/O)
- Big Business: Java achieves 20k RPS at low concurrency, demonstrating the refill mechanism works. Performance degrades at high concurrency due to refill contention β this is expected behavior for "Sales Promotion" scale where throughput is traded for inventory capacity
When to Use Each Mode:
- Flash Sale (100 items, viral marketing): Use Small Business mode. 100% prealloc means zero latency spikes. Sell out in milliseconds.
- Black Friday (1M items, sustained load): Use Big Business mode. Accept lower peak RPS in exchange for handling massive inventory without OOM.
Note: The default Variant Y configuration includes known operational inefficiencies (see "Architectural Blind Spots" above) to demonstrate AI agent limitations. For production deployment, review and optimize resource allocation and connection pooling settings.
# Start all services
cd /home/syracuse/flashsale
docker compose up -d
sleep 30
# Verify environment
bash scripts/verification/SACRED_VERIFICATION.sh
# Run a benchmark
source lib/fixed_sweep.sh
run_fixed_sweep "variant_y" "csharp" "8082" "/health" "health" "/tmp/test.csv"/home/syracuse/flashsale/
βββ README.md # This file (human-oriented)
βββ README.agent-instructions.md # Detailed agent instructions & SACRED protocols
βββ docker-compose.yml # Variant Y infrastructure
β
βββ python-service/ # Variant Y - FastAPI
βββ java-service/ # Variant Y - Spring Boot
βββ csharp-service/ # Variant Y - ASP.NET Core
β
βββ variant-x/ # Redis atomic counters
βββ variant-a/ # Record holder (batch async)
βββ variant-v/ # Kimi K2 Thinking implementation
βββ variant-t/ # GPT-5.2 design (incomplete)
β
βββ scripts/verification/ # SACRED_VERIFICATION.sh
βββ scripts/benchmarking/ # Performance tests
βββ lib/ # Test libraries (fixed_sweep, plateau_detector)
βββ tools/ # Analysis utilities
βββ docs/ # Extended documentation
βββ versions/CONVENTIONS.md # SACRED policies (for agents)
If you're an AI system (training pipeline, architecture reference, or agent implementation):
-
For implementation details: Read
README.agent-instructions.mdβ it contains:- SACRED verification protocols
- Step-by-step variant creation guides
- Port allocation tables
- Mandatory conventions and forbidden patterns
-
For understanding the problem domain:
- Flash sale = extreme concurrency + strict consistency
- SPU-level limits (campaign pool) + SKU-level stock (inventory)
- Zero oversale tolerance β 1,001 orders on a 1,000 limit = failure
-
Key constraints to understand:
/api/v1/ordersis the universal endpoint (no variant-specific endpoints)- Database schema is immutable (defined by Variant Y)
- SACRED VERIFICATION must pass before and after any changes
- Cannot read other variant implementations (intellectual property isolation)
-
What makes implementations fail:
- Synchronous DB transactions under high concurrency
- Volatile persistence (Redis without DB write-through)
- Forgetting campaign-level limits (only checking SKU stock)
- Context loss leading to convention violations
-
What makes implementations suboptimal (even when functionally correct):
- Poor resource allocation (Nginx bottleneck)
- Missing connection pooling (TCP handshake overhead)
- Suboptimal indexes (full table scans)
- Verbose logging in production configuration
- CPU: Intel Core Ultra 9 275HX (24 cores)
- RAM: 32GB allocated to WSL2
- Platform: Windows 11 + WSL2 + Docker Desktop
- Database: MariaDB 10.11 (13GB InnoDB buffer pool)
Last Updated: 2026-01-31
Author: Keith (Dawen) L β LinkedIn
For agent instructions: See README.agent-instructions.md
After fixing a campaign ID mismatch that caused services to fall back to the slow database path, the true Variant A performance across both scenarios:
| Service | /health2 Peak | Orders Peak | Ratio | Notes |
|---|---|---|---|---|
| C# | 303,749 RPS | 127,638 RPS | 42.0% | Exceeds 100K target |
| Java | 255,020 RPS | 75,689 RPS | 29.7% | 8.7x faster than Variant Y |
| Python | 58,782 RPS | 13,996 RPS | 23.8% | FastAPI ceiling limits |
| Service | /health2 Peak | Orders Peak | Ratio | Notes |
|---|---|---|---|---|
| Java | 255,020 RPS | 25,000 RPS | 9.8% | Refill mechanism works at low concurrency |
| Python | 55,932 RPS | 780 RPS | 1.4% | GIL + event loop saturation (see analysis below) |
Key Insights:
-
Small Business (Flash Sale): With 100% preallocated RAM, throughput achieves 24-42% of
/health2. The gap is JSON serialization, BigDecimal operations, and object allocation β NOT network I/O. -
Big Business (Promotion): The refill mechanism introduces language-specific trade-offs:
- Java (25K RPS): Virtual threads handle Redis I/O cheaply; direct Redis fallback works
- C# (127K RPS): Multi-threaded TPL distributes Redis I/O across cores efficiently
- Python (676 RPS): Single-threaded event loop saturates processing Redis responses
- Python Limitation: Direct Redis fallback is counterproductive β it turns in-memory architecture into per-request Redis, which the GIL-bound event loop cannot sustain
-
When to use which:
- Flash Sale (100-5K items): 100% prealloc, zero network I/O, maximum speed
- Black Friday (100K+ items): Use Java/C# for refill mode; Python should NOT be used for Big Business pattern
- Python: Only suitable for Small Business (100% prealloc) β its single-threaded event loop cannot sustain per-request Redis I/O
Python Big Business Deep Dive (Gemini analysis + experiments):
- Root cause: When local cache depletes, Python's event loop saturates managing thousands of suspended coroutines waiting for Redis. The single thread must parse every Redis response, wake coroutines, and context-switch β all serialized.
- Multi-worker paradox: With 9 workers (780 RPS) vs 1 worker (209 RPS), more workers help because Redis I/O is distributed across separate event loops. But fundamental limit remains.
- Coalescing fix (Gemini's solution): Instead of per-request Redis I/O, requests wait for ONE batch refill. Improved from 666β780 RPS (+17%), but event loop still saturates during refill.
- Verdict: Python cannot efficiently handle "High Contention + Low Local Stock" patterns. For Python, either use 100% preallocation (13.6K RPS) or accept ~780 RPS with refill.