Inference Worker Lab
Queue-decoupled inference: FastAPI gateway, Redis Streams workers (mock or vLLM), idle-claim recovery, bounded retries, and a lag/pending autoscaler.
Problem
I wanted a lab that shows how inference requests survive worker crashes: reclaim idle pending entries, avoid double model calls when a result already succeeded, bound retries into a DLQ, and scale on lag without dropping claimed work.
Architecture
- POST /v1/jobs XADDs to the inference:jobs stream (consumer group workers). GET /v1/jobs/{id} reads result keys; GET /v1/scale exposes the scaler view.
- Workers XAUTOCLAIM messages idle for IWL_CLAIM_IDLE_MS (default 30s). A crash after XREADGROUP and before XACK is picked up by another worker. If the result key is already succeeded, the reclaim acks and does not call the model again.
- A failed attempt is XADDed again with attempt+1, then the old id is acked. At IWL_MAX_ATTEMPTS the job is marked failed and appended to inference:dlq. A crash between that XADD and XACK can leave two copies; both may run (at-least-once, not exactly-once).
- POST /v1/jobs accepts idempotency_key: the same key returns the same job_id and does not append a second stream entry (there is a crash window after the key is stored and before XADD). Enqueue returns 429 when unfinished work (lag) hits IWL_MAX_QUEUE_DEPTH (default 10000; 0 disables).
- The scaler uses never-delivered lag plus PEL pending, with hysteresis. It does not start or stop containers; it writes a desired count. A worker whose id is worker-N idles when N-1 >= desired.
- MOCK mode (default) uses fixed latency + token echo. VLLM mode calls an OpenAI-compatible chat completions endpoint.
Design decisions
XAUTOCLAIM for crash recovery
Idle pending entries are claimed after CLAIM_IDLE_MS. Succeeded results short-circuit reclaim so the model is not called twice.
Trade-off: At-least-once delivery: a crash between retry XADD and XACK can leave duplicates that both run.
Lag vs pending for scaling
Scale-up can use never-delivered lag or claimed-but-unacked in-flight. Scale-down only when lag is at or below SCALE_DOWN_LAG and pending is 0.
Trade-off: Avoids shrinking while claimed work is outstanding; the scaler does not orchestrate containers itself.
429 at max queue depth
When unfinished work reaches IWL_MAX_QUEUE_DEPTH, enqueue refuses with HTTP 429.
Trade-off: Explicit backpressure to clients; depth 0 disables the limit.
Stack
What I'd improve next
- Exactly-once or stronger dedupe across the retry XADD/XACK window.
- GPU / compose profile measurements with a real vLLM backend (pytest skips GPU).
Results
- pytest cases (MemoryRedis + 2 against real Redis 7.0.15 when listening)
- 26
- mock queue_to_done p95 (80 jobs, concurrency 4, 5 ms backend sleep; not GPU/production)
- ~214 ms