Inference Engineering
LLM inference pipelines and serving infrastructure for low-latency, high-throughput generation. The current focus is continuous batching and KV-cache reuse — keeping GPUs saturated across concurrent requests instead of processing them one at a time.
Key findings
- Continuous batching increased throughput substantially over static batching under mixed-length request loads.
- KV-cache reuse across shared prompt prefixes cut redundant compute on multi-turn agent workloads.
- Speculative decoding gains shrink fast once draft-model acceptance rate drops below a threshold.
Benchmark
| Approach | Tokens/sec | P50 latency | Δ vs. baseline |
|---|---|---|---|
| Baseline (static batching) | 1,120 | 410ms | — |
| Continuous batching | 1,740 | 260ms | +55% |
| Continuous batching + speculative decoding | 2,280 | 190ms | +104% |
Screenshots
scheduler-timeline-before.png
scheduler-timeline-after.png
Attachment
inference-pipeline-optimization-notes.pdf
Download