InferBench
TensorRT cut GPU latency 5.6x. A Redis cache cut repeat requests 2,377x. Every number is a real, reproducible measurement.
InferBench benchmarks a Vision Transformer (ViT-Base) across PyTorch, ONNX Runtime, and TensorRT on GPU, and across PyTorch, ONNX Runtime, and TorchScript on CPU, then serves it through a FastAPI backend with a from-scratch dynamic batching layer (asyncio.Queue and asyncio.Future, no batching library), a Redis response cache, PostgreSQL request logging, and a Prometheus and Grafana observability stack, all orchestrated with Docker Compose and tested via GitHub Actions CI against real Redis and PostgreSQL service containers. TensorRT FP16 delivered a 2.9x to 5.6x GPU speedup over PyTorch with 100% prediction match on a 100-sample validation set; the Redis cache cut a 1,996.88ms cache-miss request to 0.84ms, a 2,377x speedup, verified end to end. On CPU-only hardware, the honest result reversed: plain PyTorch eager mode beat both ONNX Runtime and TorchScript at every batch size, reported as measured rather than assumed.
What I built
- Benchmarked ViT-Base (86M parameters) across PyTorch, ONNX Runtime, and TensorRT FP16 on an NVIDIA T4 GPU across batch sizes 1 to 16 with full p50/p95/p99 latency, TensorRT delivered a 2.9x to 5.6x speedup while matching 100% of predictions on a 100-sample validation set.
- Ran a matching CPU-only benchmark and found the honest, counter-intuitive result: plain PyTorch eager mode beat both ONNX Runtime and TorchScript at every batch size, the opposite of the GPU result.
- Built a dynamic request-batching layer entirely from scratch using asyncio.Queue and asyncio.Future (no batching library), collecting requests in a 10ms window and fanning results back out per caller, verified under real concurrent load (20 simultaneous requests forming actual batches of size 2 and 16).
- Built a Redis response cache keyed on image-content hash plus backend, with graceful degradation if Redis is unreachable, verified end to end with a 2,377x latency reduction (1,996.88ms cache miss to 0.84ms cache hit).
- Logged every request's backend, cache status, batch size, prediction, and latency to PostgreSQL, and instrumented Prometheus and Grafana with custom latency histograms and a version-controlled, auto-provisioned dashboard.
- Wired a GitHub Actions CI pipeline running automated tests against real Redis and PostgreSQL service containers on every push, with the full stack deployable via a single Docker Compose command.
Tech stack
Key results
up to 5.6x
GPU speedup (TensorRT FP16)
2,377x
Cache speedup
100%
Prediction match
16
Max verified batch size



