Back to projects

InferBench

TensorRT cut GPU latency 5.6x. A Redis cache cut repeat requests 2,377x. Every number is a real, reproducible measurement.

SWE (AI/MLOps)MLOpsInference OptimizationONNXTensorRTFastAPIObservability

InferBench benchmarks a Vision Transformer (ViT-Base) across PyTorch, ONNX Runtime, and TensorRT on GPU, and across PyTorch, ONNX Runtime, and TorchScript on CPU, then serves it through a FastAPI backend with a from-scratch dynamic batching layer (asyncio.Queue and asyncio.Future, no batching library), a Redis response cache, PostgreSQL request logging, and a Prometheus and Grafana observability stack, all orchestrated with Docker Compose and tested via GitHub Actions CI against real Redis and PostgreSQL service containers. TensorRT FP16 delivered a 2.9x to 5.6x GPU speedup over PyTorch with 100% prediction match on a 100-sample validation set; the Redis cache cut a 1,996.88ms cache-miss request to 0.84ms, a 2,377x speedup, verified end to end. On CPU-only hardware, the honest result reversed: plain PyTorch eager mode beat both ONNX Runtime and TorchScript at every batch size, reported as measured rather than assumed.

System architecture: FastAPI, dynamic batching, Redis, PostgreSQL, Prometheus, Grafana
Request lifecycle: cache check, async batching window, fan-out, and metrics logging
Local demo frontend: live prediction with cache status and latency
Live Grafana dashboard: request latency, throughput, and cache hit rate

What I built

  • Benchmarked ViT-Base (86M parameters) across PyTorch, ONNX Runtime, and TensorRT FP16 on an NVIDIA T4 GPU across batch sizes 1 to 16 with full p50/p95/p99 latency, TensorRT delivered a 2.9x to 5.6x speedup while matching 100% of predictions on a 100-sample validation set.
  • Ran a matching CPU-only benchmark and found the honest, counter-intuitive result: plain PyTorch eager mode beat both ONNX Runtime and TorchScript at every batch size, the opposite of the GPU result.
  • Built a dynamic request-batching layer entirely from scratch using asyncio.Queue and asyncio.Future (no batching library), collecting requests in a 10ms window and fanning results back out per caller, verified under real concurrent load (20 simultaneous requests forming actual batches of size 2 and 16).
  • Built a Redis response cache keyed on image-content hash plus backend, with graceful degradation if Redis is unreachable, verified end to end with a 2,377x latency reduction (1,996.88ms cache miss to 0.84ms cache hit).
  • Logged every request's backend, cache status, batch size, prediction, and latency to PostgreSQL, and instrumented Prometheus and Grafana with custom latency histograms and a version-controlled, auto-provisioned dashboard.
  • Wired a GitHub Actions CI pipeline running automated tests against real Redis and PostgreSQL service containers on every push, with the full stack deployable via a single Docker Compose command.

Tech stack

SWE (AI/MLOps)MLOpsInference OptimizationONNXTensorRTFastAPIObservability

Key results

up to 5.6x

GPU speedup (TensorRT FP16)

2,377x

Cache speedup

100%

Prediction match

16

Max verified batch size

Back to all projects