Back to projects

DocDesk: Healthcare RAG with an Evaluation Harness

A medical RAG service that cites its sources, refuses what it should not answer, and tells me honestly whether a change actually helped.

SWE (AI/MLOps)RAGLLM EvaluationLLM-as-JudgeFastAPIPostgreSQLpgvectorRedisCeleryDockerGitHub ActionsGroqONNX

DocDesk is a medical RAG service that is containerised and runs locally with Docker, built over 298 public MedlinePlus and DailyMed documents (5,919 chunks) using FastAPI, PostgreSQL plus pgvector, Redis, and Celery. Every answer carries citations, and the service has two honest failure modes: it abstains when the retrieved sources do not cover a question, and it refuses outright, no model call, when a question asks for a dose or a diagnosis. Around it sits an evaluation harness: 75 verified questions, LLM judges for faithfulness and correctness, a deterministic failure labeler, and paired bootstrap intervals, run against a frozen baseline so a change only counts once it is measured honestly. CI on GitHub Actions gates every image before it reaches GHCR.

System architecture: ingest, query, and the evaluation harness, including the refuse and abstain paths
Cited answer with the source chunk
The sources do not cover this, so it says so
Dose questions are refused, no model call

What I built

  • Built a FastAPI service with Pydantic v2 models, SQLAlchemy 2 and Alembic migrations, PostgreSQL with pgvector (HNSW index) plus full text search, Redis, and Celery for background work, backed by ONNX MiniLM embeddings and argon2/JWT auth.
  • Added an abstain path for questions the retrieved sources do not cover and a refuse path for dose or diagnosis questions that never calls the model, keeping the service honest about what it does and does not know.
  • Built an LLM layer with a content hash response cache, a rolling token budget guard, and a rate pacer sitting in front of the real client (generator openai/gpt-oss-120b, judge qwen/qwen3.8-27b, both via Groq); a complete 44 item re-run was served entirely from the cache, 91 model calls, 0 tokens billed.
  • Hash-pinned every prompt, judge prompt, and config file; the evaluation run refuses to start if the corpus fingerprint or a prompt hash has changed since the baseline was frozen.
  • Ran CI on GitHub Actions: tests, a retrieval regression gate, and a Docker image build with a smoke test, pushing the image to GHCR only after all three pass.
  • An independent review of the evaluation harness itself found 8 major defects, including a run where a judge failure was silently counted as a pass, all fixed before any configuration comparison was reported.
  • Raised the per document chunk cap from 2 to 4 for a dev set gain from 15 of 30 to 19 of 30, then ran it once on the held out test split (13 of 20 to 15 of 20) and reported the paired interval honestly: it crosses zero, so the gain is not statistically confirmed. The other 6 experiments tried, hybrid search, a deeper candidate pool, dropping title only chunks, query splitting, and two abstain thresholds, showed no clear gain, and one cost 2 answers.

Tech stack

SWE (AI/MLOps)RAGLLM EvaluationLLM-as-JudgeFastAPIPostgreSQLpgvectorRedisCeleryDockerGitHub ActionsGroqONNX

Key results

15/30 to 19/30

Dev set pass rate

13/20 to 15/20

Held out test pass rate

1,268 passing

Automated tests

0 tokens billed

Cache replay cost

Back to all projects