Most RAG write-ups stop at "it retrieves some chunks and an LLM answers." That's the easy 80%. The harder, more interesting question — the one that actually separates a demo from an engineered system — is: when you change the chunk size, the top-k, or the retrieval strategy, how do you know it helped? Not "the answers still look fine when I try three questions," but an actual measurement you'd trust enough to ship.
That's what I built on top of a customer-support chatbot over policy PDFs: a small evaluation harness that turns "does this change help?" from a guess into a keep-or-revert decision.
The pipeline, briefly
Nothing exotic: PDFs are chunked, embedded, and indexed into FAISS; a query retrieves the top-k chunks by similarity; those chunks plus the question go to a local LLM (Ollama running Gemma3) for the final answer. Streamlit sits on top as the chat UI.
The interesting part isn't this pipeline — it's what sits next to it.
A benchmark, not a vibe check
The harness runs against 31 questions drawn from the actual policy documents the bot answers over: 28 answerable questions, each with a verified source document and an acceptable-answer key, plus 3 deliberately unanswerable ones — questions the documents simply don't cover — to check whether the system knows to abstain instead of confidently making something up.
That last part matters as much as the 28. A RAG system that never abstains isn't accurate, it's just quiet about being wrong.
Metrics, in order of how much they cost to compute
fact_recall is the primary metric, and it's deliberately boring: does the expected answer text actually show up in the retrieved context window? It's deterministic, needs no LLM call to check, and runs in milliseconds — which means it's cheap enough to run on every single candidate change.
context_chars is the tie-breaker. Two configurations that answer equally well are not actually equal — the one using half the context is cheaper to run, every single time, at whatever scale you eventually operate at. Recall tells you if it works; context size tells you what it costs.
Only past that, with --mode full, does the harness spend the money on LLM-judged metrics: answer_accuracy, citation_rate, abstain_accuracy, and mean_latency_s. These are slower and noisier — an LLM judging an LLM's output isn't deterministic — so they're a deeper check, not something you run on every iteration.
The experiment loop
eval/config.py is the only file an experiment is allowed to touch — it holds BASELINE, the shipped, known-good settings, plus whatever candidate configuration is being tried. eval/loop.py runs that candidate against the benchmark in eval/benchmark.yaml, logs the full result to results.jsonl, and the candidate only replaces the baseline if it doesn't regress the metrics that matter. Otherwise, it's reverted.
That's the whole point: it turns "I tweaked the retrieval and it seems fine" into a logged, comparable, keep-or-revert decision — the same discipline as a feature flag with an automatic rollback, applied to retrieval configuration instead of production traffic.
What the harness actually caught
The shipped configuration retrieved the top 8 chunks per question. Running top-4 through the harness instead produced identical 100% fact_recall and a 0.9405 mean reciprocal rank — while cutting context to 1,339 characters, roughly half the baseline.
Same answer quality. Half the tokens shipped to the LLM on every single question. That's a real latency and cost win, and it's exactly the kind of change that looks completely invisible to eyeballing outputs — both configurations "look right" to a human reading the chat transcript. The only way to catch it is to measure it.
Why this is the actual engineering
Prompt tweaking by feel doesn't scale past a demo. Treating retrieval configuration as something with a benchmark, a baseline, and a revert path is what makes a RAG system an engineered system instead of a collection of settings someone got to feel good about once. That's the mindset I think matters more than any specific framework choice: AI-powered systems still need the same evaluation discipline as everything else in production — a way to know, not guess, whether a change made things better.
The full project, including the eval harness, is on GitHub.