The demo version of our RAG system was great. Clean docs, a handful of test questions we'd hand-picked because we knew the answers were retrievable, everyone nodding in the review meeting. Three weeks after launch, support started flagging answers that were confidently wrong, and it took me longer than I'd like to admit to figure out why, because nothing had obviously changed.

The chunking decision you don't get to make once

We'd fixed chunk size early — 512 tokens, some overlap, seemed reasonable, moved on. What actually happened is that as new docs got added by different teams with different writing styles, some docs had dense technical tables that got sliced mid-table, and the retrieved chunk would contain half a table with no header row. The model would then confidently make up what the missing columns probably meant. The chunking strategy that worked for our original doc set silently stopped working for a new doc format, and nothing in our monitoring caught it because the pipeline wasn't erroring — it was just wrong.

The fix wasn't a smarter chunker, it was chunking that respects document structure — splitting on headings and table boundaries instead of a fixed token count, and attaching the nearest heading as context to every chunk even when it falls outside the chunk boundary. More engineering effort than the naive version, and it should have been the default from day one.

Embedding drift is real and nobody tells you when it happens

We updated our embedding model for a routine cost-optimization pass — newer model, cheaper, similar benchmark numbers. Retrieval quality on our internal eval set dropped by a few points and I almost didn't notice, because a few points on an aggregate score doesn't look alarming. What it actually meant was that a specific cluster of technical questions, the ones with heavy jargon overlap between similar-sounding-but-different answers, got noticeably worse. Aggregate metrics hid a real regression in a subset of queries that mattered a lot to specific users.

An aggregate retrieval score going from 0.87 to 0.84 sounds like noise. It can also mean one entire category of question just broke.

What I'd do differently starting over

  • Build a small eval set from real user queries, not hand-picked demo questions, before you ship — and keep adding to it from actual production misses.
  • Segment your eval by query type, not just an aggregate score, so a regression in one category doesn't hide inside a healthy-looking average.
  • Re-run the full eval suite on every embedding model change, every chunking change, and treat a swap as a migration, not a config tweak.

None of this is exotic advice. It's the boring stuff that's easy to skip when the demo already works and there's pressure to ship. The cost of skipping it doesn't show up until someone downstream trusts a wrong answer.