"The answers are bad. Let's try GPT-4 instead of GPT-3.5."
I've heard a version of this sentence in almost every RAG postmortem I've sat in. It's always the first move, and it's almost always wrong.
Here's the test I run before anyone touches the model config: print the retrieved chunks. Not the answer — the raw context the model was given before it generated anything. Nine times out of ten, the chunks are irrelevant, truncated mid-sentence, or answering a slightly different question than the one asked. The model didn't fail. It did a competent job reasoning over garbage.
The instinct is backwards, and it's understandable why
RAG systems fail silently at the retrieval layer. A bad retrieval doesn't throw an error — it just returns something, and the LLM, being a good sport about it, generates a fluent, confident-sounding answer anyway. The failure is invisible unless you go looking for it, while the output — a wrong or vague answer — looks exactly like what you'd expect from "the model isn't smart enough."
So the fix people reach for is upgrading the model. It feels like progress. It's also treating a plumbing problem with a better faucet.
What's usually actually wrong
Chunks that don't align with how questions get asked. If you chunk a document into fixed 500-token blocks with no regard for section boundaries, you'll routinely split the answer to a question across two chunks — half the fact in one, half in the other — and retrieve only one of them.
Embeddings that measure the wrong kind of similarity. A query like "how do I cancel my subscription" and a doc chunk containing "cancellation is not available after the trial period" are semantically close in embedding space — but retrieving the second doesn't help answer the first. Vector similarity finds topically related text, not necessarily answer-bearing text.
Top-k that's too small or too generic. Retrieving the top 3 chunks with no reranking step means you're betting the entire answer on cosine similarity being a perfect proxy for relevance. It isn't. It's a decent first pass.
