RAG Explained in 3 Levels (The Version That Gets You the Job)

RAG is the single most common system-design question in AI engineering interviews right now, and most candidates only know the first level. They can say what the letters stand for, then stall the moment someone asks how it actually works or where it falls over.

This is the three-level version I use. Level 1 is the definition. Level 2 is the mechanism, the part that separates people who have built one from people who have read a blog post. Level 3 is where it breaks, which is the level that actually gets you hired, because it proves you have run one in production and watched it fail.

Level 1: What RAG is

RAG stands for Retrieval Augmented Generation. Instead of answering from what it memorised during training or whatever happens to be in its context window, the model looks the answer up from an external database first, then writes its response from what it found.

The mental model is an open book exam versus a closed book one. A plain LLM is sitting the exam from memory. It is confident, fast, and will happily make things up when it does not know. RAG hands the model the textbook and tells it to answer from the page in front of it. Same student, very different failure mode.

That is the whole reason RAG exists: it grounds the model in real, current, private data it was never trained on, so answers can cite something instead of guessing.

Level 2: How it actually works

This is the part interviewers probe, because it is where the fakers drop out. RAG has two phases: one that runs once when you load your documents, and one that runs on every question.

Indexing (runs once, ahead of time):

Chunk. Split each document into smaller pieces, maybe a few paragraphs each. You retrieve chunks, not whole files.
Embed. Run every chunk through an embedding model, which turns the text into a vector, a list of numbers that captures its meaning. Chunks about similar things end up close together in that vector space.
Store. Save those vectors in a vector database like ChromaDB, Pinecone, or pgvector, each one linked back to the original text.

Retrieval (runs on every query):

Embed the question with the exact same embedding model. This part matters: the query and the chunks have to live in the same vector space, or the distances mean nothing.
Find the nearest chunks. The database returns the top few chunks closest to the question vector (the demo uses the two nearest). Closest in vector space means most similar in meaning.
Inject. Paste those chunks into the prompt as context, then ask the model to answer using them. Now it is answering from retrieved facts instead of guessing from memory.

If you can walk through those two phases cleanly, you are already ahead of most candidates. But the level that actually lands the offer is the next one.

Level 3: Where RAG breaks

Anyone can describe the happy path. The signal an interviewer is listening for is whether you know the failure modes, because that only comes from running one.

Chunk size is a tradeoff with no perfect answer. Chunks too small and you slice a single idea in half, so no retrieved piece carries the full context. Chunks too big and each one covers several topics, so the embedding blurs into a vague average and retrieval gets less precise. Tuning chunk size for your specific documents is most of the real work.

Retrieval can return garbage, silently. If the right chunk was never retrieved, the model does not know that. It cannot tell the difference between good context and irrelevant context. It will take whatever you injected and answer over it with total confidence. This is the failure that surprises people, because nothing errors out. You just get a wrong answer that looks certain.

The hallucination question (the one that gets you the job)

When an interviewer asks how you would handle hallucinations in a RAG system, do not say "better prompts" or "a bigger model." Say this:

"First I check whether it failed at retrieval or at generation."

That one sentence tells them you understand RAG has two independent places to fail. Retrieval failure means the right chunk never made it into the prompt, so you fix the index: chunk size, the embedding model, how many chunks you pull, or adding a reranker. Generation failure means the correct context was right there and the model still ignored it or made something up, so you fix the prompt, the instructions, or the model. Same symptom, two completely different fixes, and knowing which one you are looking at is the whole skill.

10 follow-up questions interviewers ask about RAG

Once you handle the three levels, this is where the conversation usually goes. Short answers you can build on:

What is an embedding, really? A vector that places a piece of text in a space where distance means semantic similarity. Similar meaning, nearby vectors.
Why not just stuff everything into the context window? Cost, latency, and the model losing focus in long context. Retrieval sends only the relevant slice, so it scales to millions of documents a context window could never hold.
How do you pick chunk size? You test it on your own data. Start around a few hundred tokens with some overlap so ideas are not cut at the boundary, then measure retrieval quality and adjust.
What is a reranker? A second, slower model that re-scores the top retrieved chunks for actual relevance. You retrieve a wide net cheaply, then rerank to keep only the best few before injecting.
What is chunk overlap and why use it? Repeating a bit of text between neighbouring chunks so a sentence split across a boundary still shows up whole in at least one chunk.
How do you evaluate a RAG system? Split it in two. Retrieval metrics (did the right chunks come back) and generation metrics (was the final answer faithful to them). Measuring end to end only hides which half is broken.
What is hybrid search? Combining vector similarity with old-fashioned keyword search. Keywords catch exact terms, product codes, and names that embeddings blur; vectors catch meaning. Together they beat either alone.
When should you not use RAG? When the knowledge is small and static enough to fit in the prompt, or when the task needs reasoning rather than lookup. RAG adds moving parts, so only pay for it when you need fresh or private facts.
RAG or fine-tuning? RAG teaches the model new facts it can cite and update instantly. Fine-tuning teaches it new behaviour or style. Facts change, so most knowledge problems want RAG, not a retrain.
How do you stop it answering when retrieval finds nothing good? Threshold on the similarity score, and instruct the model to say it does not know when the top chunks are below it. A RAG system that admits it has no answer beats one that invents a confident wrong one.