GenAI Interview Question Check
Section 1 — Production RAG
1. Explain the architecture of a production RAG system.
Interview answer:
A production RAG system typically has five major stages: ingestion, indexing, retrieval, generation, and evaluation/observability.
During ingestion, documents are loaded from sources such as S3, SharePoint, databases, or APIs. We clean and normalize the documents, split them into meaningful chunks, attach metadata, generate embeddings, and store them in a vector database.
At query time, the user's query is optionally rewritten or expanded. We then perform retrieval, usually using hybrid search combining semantic vector search and keyword search. The retrieved documents are passed through a reranker, and the top relevant chunks are sent to the LLM along with the prompt.
Finally, we validate the response, apply guardrails, return the answer, and capture traces and metrics for observability.
A good architecture to draw:
User Query
|
v
Query Processing
|
+----------+----------+
| |
Vector Search Keyword Search
| |
+----------+----------+
|
Reranker
|
Top-K Context
|
v
LLM
|
Guardrails/Validation
|
v
Answer
|
Observability
2. How do you decide chunk size and overlap?
Answer:
I don't choose chunk size purely based on a fixed number like 500 tokens. I consider the document structure, semantic boundaries, embedding model, retrieval performance, and context-window limitations.
For example, for technical documentation, I prefer structure-aware chunking based on headings, paragraphs, sections, and code blocks.
I would start with something like 300–800 tokens and a small overlap, then evaluate retrieval quality using a golden dataset. If important information frequently spans chunk boundaries, I increase overlap or use parent-child retrieval.
Important point
Don't say:
"I always use 500 tokens."
Instead say:
"Chunking is an empirical optimization problem."
3. Dense vs sparse vs hybrid retrieval
Dense retrieval
Uses embeddings.
Query → Embedding → Vector similarity → Documents
Good for semantic similarity.
Sparse retrieval
Usually keyword-based approaches such as BM25.
Query → Keyword matching → Documents
Good for exact terms, IDs, names, codes, etc.
Hybrid
Combines both.
Interview answer:
In enterprise RAG, I generally prefer hybrid retrieval because dense retrieval handles semantic similarity while sparse retrieval handles exact lexical matches. For example, if the user searches for an invoice number or product code, keyword search can be much more reliable than embeddings.
4. What is reranking?
Answer:
Retrieval gives us candidate documents, but the initial ranking isn't always optimal. A reranker takes the query and retrieved candidates and calculates a more accurate relevance score, usually using a cross-encoder or another reranking model.
So I generally think of retrieval as high-recall candidate generation and reranking as high-precision selection.
100,000 documents
↓
Vector/BM25 retrieval
↓
Top 50
↓
Reranker
↓
Top 5
↓
LLM
5. How do you debug retrieval failures?
This is very likely to be asked.
Answer:
I first separate retrieval failure from generation failure.
If the correct document isn't present in the retrieved context, it's a retrieval problem. If the correct document is present but the LLM still gives the wrong answer, it's a generation or prompting problem.
For retrieval failures, I inspect the query, chunking, embeddings, metadata filters, top-K, similarity threshold, vector database configuration, and reranking.
I also use a labeled evaluation dataset and measure metrics such as Recall@K, MRR, and NDCG.
Debugging flow:
Wrong Answer
|
v
Was correct context retrieved?
|
/ \
No Yes
| |
Retrieval Generation
problem problem
6. How do you evaluate RAG?
Break evaluation into retrieval and generation.
Retrieval
Recall@K
Precision@K
MRR
NDCG
Generation
Faithfulness
Answer relevance
Context relevance
Groundedness
Citation correctness
Answer:
I don't evaluate RAG as a single black box. I evaluate retrieval and generation separately.
For retrieval, I measure whether the relevant chunks are actually retrieved. For generation, I evaluate whether the answer is grounded in those chunks, whether it answers the question, and whether it contains unsupported claims.
I maintain a golden dataset containing queries, expected answers, and relevant documents, and run this dataset whenever I change the embedding model, chunking strategy, retriever, prompt, or LLM.
7. Explain Recall@K
Suppose:
Relevant documents = 5
Retrieved top 10 = 10
Relevant documents retrieved = 4
Then:
Recall@10 = 4 / 5 = 80%
Interview answer:
Recall@K tells me how many of the relevant documents were successfully retrieved within the top K results.
8. Explain MRR
MRR focuses on the position of the first relevant result.
If the first relevant document appears at:
Position 1 → 1
Position 2 → 1/2
Position 5 → 1/5
Answer:
MRR is useful when I care about how early the first relevant result appears. It is particularly useful for evaluating ranking quality.
9. How do you detect hallucinations?
Answer:
I use a combination of automated evaluation and production monitoring. The response is compared against the retrieved context to determine whether claims are supported. I can use an LLM-as-a-judge evaluator, rule-based checks, citation validation, and human evaluation for high-risk use cases.
I also design the prompt to explicitly instruct the model to answer only from the provided context and say that it doesn't know when sufficient evidence isn't available.
But add:
LLM-as-a-judge shouldn't be the only evaluation mechanism because the judge itself can make mistakes.
That's a strong production answer.
10. What is query rewriting?
Suppose the user asks:
"What about its pricing?"
The query is ambiguous.
A query rewriting component can convert it to:
"What is the pricing of Product X?"
Answer:
Query rewriting transforms an ambiguous or conversational query into a search-optimized query. It is particularly useful in conversational RAG where the current question depends on previous turns.
11. What is HyDE?
Answer:
HyDE stands for Hypothetical Document Embeddings. Instead of directly embedding the user's question, we first ask an LLM to generate a hypothetical answer or document, embed that generated text, and use it for retrieval.
The intuition is that a hypothetical answer may be semantically closer to the actual documents than the short user query.
User Query
↓
LLM
↓
Hypothetical Answer
↓
Embedding
↓
Vector Search
12. How do you reduce RAG latency?
I would answer systematically:
I first instrument the pipeline and identify where the latency is coming from rather than optimizing blindly.
I measure:
embedding latency
vector DB latency
reranker latency
LLM time-to-first-token
total generation time
external tool latency
Then I optimize accordingly.
Possible optimizations:
caching
smaller embedding models
parallel retrieval
reduce retrieved chunks
reranking only when necessary
prompt compression
streaming
async API calls
connection pooling
model selection
response caching
batching
13. How do you handle document updates?
Answer:
I maintain document-level metadata such as document ID, version, source, timestamp, and chunk IDs.
When a document changes, I identify all chunks associated with that document, remove or invalidate the old embeddings, generate embeddings for the updated chunks, and upsert them.
I also maintain versioning so that stale documents don't remain retrievable.
14. How do you handle a vector DB failure?
Answer:
I treat the vector database as a dependency and design for graceful degradation.
I would use connection timeouts, retries with exponential backoff, circuit breakers, health checks, and appropriate fallbacks.
Depending on the application, we could fall back to keyword search, cached results, or return a controlled response rather than allowing the entire application to fail.
15. How would you design RAG for millions of documents?
Answer:
I would separate ingestion from query serving.
Ingestion would be asynchronous and event-driven. Documents would be processed, chunked, embedded, and indexed through workers.
For serving, I'd use a scalable API layer, horizontally scalable retrieval services, vector database partitioning/indexing, metadata filtering, caching, and potentially hierarchical retrieval.
I'd also monitor ingestion lag, query latency, retrieval quality, vector DB utilization, token consumption, and error rates.
Section 2 — Multi-Agent AI
1. What is an AI agent?
Answer:
An LLM application generally follows a predefined workflow, whereas an agent has the ability to reason about a task, decide what actions or tools are required, execute those actions, observe the results, and continue until the task is completed.
In simple terms, an agent is an LLM combined with tools, state, decision-making, and an execution loop.
Goal
↓
LLM decides
↓
Tool
↓
Observation
↓
LLM decides again
↓
Final answer
2. Why use multiple agents?
Answer:
I use multi-agent architecture when the problem naturally decomposes into specialized responsibilities.
For example, a business analyst system might have separate SQL, data analysis, competitor research, and visualization agents.
Specialization improves modularity, allows independent testing, and makes complex workflows easier to manage. However, I wouldn't automatically use multi-agent architecture because it increases latency, complexity, and cost.
That last sentence is very important.
3. What is an orchestrator agent?
Answer:
The orchestrator is responsible for understanding the user's objective, decomposing the task, selecting the appropriate agents or tools, coordinating their execution, and passing their results to a synthesis or validation stage.
Example:
User:
"Why is ARR decreasing?"
↓
Orchestrator
/ | \
↓ ↓ ↓
SQL Data Competitor
Agent Agent Agent
\ | /
\ | /
↓
Synthesis
↓
Validation
↓
Answer
4. How does the orchestrator decide which agent to call?
Answer:
I wouldn't rely only on free-form LLM reasoning. I would define explicit agent capabilities and routing criteria.
The orchestrator first classifies the task and generates a structured plan. Based on that plan, it selects only the required agents.
For example:
Question → Intent
↓
Required capabilities
↓
Agent selection
↓
Execution plan
You can also use:
structured output
tool schemas
routing rules
confidence thresholds
policy checks
5. How do you prevent infinite agent loops?
Answer:
I use multiple safeguards:
Maximum number of iterations.
Maximum execution time.
Maximum token budget.
State-based termination conditions.
Duplicate tool-call detection.
Retry limits.
Explicit success/failure states.
Example:
if state["iterations"] >= MAX_ITERATIONS:
return "TERMINATE"
In production, I prefer several layers of protection rather than relying on a single condition.
6. How do you handle conflicting agent outputs?
Answer:
I don't blindly concatenate agent responses. I pass structured outputs to a synthesis or validation agent.
Each agent should ideally provide its conclusion along with evidence, confidence, and source information.
If two agents disagree, the synthesis layer should identify the conflict and either resolve it using evidence or explicitly report uncertainty.
Section 3 — LangGraph
1. Why LangGraph?
Answer:
LangGraph is useful when I need explicit control over stateful, multi-step, cyclic workflows.
Instead of treating an agent as a simple loop, I can model the workflow as a graph where nodes perform operations, edges define transitions, and conditional edges control routing.
This makes complex agent workflows easier to debug, persist, retry, and observe.
2. What is a LangGraph node?
A node represents a unit of computation. It could be an LLM call, retrieval operation, SQL execution, validation step, or an entire sub-agent.
3. What is state?
State is the shared information passed through the workflow. It can contain the user query, retrieved documents, intermediate agent outputs, tool results, errors, iteration counts, and final response.
Example:
state = {
"query": "...",
"plan": [],
"documents": [],
"agent_results": [],
"errors": [],
"final_answer": None
}
4. Why not simply use LangChain?
Answer:
LangChain is useful for building LLM components and integrations, but when I have a complex stateful workflow involving branching, loops, retries, human approval, and multiple agents, LangGraph gives me more explicit workflow control.
Section 4 — MCP
1. What is MCP?
Answer:
MCP, or Model Context Protocol, is a standardized protocol for connecting AI applications to external tools, resources, and data sources.
Instead of implementing custom integrations for every agent and every tool, MCP provides a common interface through which an AI application can discover and invoke capabilities.
2. MCP vs function calling
This is a likely question.
Answer:
Function calling is generally a model-level mechanism where the model produces structured arguments for a predefined function.
MCP is an interoperability protocol that standardizes how AI applications discover and interact with external tools and resources.
So function calling can be part of an MCP-based architecture, but MCP addresses the broader integration and interoperability problem.
3. MCP vs A2A
Memorize this distinction:
MCP is primarily about agent-to-tool/resource interaction, whereas A2A is about agent-to-agent communication and collaboration.
Example:
Agent
|
| MCP
↓
Database / API / Search
Agent
|
| A2A
↓
Another Agent
Section 5 — LLM Evaluation
How would you evaluate an LLM application?
Answer:
I evaluate it at multiple levels: component-level, workflow-level, and end-to-end.
For RAG, I evaluate retrieval quality and generation quality separately. For agents, I additionally evaluate tool selection, task completion, planning quality, and execution reliability.
I maintain a fixed evaluation dataset and run regression tests whenever I change the model, prompt, retriever, embeddings, or agent workflow.
Section 6 — Guardrails
How do you prevent prompt injection?
Answer:
I treat retrieved documents and external tool outputs as untrusted data rather than instructions.
I separate system instructions from retrieved content, constrain tool permissions, validate tool arguments, use input and output guardrails, and prevent the model from directly executing high-risk operations.
For sensitive operations, I use deterministic authorization checks and potentially human approval rather than relying solely on the LLM.
Very strong phrase:
"Never trust the LLM to enforce security policy."
The LLM can help decide, but the actual authorization should be enforced by deterministic application code.
Section 7 — Production Debugging
RAG accuracy falls from 90% → 60%. What do you do?
This is one of the most important answers to prepare.
I'd answer:
First, I would avoid assuming that the model became worse. I would compare the production system against the last known-good version and isolate each layer.
I would check:
Has the input/query distribution changed?
Has the document corpus changed?
Did chunking change?
Did the embedding model/version change?
Did vector indexing change?
Did retrieval parameters change?
Did reranking change?
Did the prompt change?
Did the LLM model/version change?
Did latency/timeouts cause incomplete execution?
I would use traces and an evaluation dataset to determine whether the failure is retrieval, generation, infrastructure, or data related.
Then:
90% → 60%
|
+-- Data?
+-- Retrieval?
+-- Embeddings?
+-- Reranker?
+-- Prompt?
+-- Model?
+-- Infrastructure?
+-- Evaluation?
This is the kind of answer that demonstrates production thinking.
Section 8 — Latency Optimization
The application takes 8 seconds. How do you reduce it?
Answer:
First I would establish a latency budget and trace every stage rather than optimizing the entire application blindly.
For example:
Total = 8 sec
Retrieval 0.5 sec
Reranker 1.0 sec
LLM 5.0 sec
Tool calls 1.0 sec
Network 0.5 sec
Now the LLM is clearly the biggest contributor.
Potential solutions:
smaller/faster model
streaming
reduce context
parallel tool calls
caching
prompt optimization
speculative techniques where appropriate
async execution
eliminate unnecessary agent iterations
Section 9 — Cloud / DevOps
How would you deploy a RAG system?
A strong answer:
API Gateway / LB
|
v
FastAPI Pods
/ \
/ \
RAG Service Agent Service
| |
+------+-------+
|
+------------+------------+
| | |
S3 Vector DB Redis
|
Documents
Monitoring:
CloudWatch / OpenTelemetry
Then explain:
I would containerize the application with Docker, deploy it on ECS or Kubernetes depending on the operational requirements, use S3 for document storage, a managed vector database for retrieval, Redis for caching where useful, and centralized logging and tracing for observability.
Section 10 — Behavioral Questions
Tell me about yourself
For this interview, your answer should follow:
Current role/background
↓
GenAI experience
↓
RAG
↓
Agentic AI
↓
Production/backend
↓
Why this role
Don't spend 2 minutes discussing everything you've ever done.
Aim for 60–90 seconds.
"Tell me about your most complex GenAI project."
Use this structure:
Problem → Architecture → Your contribution → Challenges → Solution → Metrics → Production impact
For example:
"The objective was to build an enterprise RAG/agentic system..."
"I designed the orchestration layer..."
"The main challenge was retrieval quality..."
"I introduced hybrid retrieval and reranking..."
"We evaluated using Recall@K and answer-level metrics..."
"We also added tracing and monitoring..."
Do not invent metrics. Use the actual numbers from your projects/resume.