🧠 Context Engineering Quiz (2026)
Test your understanding of the hottest skill in AI Engineering
out of 10 correct
Read the full guide: Top 10 Context Engineering Interview Questions & Answers (2026)
Saturday, June 27, 2026
Context Engineering Quiz (2026) — Test Your Knowledge
Top 10 Context Engineering Interview Questions & Answers (2026)
Top 10 Context Engineering Interview Questions & Answers (2026) — Complete Guide
📋 Table of Contents
Why Context Engineering Is Trending in 2026
Context Engineering has emerged as one of the hottest skills in AI engineering in 2026 — displacing "prompt engineering" as the go-to discipline for building reliable LLM-powered systems.
Where prompt engineering focused on crafting clever one-off instructions, context engineering is about systematically designing, populating, and managing the information an LLM receives at runtime. As models now sport 128k–1M+ token context windows and power production workloads in finance, healthcare, and data platforms, knowing what to put in that window — and how — separates senior AI engineers from juniors.
Key drivers behind the 2026 boom:
- Agentic AI explosion: Autonomous agent frameworks more than doubled in adoption year-over-year. Agents depend entirely on well-crafted context to reason and act correctly.
- RAG is now table stakes: Retrieval-Augmented Generation is the default architecture for enterprise AI. Context engineering is what makes RAG work in production.
- Model commoditization: Teams run multiple models in parallel; context strategy — not model choice — has become the competitive differentiator.
- "Lost in the middle" problem: Research shows LLM accuracy degrades for information buried in the middle of long contexts, making thoughtful context design critical.
Expect context engineering questions in virtually every senior ML/data/AI engineering interview in 2026. Here's what you need to know.
Core Concepts Q&A (Q1–Q4)
What is context engineering, and how does it differ from prompt engineering?
Prompt engineering is the craft of writing effective instructions for an LLM in a single, mostly static prompt. Context engineering is the broader discipline of dynamically constructing the entire input an LLM receives at runtime — including system prompts, retrieved documents, conversation history, tool outputs, and structured data — so the model can reason and act correctly.
In practical terms: a prompt engineer writes better sentences; a context engineer builds the pipeline that decides which sentences, data, and history enter the model's context window, in what order, and how they're formatted. Context engineering is software engineering applied to LLM inputs.
What is the "lost in the middle" problem and how do you address it?
Research (Liu et al., 2023, widely cited into 2026) demonstrated that LLMs perform best when critical information is at the beginning or end of a long context. Information buried in the middle of a large context window is disproportionately ignored, causing retrieval-style tasks to degrade.
Mitigation strategies:
- Position-aware retrieval: Place the most relevant retrieved chunks at the top or bottom of the context, not sandwiched in the middle.
- Reranking: Use a cross-encoder reranker to ensure the most relevant docs land in high-attention positions.
- Context compression: Summarize or prune irrelevant chunks before injection to keep the total token count low.
- Structured separators: Use explicit XML or markdown headers (e.g.,
<context>,<question>) to help the model attend to the right regions.
Explain the anatomy of a well-structured context window for a production RAG system.
A production RAG context window typically follows a layered structure:
- System prompt — role, persona, guardrails, output format instructions.
- Injected knowledge / retrieved docs — chunks from the vector store, ordered by relevance score descending, with source metadata.
- Conversation history — a compressed or windowed version of prior turns (not the raw full history).
- Tool outputs — results from function calls, API responses, code execution outputs.
- Current user message — always at the end for recency bias advantage.
Each layer is separated by explicit delimiters and token-budgeted. The entire payload is assembled by an orchestration layer (LangChain, LlamaIndex, custom pipeline) before the API call.
What is context compression and when would you use it?
Context compression is the process of reducing the number of tokens in a context window without losing the information the model needs to answer the query. Techniques include:
- Extractive compression: A smaller LLM (or a retriever) extracts only the sentences from a retrieved chunk that are relevant to the current query before injecting them.
- Abstractive summarization: Summarize retrieved passages or long conversation histories into concise paragraphs.
- Sliding window history: Keep only the last N turns of conversation, replacing older turns with a summary.
- Selective tool-output pruning: Strip verbose API responses down to only the fields needed for the current reasoning step.
Use compression when your raw inputs would exceed the model's context limit, or when cost and latency are concerns — fewer tokens = lower cost and faster inference.
Advanced Techniques Q&A (Q5–Q8)
What is a "context budget" and how do you implement one in code?
A context budget is a token allocation strategy that sets hard limits for each layer of the context window so the total never exceeds the model's context limit (with a safety margin). For example, with a 128k model:
- System prompt: ≤ 1,000 tokens
- Retrieved docs: ≤ 50,000 tokens
- Conversation history: ≤ 20,000 tokens
- Tool outputs: ≤ 10,000 tokens
- User message + reserved output: ≤ 47,000 tokens
You enforce the budget by counting tokens (via tiktoken for OpenAI or anthropic.count_tokens()) before assembly and truncating/compressing layers that exceed their allocation.
How does context engineering differ for agentic multi-step workflows versus single-turn RAG?
In single-turn RAG, context is assembled once per request — retrieve, inject, generate. In agentic workflows, context is stateful and dynamic:
- The agent accumulates tool results and sub-task outputs across multiple LLM calls in a single session.
- You must manage a scratchpad — an intermediate workspace in the context where the agent stores its reasoning and intermediate state.
- History from prior agent steps must be compressed or summarized to prevent unbounded context growth across a long multi-step task.
- Context must be branched when agents spawn sub-agents, and merged when sub-agents return results to the parent.
Senior engineers design explicit state machines or context managers (e.g., LangGraph's state graph) that control what information persists, what is pruned, and what is summarized at each node in the agent graph.
What is structured context injection and why does it improve model reliability?
Structured context injection means formatting injected information with explicit XML or Markdown tags that help the model parse and attend to different information types. Instead of concatenating raw text, you wrap each piece of context in semantic tags:
<system>You are a data engineering assistant. Answer only from the provided context.</system>
<retrieved_docs>
<doc id="1" source="snowflake_docs">
Snowflake's MERGE statement syntax: ...
</doc>
<doc id="2" source="dbt_docs">
dbt incremental models use ...
</doc>
</retrieved_docs>
<user_query>How do I run an incremental dbt model on Snowflake?</user_query>This improves reliability because models trained on structured data (HTML, XML, code) are better at following schema boundaries. Claude and GPT-4o class models show measurably higher answer accuracy with tagged context versus flat concatenation in internal benchmarks.
How do you evaluate context engineering quality in a production pipeline?
Evaluation is the hardest part of context engineering. Production approaches in 2026:
- Retrieval precision/recall: Measure whether the chunks injected for a query actually contained the answer (using a labeled eval set).
- Context faithfulness: Use an LLM-as-judge to check whether the model's answer is grounded in the injected context (no hallucination).
- Context relevance: Score how relevant injected docs are to the query — irrelevant tokens waste budget and can distract the model.
- Answer correctness: End-to-end correctness on a held-out QA benchmark.
- Token efficiency: Track average tokens-per-correct-answer. A well-compressed context answers correctly with fewer tokens.
Frameworks like RAGAS, TruLens, and Arize Phoenix are commonly used for automated eval pipelines in 2026.
Production & Code Examples (Q9–Q10)
Write a Python function that assembles a token-budgeted context for a RAG call.
Here's a production-style context assembler with per-layer token budgets:
import tiktoken
from dataclasses import dataclass
from typing import List
ENCODER = tiktoken.encoding_for_model("gpt-4o")
def count_tokens(text: str) -> int:
return len(ENCODER.encode(text))
def truncate_to_budget(text: str, budget: int) -> str:
tokens = ENCODER.encode(text)
if len(tokens) <= budget:
return text
return ENCODER.decode(tokens[:budget])
def assemble_rag_context(
system_prompt: str,
retrieved_docs: List[dict], # [{id, source, content}]
history_summary: str,
user_query: str,
budgets: dict = None
) -> str:
"""Assemble a token-budgeted context window for a RAG LLM call."""
budgets = budgets or {
"system": 800,
"docs": 40_000,
"history": 8_000,
"query": 2_000,
}
# --- Layer 1: System prompt (truncate if over budget) ---
system_block = truncate_to_budget(system_prompt, budgets["system"])
# --- Layer 2: Retrieved docs (fill budget greedily by rank) ---
doc_tokens_used = 0
doc_blocks = []
for doc in retrieved_docs:
chunk = (
f'<doc id="{doc["id"]}" source="{doc["source"]}">\n'
f'{doc["content"]}\n</doc>'
)
tokens = count_tokens(chunk)
if doc_tokens_used + tokens > budgets["docs"]:
break
doc_blocks.append(chunk)
doc_tokens_used += tokens
docs_block = f'<retrieved_docs>\n{chr(10).join(doc_blocks)}\n</retrieved_docs>'
# --- Layer 3: Conversation history ---
history_block = (
f'<history>\n{truncate_to_budget(history_summary, budgets["history"])}\n</history>'
)
# --- Layer 4: User query ---
query_block = (
f'<user_query>\n{truncate_to_budget(user_query, budgets["query"])}\n</user_query>'
)
return "\n\n".join([system_block, docs_block, history_block, query_block])
# Example usage
context = assemble_rag_context(
system_prompt="You are a Snowflake SQL expert. Answer only from provided docs.",
retrieved_docs=[
{"id": "1", "source": "snowflake_docs", "content": "MERGE INTO syntax..."},
{"id": "2", "source": "dbt_docs", "content": "Incremental models..."},
],
history_summary="User asked about dbt setup previously.",
user_query="How do I merge records in Snowflake using dbt incremental?"
)
print(f"Total tokens: {count_tokens(context)}")
How do you implement conversation history compression for a long-running agent session?
For agents with many turns, storing the full history quickly blows the context budget. A common production pattern is a rolling summary buffer:
from anthropic import Anthropic
client = Anthropic()
class CompressedHistoryManager:
"""Maintains a compressed summary of conversation history."""
def __init__(self, max_recent_turns: int = 6, summary_token_limit: int = 2000):
self.summary = ""
self.recent_turns = []
self.max_recent_turns = max_recent_turns
self.summary_token_limit = summary_token_limit
def add_turn(self, role: str, content: str):
self.recent_turns.append({"role": role, "content": content})
if len(self.recent_turns) > self.max_recent_turns:
# Compress oldest turns into the rolling summary
to_compress = self.recent_turns[:-self.max_recent_turns]
self.recent_turns = self.recent_turns[-self.max_recent_turns:]
self.summary = self._compress(to_compress)
def _compress(self, turns: list) -> str:
turns_text = "\n".join(
[f'{t["role"].upper()}: {t["content"]}' for t in turns]
)
prompt = (
f"Existing summary:\n{self.summary}\n\n"
f"New turns to integrate:\n{turns_text}\n\n"
"Write a concise updated summary capturing all key decisions, "
"facts, and context. Max 300 words."
)
response = client.messages.create(
model="claude-haiku-4-5",
max_tokens=500,
messages=[{"role": "user", "content": prompt}]
)
return response.content[0].text
def get_context_block(self) -> str:
recent_text = "\n".join(
[f'{t["role"].upper()}: {t["content"]}' for t in self.recent_turns]
)
return (
f"<history_summary>\n{self.summary}\n</history_summary>\n\n"
f"<recent_turns>\n{recent_text}\n</recent_turns>"
)
# Usage
history = CompressedHistoryManager(max_recent_turns=6)
history.add_turn("user", "What is a dbt incremental model?")
history.add_turn("assistant", "A dbt incremental model processes only new rows...")
# ... many more turns ...
print(history.get_context_block())
This approach uses a cheap, fast model (Haiku) for compression and reserves the expensive frontier model for the actual reasoning task — a key cost optimization in production 2026 pipelines.
✅ Key Takeaways
- Context engineering > prompt engineering in 2026. The discipline has matured from tweaking sentences to designing full information pipelines.
- Know the "lost in the middle" problem — position your most critical information at the beginning or end of the context window.
- Always budget your tokens — implement per-layer limits and enforce them programmatically, not as an afterthought.
- Structure beats flat text — XML/Markdown tags around context layers improve model attention and answer reliability.
- Compress aggressively in agent systems — use a cheap model to summarize history; never let unbounded turns eat your context budget.
- Evaluate rigorously — measure context faithfulness, retrieval precision, and token efficiency on labeled eval sets using frameworks like RAGAS or TruLens.
- Context engineering is the skill that turns a demo RAG app into a production-grade AI system. Master it to stand out in 2026 interviews.
Wednesday, June 24, 2026
10 SQL Interview Tips That Senior Engineers Swear By (2026)
SQL Interview Prep
10 SQL Interview Tips That Senior Engineers Swear By (2026)
The difference between passing and failing an SQL interview isn't just knowing the syntax — it's knowing how to think, structure your answers, and avoid the traps most candidates fall into.
The 10 Tips
1Always Think Out Loud Before WritingStrategy
Before touching the keyboard, spend 60 seconds clarifying: "Do I need to handle NULLs? What's the expected output for ties? Should I include/exclude rows with no matches?"
Interviewers give credit for asking the right questions. A query that handles edge cases correctly scores higher than a "perfect" query with silent bugs. Silence is the worst approach.
✅ Script for the First 60 Seconds "Let me understand the expected output first. If a customer has no orders, should they appear in the result? And should I count NULL amounts as zero or exclude them?" — This one habit alone separates senior candidates.
2Know Your NULL Traps ColdCorrectness
NULLs trip up the majority of SQL interview candidates. Remember: any comparison with NULL returns NULL, not TRUE or FALSE.
-- WRONG — will silently exclude NULLs SELECT * FROM users WHERE discount = NULL; -- CORRECT SELECT * FROM users WHERE discount IS NULL; -- WRONG — COUNT(*) counts NULLs, COUNT(col) does NOT SELECT COUNT(discount) FROM orders; -- excludes NULL discounts SELECT COUNT(*) FROM orders; -- counts everything -- NULL-safe aggregation pattern SELECT COALESCE(SUM(discount), 0) AS total_discount FROM orders; -- Returns 0 even if all discounts are NULL
❌ Common Trap
AVG(col) automatically ignores NULLs. If 3 of 10 rows have NULL revenue, AVG(revenue) divides the sum by 7, not 10. If you want a true average treating NULLs as zero: AVG(COALESCE(revenue, 0)).3Understand WHERE vs HAVING — Never Confuse ThemCorrectness
WHERE filters individual rows before aggregation. HAVING filters groups after aggregation.
-- Find departments with average salary > 80k (only active employees) SELECT department, AVG(salary) AS avg_salary FROM employees WHERE status = 'active' -- WHERE: filter rows first GROUP BY department HAVING AVG(salary) > 80000; -- HAVING: filter groups after aggregation -- Performance tip: always push filters to WHERE when possible -- HAVING(some_column = value) is slower than WHERE(some_column = value) -- because HAVING runs after grouping and aggregating all rows
4Know the Execution Order of SQL ClausesPerformance
SQL doesn't execute top-to-bottom. The actual execution order is:
1. FROM + JOINs — identify and join tables 2. WHERE — filter individual rows 3. GROUP BY — group remaining rows 4. HAVING — filter groups 5. SELECT — compute output columns 6. DISTINCT — remove duplicates 7. ORDER BY — sort the result 8. LIMIT/OFFSET — return subset
This is why you cannot reference a SELECT alias in a WHERE clause — WHERE runs before SELECT computes the alias.
❌ Common Trap
SELECT salary * 1.1 AS adjusted, ... WHERE adjusted > 50000 — This FAILS. adjusted doesn't exist yet when WHERE runs. Solution: use a subquery or CTE, or repeat the expression: WHERE salary * 1.1 > 50000.5Use CTEs to Make Complex Queries ReadableStyle
Nothing impresses an interviewer more than clean, readable SQL. CTEs (Common Table Expressions) break complex logic into named, readable steps — and they also signal seniority.
-- Messy nested subquery approach (junior style)
SELECT * FROM (
SELECT customer_id, SUM(amount) AS total
FROM (SELECT * FROM orders WHERE status = 'completed') o
GROUP BY customer_id
) totals WHERE total > 1000;
-- CTE approach (senior style — clearly shows intent)
WITH completed_orders AS (
SELECT customer_id, amount
FROM orders
WHERE status = 'completed'
),
customer_totals AS (
SELECT customer_id, SUM(amount) AS total
FROM completed_orders
GROUP BY customer_id
)
SELECT customer_id, total
FROM customer_totals
WHERE total > 1000;
6Know When to Use Window Functions vs GROUP BYEfficiency
The golden rule: use GROUP BY when you want to collapse rows into one per group. Use Window Functions when you want to add aggregated values to each row without collapsing the result set.
-- "Show each employee and how their salary compares to their department average"
-- GROUP BY CANNOT do this cleanly — it would lose employee details
-- Window function approach:
SELECT
name, department, salary,
AVG(salary) OVER (PARTITION BY department) AS dept_avg,
salary - AVG(salary) OVER (PARTITION BY department) AS diff_from_avg,
RANK() OVER (PARTITION BY department ORDER BY salary DESC) AS dept_rank
FROM employees;
🎯 Interview Signal If a question asks "for each row, show X compared to group average/rank" — that's a window function. Any time the word "per" or "within each" appears alongside keeping all original rows → Window Function.
7Always Mention Index ImplicationsPerformance
For any non-trivial query, add: "In production, I'd ensure there's an index on the join keys and filter columns." This one sentence signals senior-level thinking.
-- This query will be slow on large tables without indexes SELECT o.*, c.name FROM orders o JOIN customers c ON o.customer_id = c.id -- index on customer_id and c.id WHERE o.created_at > '2026-01-01' -- index on created_at AND o.status = 'pending'; -- consider composite index -- In interviews, say: -- "I'd add a composite index on (status, created_at) for this filter -- since both columns appear in the WHERE clause together frequently."
8The "Find Second Highest" PatternClassic Question
A perennial interview question — know 3 ways to solve it.
-- Method 1: Subquery (simple, widely supported)
SELECT MAX(salary) FROM employees
WHERE salary < (SELECT MAX(salary) FROM employees);
-- Method 2: LIMIT/OFFSET (clean, PostgreSQL/MySQL)
SELECT DISTINCT salary FROM employees
ORDER BY salary DESC LIMIT 1 OFFSET 1;
-- Method 3: Window Function (best for Nth highest in general)
WITH ranked AS (
SELECT salary,
DENSE_RANK() OVER (ORDER BY salary DESC) AS rnk
FROM employees
)
SELECT salary FROM ranked WHERE rnk = 2;
-- Change rnk = 2 to any N for the Nth highest
🎯 Pro Tip Use Method 3 in interviews — it shows you know window functions and is easily generalizable to "Nth highest", which the interviewer often follows up with.
9Know the JOIN Types and When Rows Are LostCorrectness
-- INNER JOIN — only rows matching in BOTH tables -- LEFT JOIN — all rows from left, NULLs for unmatched right -- RIGHT JOIN — all rows from right, NULLs for unmatched left -- FULL OUTER — all rows from both, NULLs on non-matching sides -- CROSS JOIN — cartesian product (every combination) -- Interviewer trap: "Find customers with NO orders" -- WRONG — this returns customers WITH orders: SELECT c.* FROM customers c JOIN orders o ON c.id = o.customer_id; -- CORRECT — LEFT JOIN + filter for NULL SELECT c.* FROM customers c LEFT JOIN orders o ON c.id = o.customer_id WHERE o.customer_id IS NULL;
10Practice Explaining Your Query After Writing ItCommunication
After writing any query in an interview, do this: "Let me walk through what this does..." and narrate each clause in plain English.
This shows you understand your own code (not just copied a pattern), catches any bugs before the interviewer does, and demonstrates the communication skills every senior role requires.
✅ The 30-Second Walk-Through Template "First, I join X with Y on the customer ID to connect orders with customer info. Then I filter to only completed orders in the WHERE clause. Then I group by department and use HAVING to keep only groups with more than 100 orders. Finally I order by total revenue descending."
📊 Test Your SQL with 100 Interview Questions
Practice our free interactive SQL quiz — 100 real interview questions with instant colour-coded feedback and full explanations.
Take the SQL Quiz → Get the PDF BundleParquet vs ORC vs Avro: Big Data File Formats Interview Guide (2026)
Big Data Storage
Parquet vs ORC vs Avro: Big Data File Formats Interview Guide (2026)
One of the most common interview topics for data engineers. Know when to use each format, the trade-offs, and how to choose for any scenario.
📋 Formats Covered
Choosing the wrong file format can mean 10x slower queries and 5x higher storage costs. Interviewers ask this to see if you understand the fundamentals of how data is physically stored.
Parquet
Columnar Storage — The Default Choice for Analytics
Spark · Athena · BigQuery · Presto
Parquet stores data column by column rather than row by row. When you run SELECT revenue, date FROM sales on a 500-column table, Parquet reads only those 2 column files — ignoring all other 498 columns entirely. This makes analytical queries dramatically faster.
Key features:
• Predicate pushdown — Stores min/max statistics per row group. Spark can skip entire chunks without reading them based on your WHERE clause.
• Dictionary encoding — Repeated values (like country codes, status flags) are stored as integer lookups, not repeated strings.
• Nested data support — Natively handles arrays and nested structs (Dremel encoding).
• Schema embedded in footer — File is self-describing.
# Writing Parquet in PySpark
df.write.mode('overwrite') \
.partitionBy('year', 'month') \
.parquet('s3://my-bucket/sales/')
# Reading with column pruning (only reads 2 columns from disk)
spark.read.parquet('s3://my-bucket/sales/') \
.select('revenue', 'date') \
.filter('year = 2026')
✅ Use Parquet When
- Analytical workloads (OLAP)
- Reading a few columns from wide tables
- Spark, Presto, Athena, BigQuery
- Storing data lake tables
❌ Avoid Parquet When
- Streaming (Kafka) — row-by-row writes are expensive
- Frequent small appends
- Schema changes very often
ORC
Optimized Row Columnar — Hive's Preferred Format
Hive · Spark · Impala
ORC (Optimized Row Columnar) is also a columnar format, similar to Parquet. It was designed specifically for Hive and offers slightly better compression ratios and faster Hive queries, but Parquet is more universally supported across tools.
ORC vs Parquet key differences:
• ORC uses ACID transactions natively — supports UPDATE and DELETE in Hive (Parquet doesn't in Hive).
• ORC stores Bloom filters by default — faster equality lookups.
• Parquet has better ecosystem support — Spark, Arrow, Pandas, BigQuery all prefer Parquet.
• ORC has better Hive performance — if your stack is Hive-heavy, ORC is the native choice.
⚠️ Interview Answer "Which is better, Parquet or ORC?" — Answer: "It depends on the ecosystem. Pure Spark/cloud → Parquet. Hive-heavy or need ACID transactions → ORC." Interviewers don't want a "one size fits all" answer.
Avro
Row-Based with Schema — The Streaming Champion
Kafka · Schema Registry · Event Streaming
Avro is a row-based serialization format with an embedded JSON schema. It's the dominant format for Kafka messages because it writes one record at a time efficiently and supports robust schema evolution.
Key features:
• Schema evolution — Forward and backward compatible changes (add optional fields, remove fields with defaults).
• Schema Registry — In Kafka pipelines, Avro schemas are registered in Confluent Schema Registry so all producers/consumers share the same schema.
• Compact binary encoding — No field names repeated in each record (stored once in schema), making messages small.
• Row storage — Writing one record at a time is efficient (unlike Parquet which needs to buffer column data).
# Avro schema example (JSON format)
{
"type": "record",
"name": "OrderEvent",
"fields": [
{"name": "order_id", "type": "string"},
{"name": "amount", "type": "double"},
{"name": "status", "type": "string"},
{"name": "discount", "type": ["null", "double"], "default": null} ← optional field
]
}
✅ Use Avro When
- Kafka event streaming
- Schema evolution needed
- Write-heavy workloads
- Serializing Java/Python objects
❌ Avoid Avro When
- Analytical queries on specific columns
- Storing in a data lake for analytics
- Need human-readable format
Quick Comparison Table
| Feature | Parquet | ORC | Avro | CSV |
|---|---|---|---|---|
| Storage type | Columnar | Columnar | Row | Row |
| Best for | Analytics (OLAP) | Hive / ACID | Streaming | Interchange only |
| Compression | Excellent | Best of all | Good | None built-in |
| Schema evolution | Limited | Limited | Excellent | None |
| Splittable | ✅ | ✅ | ✅ (with sync marks) | ✅ (if uncompressed) |
| Self-describing | ✅ (footer) | ✅ (footer) | ✅ (header) | ❌ |
| Human readable | ❌ | ❌ | ❌ | ✅ |
| Kafka use | ❌ Poor | ❌ Poor | ✅ Ideal | ⚠️ Possible but inefficient |
Scenario-Based Questions
Scenario 1
You're building a data lake on S3 for Athena. What format do you choose?
Answer: Parquet with Snappy compression, partitioned by date.
Athena is a columnar query engine — Parquet aligns perfectly. Partition by year/month/day so Athena can skip entire partitions for date-range queries. Snappy compression for a balance of speed and size. Expected cost savings vs CSV: 70–90% reduction in bytes scanned = direct cost reduction since Athena charges per TB scanned.
Scenario 2
Your team uses Kafka for order events and the schema changes frequently. What format for Kafka messages?
Answer: Avro with Confluent Schema Registry.
Avro's forward/backward compatible schema evolution means producers can add new optional fields without breaking existing consumers. Schema Registry enforces schema compatibility rules (BACKWARD, FORWARD, FULL) centrally — preventing incompatible schema changes from reaching production.
🗂️ Master More Big Data Interview Topics
Practice free interactive quizzes on Hadoop, Spark, SQL, Networking and DSA — and get the 300Q PDF bundle for your deepest prep.
Visit the Blog → Get the PDF BundleApache Airflow Interview Questions and Answers (2026)
Orchestration Interview Prep
Apache Airflow Interview Questions & Answers (2026)
The most-asked Airflow questions for data engineering roles — covering DAG design, scheduling, operators, sensors, XComs, and real-world workflow patterns.
📋 Topics Covered
1. DAG & Core Concepts
Q1What is a DAG in Airflow? What makes it "acyclic"?
A DAG (Directed Acyclic Graph) is the core object in Airflow — a Python file that defines a workflow as a collection of tasks with dependencies.
Directed — Tasks have a defined order (task A must finish before task B).
Acyclic — No circular dependencies. Task A cannot eventually depend on itself. Airflow will reject a DAG with cycles.
Directed — Tasks have a defined order (task A must finish before task B).
Acyclic — No circular dependencies. Task A cannot eventually depend on itself. Airflow will reject a DAG with cycles.
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime
with DAG(
dag_id='daily_etl_pipeline',
start_date=datetime(2026, 1, 1),
schedule_interval='0 6 * * *', # every day at 6am
catchup=False,
tags=['production', 'etl']
) as dag:
extract = PythonOperator(task_id='extract', python_callable=extract_data)
transform = PythonOperator(task_id='transform', python_callable=transform_data)
load = PythonOperator(task_id='load', python_callable=load_data)
extract >> transform >> load # dependency chain
Q2What is the Airflow Executor and what are the types?
The Executor determines how and where tasks are run.
Production recommendation: KubernetesExecutor (Airflow 2.x on GKE/EKS) or CeleryExecutor for on-prem/VM setups.
| Executor | Best For | Notes |
|---|---|---|
| SequentialExecutor | Local dev only | One task at a time, SQLite backend |
| LocalExecutor | Small teams | Parallel tasks on one machine via subprocesses |
| CeleryExecutor | Multi-worker production | Distributed workers via Celery + Redis/RabbitMQ |
| KubernetesExecutor | Cloud-native | Each task runs in its own pod — scales to zero |
2. Operators & Sensors
Q3What is the difference between an Operator and a Sensor?
Operator — Defines what a task does: run Python code, submit a Spark job, call an API, execute SQL.
Sensor — A special operator that waits for a condition to be met (polls at intervals). Blocks downstream tasks until the condition is satisfied.
Sensor — A special operator that waits for a condition to be met (polls at intervals). Blocks downstream tasks until the condition is satisfied.
# Operator example — does work
from airflow.operators.python import PythonOperator
run_transform = PythonOperator(task_id='transform', python_callable=run_transform_fn)
# Sensor example — waits for a condition
from airflow.sensors.s3_key_sensor import S3KeySensor
wait_for_file = S3KeySensor(
task_id='wait_for_input_file',
bucket_name='my-data-lake',
bucket_key='input/{{ ds }}/data.csv', # uses Jinja templating
aws_conn_id='aws_default',
poke_interval=300, # check every 5 minutes
timeout=3600 # fail after 1 hour
)
wait_for_file >> run_transform # only transform after file arrives
⚠️ Sensor Mode Default sensors use "poke" mode which occupies a worker slot while waiting. Use
mode='reschedule' to free the worker between checks — critical for long-wait sensors in production.Q4What are the most commonly used Airflow operators?
| Operator | Use Case |
|---|---|
PythonOperator | Run any Python function |
BashOperator | Run shell commands |
SparkSubmitOperator | Submit a Spark job to a cluster |
PostgresOperator / BigQueryOperator | Execute SQL queries |
S3ToGCSOperator | Cross-cloud data transfer |
DummyOperator | Placeholder for branching/grouping |
BranchPythonOperator | Conditional path selection |
TriggerDagRunOperator | Trigger another DAG |
✅ Modern Approach In Airflow 2.x, prefer the
@task decorator (TaskFlow API) over PythonOperator. It's cleaner, handles XComs automatically, and reduces boilerplate significantly.3. XComs & Task Communication
Q5What are XComs and what are their limitations?
XComs (Cross-Communications) allow tasks to pass small values to each other through Airflow's metadata database.
• Stored in Airflow's metadata DB (SQLite/MySQL/PostgreSQL) — NOT designed for large data.
• Maximum practical size: a few KB. Passing a DataFrame via XCom will crash your metadata DB.
• Correct pattern: save large data to S3/GCS/database and pass only the path/ID via XCom.
# TaskFlow API — XComs happen automatically
from airflow.decorators import dag, task
@dag(schedule_interval='@daily', start_date=datetime(2026,1,1), catchup=False)
def etl_pipeline():
@task
def extract():
return {"record_count": 5000, "source_file": "s3://bucket/data.csv"}
@task
def transform(extracted_data):
count = extracted_data["record_count"]
print(f"Transforming {count} records")
return count * 2
@task
def load(transformed_count):
print(f"Loading {transformed_count} records")
data = extract()
count = transform(data)
load(count)
etl_pipeline()
Limitations (critical to know for interviews):• Stored in Airflow's metadata DB (SQLite/MySQL/PostgreSQL) — NOT designed for large data.
• Maximum practical size: a few KB. Passing a DataFrame via XCom will crash your metadata DB.
• Correct pattern: save large data to S3/GCS/database and pass only the path/ID via XCom.
4. Scheduling & Execution
Q6What is catchup in Airflow and when would you disable it?
catchup=True (default) — When a DAG is turned on, Airflow will backfill and run all missed DAG runs between
catchup=False — Only run from the current schedule interval forward. Use this for real-time pipelines where processing old dates makes no sense (e.g., a daily report DAG).
Rule: Set
start_date and today. Useful for historical data ingestion.catchup=False — Only run from the current schedule interval forward. Use this for real-time pipelines where processing old dates makes no sense (e.g., a daily report DAG).
Rule: Set
catchup=False for 90% of pipelines unless you specifically need backfilling.
Q7What does
execution_date mean in Airflow? Why is it confusing?execution_date (Airflow 1.x) / data_interval_start (Airflow 2.x) is the start of the scheduling interval, NOT the actual time the task ran.For a DAG with
schedule_interval='@daily':• The run for
2026-04-01 (execution_date = 2026-04-01 00:00:00) will actually trigger on 2026-04-02 (after the interval closes).This "off by one" behavior trips up many engineers. Always use
{{ ds }} (the date string of the execution) in your SQL/file paths to reference the correct data date.
5. Production Best Practices
Q8How would you handle a DAG that runs for too long or has too many retries?
with DAG(
dag_id='monitored_pipeline',
dagrun_timeout=timedelta(hours=4), # fail entire DAG after 4 hours
default_args={
'retries': 2,
'retry_delay': timedelta(minutes=5),
'retry_exponential_backoff': True, # 5m, 10m, 20m delays
'execution_timeout': timedelta(hours=1), # per-task timeout
'on_failure_callback': alert_slack, # notify on failure
'sla': timedelta(hours=2), # SLA miss alert
}
) as dag:
...
Additional strategies:• Break large monolithic DAGs into smaller sub-DAGs or modular DAGs connected via
TriggerDagRunOperator.• Use
max_active_runs=1 to prevent concurrent runs from overlapping.• Set
depends_on_past=True for tasks that must not run until the previous day's run succeeded.
🎯 Master More Data Engineering Topics
Practice free quizzes on SQL, Spark, PySpark, Hadoop and Networking — and get the 300Q PDF bundle for focused offline prep.
Visit the Blog → Get the PDF BundlePython for Data Engineers: Top Interview Questions (2026)
Python Interview Prep
Python for Data Engineers: Top Interview Questions (2026)
The Python questions you'll actually face in data engineering interviews — from core language features to real-world pipeline patterns with pandas and file processing.
📋 Topics Covered
1. Core Python Features
Q1What is the difference between a list comprehension and a generator expression?
List comprehension — Creates all values immediately and stores them in memory as a list:
Generator expression — Lazy evaluation:
[x*2 for x in range(1000000)] → allocates a million integers in RAM right now.Generator expression — Lazy evaluation:
(x*2 for x in range(1000000)) → produces values one at a time, using almost no memory.# Memory comparison
import sys
lst = [x*2 for x in range(100000)]
gen = (x*2 for x in range(100000))
print(sys.getsizeof(lst)) # ~800,056 bytes
print(sys.getsizeof(gen)) # ~104 bytes — fixed size!
# Use generator for large file processing
def read_large_file(filepath):
with open(filepath) as f:
for line in f:
yield line.strip() # processes one line at a time
Rule of thumb: If you're iterating through the result once and the dataset is large → use a generator. If you need random access or multiple passes → use a list.
Q2What are Python decorators? Give a data engineering use case.
A decorator wraps a function to add behavior without modifying the original code. The syntax
@my_decorator is syntactic sugar for func = my_decorator(func).
import time
import functools
# Decorator to log execution time of pipeline steps
def timer(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
start = time.time()
result = func(*args, **kwargs)
elapsed = time.time() - start
print(f"{func.__name__} completed in {elapsed:.2f}s")
return result
return wrapper
# Retry decorator for flaky API calls
def retry(max_attempts=3):
def decorator(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
for attempt in range(max_attempts):
try:
return func(*args, **kwargs)
except Exception as e:
if attempt == max_attempts - 1: raise
print(f"Attempt {attempt+1} failed: {e}. Retrying...")
return wrapper
return decorator
@timer
@retry(max_attempts=3)
def fetch_api_data(endpoint):
# your API call here
pass
Q3What is the difference between *args and **kwargs?
*args collects extra positional arguments into a tuple. **kwargs collects extra keyword arguments into a dictionary.def pipeline_config(*sources, **options):
# sources = ('s3://bucket/a', 's3://bucket/b') ← tuple
# options = {'partition_by': 'date', 'mode': 'append'} ← dict
for source in sources:
print(f"Processing {source} with options {options}")
pipeline_config('s3://a', 's3://b', partition_by='date', mode='append')
Common DE use: Building flexible pipeline runner functions that accept variable numbers of data sources and arbitrary configuration options.
Q4Explain Python's GIL and how it affects data engineering workloads.
The Global Interpreter Lock (GIL) is a mutex in CPython that prevents multiple native threads from executing Python bytecode simultaneously. This means Python threads cannot achieve true parallelism for CPU-bound tasks.
Impact on data engineering:
• I/O-bound tasks (reading files, HTTP calls, DB queries) — GIL is released during I/O waits, so
• CPU-bound tasks (data transformation, computation) — GIL blocks parallelism. Use
• PySpark bypasses the GIL by distributing work to the JVM layer via the Py4J bridge.
Impact on data engineering:
• I/O-bound tasks (reading files, HTTP calls, DB queries) — GIL is released during I/O waits, so
threading works well for these.• CPU-bound tasks (data transformation, computation) — GIL blocks parallelism. Use
multiprocessing instead (separate processes, each with their own GIL).• PySpark bypasses the GIL by distributing work to the JVM layer via the Py4J bridge.
2. Data Processing Patterns
Q5How do you read a large CSV file in Python without loading it all into memory?
import pandas as pd
import csv
# Method 1: pandas chunking
chunk_size = 100_000
for chunk in pd.read_csv('large_file.csv', chunksize=chunk_size):
# process each 100k-row chunk
processed = chunk[chunk['status'] == 'active']
processed.to_sql('target_table', engine, if_exists='append', index=False)
# Method 2: pure Python generator (no pandas memory overhead)
def csv_row_generator(filepath):
with open(filepath, newline='') as f:
reader = csv.DictReader(f)
for row in reader:
yield row
for row in csv_row_generator('large_file.csv'):
process_row(row)
🎯 Interview Signal Always mention chunking or generators when asked about large file processing. Never say "I'd load it into a DataFrame" without qualifying the file size.
Q6What is the difference between apply(), map(), and applymap() in pandas?
Series.map(func) — Element-wise on a single Series. Fast for simple transformations.DataFrame.apply(func, axis=0/1) — Applies a function along an axis (rows or columns). Used for row/column-level aggregations.DataFrame.applymap(func) / DataFrame.map(func) (pandas 2.1+) — Element-wise on the entire DataFrame.# map — element-wise on one column
df['status_clean'] = df['status'].map({'Y': True, 'N': False})
# apply — custom function per row
df['full_name'] = df.apply(lambda r: f"{r['first']} {r['last']}", axis=1)
# vectorized (always prefer this over apply when possible)
df['revenue_usd'] = df['revenue_eur'] * 1.08 # 10-100x faster than apply
✅ Performance Tip Avoid apply() in hot paths — it's Python-level iteration. Use NumPy or pandas vectorized operations whenever possible.
3. Error Handling & Logging
Q7How do you handle errors gracefully in a data pipeline?
import logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s %(levelname)s %(message)s')
logger = logging.getLogger(__name__)
def process_record(record):
try:
# transformation logic
return transform(record)
except KeyError as e:
logger.warning(f"Missing field {e} in record {record.get('id')}")
return None # skip bad record, don't crash the pipeline
except ValueError as e:
logger.error(f"Data type error: {e} for record {record.get('id')}")
raise # re-raise critical errors
# Dead-letter queue pattern for pipelines
good_records, bad_records = [], []
for record in incoming_records:
result = process_record(record)
if result is None:
bad_records.append(record)
else:
good_records.append(result)
# Write failed records to a dead-letter file for investigation
if bad_records:
pd.DataFrame(bad_records).to_csv('dead_letter.csv', index=False)
logger.warning(f"{len(bad_records)} records written to dead_letter.csv")
4. Performance & Memory
Q8How do you profile and optimize a slow Python data processing script?
import cProfile
import pstats
from io import StringIO
# Profile the script
profiler = cProfile.Profile()
profiler.enable()
run_my_pipeline() # your function here
profiler.disable()
stream = StringIO()
stats = pstats.Stats(profiler, stream=stream).sort_stats('cumulative')
stats.print_stats(20) # top 20 slowest functions
print(stream.getvalue())
After profiling, common optimizations in order of impact:1. Replace
apply() with vectorized pandas/NumPy operations — 10–100x speedup.2. Use appropriate dtypes —
category for low-cardinality strings, int32 instead of int64 when safe.3. Filter early — reduce your DataFrame size before joins/aggregations.
4. Use
multiprocessing.Pool for CPU-bound parallel processing.5. Switch to Polars (Rust-based DataFrame library) for single-machine workloads that outgrow pandas.
Q9What is the difference between is and == in Python? Why does it matter for data pipelines?
== checks value equality (are the values the same?)is checks identity (are they the same object in memory?)Why it matters in pipelines:
# DANGEROUS - common bug
def is_null(val):
return val == None # Wrong! Can be overridden by custom __eq__
# CORRECT
def is_null(val):
return val is None # Checks identity — always reliable
# In pandas, this is why you use pd.isna() / pd.isnull()
# Never: df[df['col'] == None]
# Always: df[df['col'].isna()]
🐍 Ready to Test Your Python Skills?
Take our free interactive quizzes on SQL, Spark, PySpark, Hadoop and Networking — and grab the 300Q PDF bundle for offline prep.
Visit the Blog → Get the PDF Bundle
Subscribe to:
Posts (Atom)