Python Interview Prep

Python for Data Engineers: Top Interview Questions (2026)

The Python questions you'll actually face in data engineering interviews — from core language features to real-world pipeline patterns with pandas and file processing.

📅 Updated April 2026  |  ⏱ 11 min read  |  🎯 Beginner to Senior

📋 Topics Covered

1. Core Python Features
2. Data Processing Patterns
3. Error Handling & Logging
4. Performance & Memory

1. Core Python Features

Q1What is the difference between a list comprehension and a generator expression?

List comprehension — Creates all values immediately and stores them in memory as a list: [x*2 for x in range(1000000)] → allocates a million integers in RAM right now.

Generator expression — Lazy evaluation: (x*2 for x in range(1000000)) → produces values one at a time, using almost no memory.

# Memory comparison
import sys
lst = [x*2 for x in range(100000)]
gen = (x*2 for x in range(100000))
print(sys.getsizeof(lst))  # ~800,056 bytes
print(sys.getsizeof(gen))  # ~104 bytes — fixed size!

# Use generator for large file processing
def read_large_file(filepath):
    with open(filepath) as f:
        for line in f:
            yield line.strip()  # processes one line at a time

Rule of thumb: If you're iterating through the result once and the dataset is large → use a generator. If you need random access or multiple passes → use a list.

Q2What are Python decorators? Give a data engineering use case.

A decorator wraps a function to add behavior without modifying the original code. The syntax @my_decorator is syntactic sugar for func = my_decorator(func).

import time
import functools

# Decorator to log execution time of pipeline steps
def timer(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        start = time.time()
        result = func(*args, **kwargs)
        elapsed = time.time() - start
        print(f"{func.__name__} completed in {elapsed:.2f}s")
        return result
    return wrapper

# Retry decorator for flaky API calls
def retry(max_attempts=3):
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(max_attempts):
                try:
                    return func(*args, **kwargs)
                except Exception as e:
                    if attempt == max_attempts - 1: raise
                    print(f"Attempt {attempt+1} failed: {e}. Retrying...")
        return wrapper
    return decorator

@timer
@retry(max_attempts=3)
def fetch_api_data(endpoint):
    # your API call here
    pass

Q3What is the difference between *args and **kwargs?

*args collects extra positional arguments into a tuple. **kwargs collects extra keyword arguments into a dictionary.

def pipeline_config(*sources, **options):
    # sources = ('s3://bucket/a', 's3://bucket/b')  ← tuple
    # options = {'partition_by': 'date', 'mode': 'append'}  ← dict
    for source in sources:
        print(f"Processing {source} with options {options}")

pipeline_config('s3://a', 's3://b', partition_by='date', mode='append')

Common DE use: Building flexible pipeline runner functions that accept variable numbers of data sources and arbitrary configuration options.

Q4Explain Python's GIL and how it affects data engineering workloads.

The Global Interpreter Lock (GIL) is a mutex in CPython that prevents multiple native threads from executing Python bytecode simultaneously. This means Python threads cannot achieve true parallelism for CPU-bound tasks.

Impact on data engineering:
• I/O-bound tasks (reading files, HTTP calls, DB queries) — GIL is released during I/O waits, so threading works well for these.
• CPU-bound tasks (data transformation, computation) — GIL blocks parallelism. Use multiprocessing instead (separate processes, each with their own GIL).
• PySpark bypasses the GIL by distributing work to the JVM layer via the Py4J bridge.

2. Data Processing Patterns

Q5How do you read a large CSV file in Python without loading it all into memory?

import pandas as pd
import csv

# Method 1: pandas chunking
chunk_size = 100_000
for chunk in pd.read_csv('large_file.csv', chunksize=chunk_size):
    # process each 100k-row chunk
    processed = chunk[chunk['status'] == 'active']
    processed.to_sql('target_table', engine, if_exists='append', index=False)

# Method 2: pure Python generator (no pandas memory overhead)
def csv_row_generator(filepath):
    with open(filepath, newline='') as f:
        reader = csv.DictReader(f)
        for row in reader:
            yield row

for row in csv_row_generator('large_file.csv'):
    process_row(row)

🎯 Interview Signal Always mention chunking or generators when asked about large file processing. Never say "I'd load it into a DataFrame" without qualifying the file size.

Q6What is the difference between apply(), map(), and applymap() in pandas?

Series.map(func) — Element-wise on a single Series. Fast for simple transformations.
DataFrame.apply(func, axis=0/1) — Applies a function along an axis (rows or columns). Used for row/column-level aggregations.
DataFrame.applymap(func) / DataFrame.map(func) (pandas 2.1+) — Element-wise on the entire DataFrame.

# map — element-wise on one column
df['status_clean'] = df['status'].map({'Y': True, 'N': False})

# apply — custom function per row
df['full_name'] = df.apply(lambda r: f"{r['first']} {r['last']}", axis=1)

# vectorized (always prefer this over apply when possible)
df['revenue_usd'] = df['revenue_eur'] * 1.08  # 10-100x faster than apply

✅ Performance Tip Avoid apply() in hot paths — it's Python-level iteration. Use NumPy or pandas vectorized operations whenever possible.

3. Error Handling & Logging

Q7How do you handle errors gracefully in a data pipeline?

import logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s %(levelname)s %(message)s')
logger = logging.getLogger(__name__)

def process_record(record):
    try:
        # transformation logic
        return transform(record)
    except KeyError as e:
        logger.warning(f"Missing field {e} in record {record.get('id')}")
        return None  # skip bad record, don't crash the pipeline
    except ValueError as e:
        logger.error(f"Data type error: {e} for record {record.get('id')}")
        raise  # re-raise critical errors

# Dead-letter queue pattern for pipelines
good_records, bad_records = [], []
for record in incoming_records:
    result = process_record(record)
    if result is None:
        bad_records.append(record)
    else:
        good_records.append(result)

# Write failed records to a dead-letter file for investigation
if bad_records:
    pd.DataFrame(bad_records).to_csv('dead_letter.csv', index=False)
    logger.warning(f"{len(bad_records)} records written to dead_letter.csv")

4. Performance & Memory

Q8How do you profile and optimize a slow Python data processing script?

import cProfile
import pstats
from io import StringIO

# Profile the script
profiler = cProfile.Profile()
profiler.enable()
run_my_pipeline()  # your function here
profiler.disable()

stream = StringIO()
stats = pstats.Stats(profiler, stream=stream).sort_stats('cumulative')
stats.print_stats(20)  # top 20 slowest functions
print(stream.getvalue())

After profiling, common optimizations in order of impact:
1. Replace apply() with vectorized pandas/NumPy operations — 10–100x speedup.
2. Use appropriate dtypes — category for low-cardinality strings, int32 instead of int64 when safe.
3. Filter early — reduce your DataFrame size before joins/aggregations.
4. Use multiprocessing.Pool for CPU-bound parallel processing.
5. Switch to Polars (Rust-based DataFrame library) for single-machine workloads that outgrow pandas.

Q9What is the difference between is and == in Python? Why does it matter for data pipelines?

== checks value equality (are the values the same?)
is checks identity (are they the same object in memory?)

Why it matters in pipelines:

# DANGEROUS - common bug
def is_null(val):
    return val == None  # Wrong! Can be overridden by custom __eq__

# CORRECT
def is_null(val):
    return val is None  # Checks identity — always reliable

# In pandas, this is why you use pd.isna() / pd.isnull()
# Never: df[df['col'] == None]
# Always: df[df['col'].isna()]

🐍 Ready to Test Your Python Skills?

Take our free interactive quizzes on SQL, Spark, PySpark, Hadoop and Networking — and grab the 300Q PDF bundle for offline prep.

Visit the Blog → Get the PDF Bundle