Data Engineer Interview Questions: The Complete 2026 Guide

I'd start by checking the Spark UI to understand where time is being spent — specifically the stage timeline, task duration distribution, and shuffle read/write sizes.

The most common culprits are data skew, insufficient partitioning, and excessive shuffles from wide transformations.

For data skew, if one partition handles 70% of the data, I'd add a salt key to distribute the load — appending a random integer between 0–9 to the join key and broadcasting a replicated version of the smaller dataset.

For shuffle reduction, I'd check if I can repartition data earlier in the pipeline, cache intermediate results that are reused, and replace groupBy with reduceByKey where possible.

If it's an I/O bottleneck, I'd look at file sizes — too many small files kill performance. Coalescing output to 128–256MB files and switching from CSV to Parquet with Snappy compression typically gives a 3–4x improvement on read performance.

Finally, I'd review executor memory configuration. If GC time is high in the Spark UI, the job is memory-constrained and needs more executor memory or a smaller partition size.

The medallion architecture organises data into three layers: Bronze, Silver, and Gold.

Bronze is raw ingestion — untransformed data exactly as it arrived from the source. We keep this indefinitely as an audit trail and reprocessing safety net.

Silver is cleaned and conformed data — nulls handled, schema enforced, deduplication applied, possibly joined with reference data. This is what data engineers primarily work with.

Gold is business-ready aggregates — pre-computed metrics, summaries, and domain-specific views built for analysts and dashboards.

The practical benefit is that each layer has a clear owner and SLA. If a Gold metric is wrong, you debug from Gold back to Silver, then Silver back to Bronze — rather than re-ingesting everything from source.

In a Delta Lake implementation, this pairs naturally with time travel — you can replay Silver transformations against Bronze history without touching the source system again.

Schema evolution is one of the trickier operational problems in data engineering because the impact is often downstream and invisible until something breaks.

My approach depends on the storage format. With Parquet and Delta Lake, schema evolution is relatively safe — you can add new nullable columns without breaking existing readers. But removing or renaming columns will break things.

For Kafka-based pipelines, I use Confluent Schema Registry with Avro. The registry enforces compatibility checks — typically backward compatibility, meaning new schemas can read old messages.

For SQL-based warehouses, I version transformation logic in dbt and make schema changes in separate migration PRs reviewed by the team. Any breaking change goes through a deprecation window where both the old and new column exist simultaneously.

The most important practice is automated compatibility testing in CI — run your pipeline against a sample of historical data whenever schema changes are proposed. Catching breaks in staging is much cheaper than in production.