Apache Spark for Databricks Exam: Key Concepts Cheat Sheet

Why Apache Spark Matters for the Databricks Exam

If you're preparing for the Databricks Certified Data Engineer Associate exam, Apache Spark is arguably the single most important technology to master. Databricks was built by the creators of Apache Spark, and the platform runs on Spark under the hood. This means Spark concepts permeate nearly every exam domain - from data ingestion to transformations to productionizing pipelines.

The updated July 2025 exam version doubles down on practical Spark knowledge. With 45 multiple-choice questions, a 90-minute time limit, and a passing score of 70%, there's little room to be fuzzy on how Spark works. Candidates who treat Spark as a peripheral topic rather than the core engine of Databricks often find themselves on the wrong side of that 70% threshold.

This cheat sheet is designed to give you a dense, exam-focused reference for every major Spark concept tested on the certification. Whether you're looking for a quick refresher before exam day or building your databricks exam prep plan from scratch, this guide has you covered. For a broader look at the full exam scope, check out the Databricks Data Engineer Associate Study Guide 2026 (Updated July 2025 Exam).

Spark Architecture: The Foundation You Must Know

Understanding Spark's distributed architecture is non-negotiable for the exam. Questions in Domain 1 (Databricks Intelligence Platform) and Domain 2 (Development and Ingestion) frequently test whether you understand how Spark distributes work across a cluster.

Driver and Executor Model

Every Spark application has a Driver and one or more Executors. The Driver runs the main application code, coordinates the execution plan, and communicates with the cluster manager. Executors are worker processes that run tasks and store data in memory or on disk.

• Driver: Hosts the SparkContext, creates the DAG, schedules tasks, and collects results.
• Executors: Run individual tasks assigned by the Driver, cache data in memory, and report status back.
• Cluster Manager: Allocates resources (Databricks uses its own optimized cluster manager on top of cloud providers).

DAG, Jobs, Stages, and Tasks

When you run a Spark action, the framework builds a Directed Acyclic Graph (DAG) of operations. This DAG is then broken into:

1. Jobs: Triggered by each action (e.g., collect(), count()).
2. Stages: Subsets of a job separated by shuffle boundaries.
3. Tasks: Individual units of work sent to Executors, one per partition per stage.

RDDs, DataFrames, and Datasets Explained

Spark has evolved through three major APIs, and the exam tests all three - though with different weight.

Resilient Distributed Datasets (RDDs)

RDDs are the lowest-level abstraction in Spark. They are immutable, distributed collections of objects with no schema. Key properties:

• Resilient: Can be recomputed from lineage if a partition is lost.
• Distributed: Split across multiple nodes in the cluster.
• Lazy: Transformations are recorded but not executed until an action fires.

While RDDs are foundational, modern Databricks workflows rarely use them directly. Understand their concept, but don't over-invest study time here.

DataFrames

DataFrames are the bread and butter of modern Spark. They are distributed collections of data organized into named columns - think of them as distributed relational tables with a schema. DataFrames are optimized by the Catalyst optimizer and Tungsten execution engine, making them significantly faster than raw RDDs for most workloads.

• Created via spark.read, SQL queries, or transforming existing DataFrames.
• Support both the DataFrame API (Python, Scala, Java, R) and Spark SQL.
• Column operations use the Column class (df["column_name"] or col("column_name")).

Datasets

Datasets combine the benefits of RDDs (strong typing) and DataFrames (optimization). In practice, they are primarily a Scala/Java API feature. Python users work exclusively with DataFrames (Python's dynamic typing makes Datasets redundant in PySpark).

Transformations vs. Actions: A Critical Distinction

This is one of the highest-frequency topics across all spark certification practice test material and real exam questions. You must know the difference cold.

Transformations

Transformations return a new DataFrame or RDD. They are lazy - they define what should happen but don't execute immediately. Transformations are divided into:

• Narrow Transformations: Each input partition contributes to at most one output partition. No shuffle required. Examples: filter(), map(), select(), withColumn().
• Wide Transformations: Data from multiple input partitions may be combined into output partitions. Requires a shuffle. Examples: groupBy(), join(), distinct(), orderBy().

Actions

Actions trigger the execution of the DAG and return results to the Driver or write data to storage. Common actions:

• collect() - Returns all rows to the Driver (use with caution on large datasets).
• count() - Returns the number of rows.
• show() - Displays rows in a human-readable format.
• write() - Saves data to storage (Delta, Parquet, CSV, etc.).
• take(n) - Returns the first n rows.

Concept	Transformation	Action
Execution	Lazy (deferred)	Eager (immediate)
Returns	New DataFrame/RDD	Value or side effect
Triggers DAG	No	Yes
Examples	filter, select, groupBy	collect, count, write
Shuffle Possible?	Yes (wide only)	N/A

Spark SQL and the Catalyst Optimizer

Spark SQL is the module that allows you to run SQL queries against DataFrames and registered tables. In the Databricks environment, Spark SQL is deeply integrated - you can mix SQL cells and Python/Scala cells in the same notebook.

Catalyst Optimizer

The Catalyst Optimizer is Spark's query optimization engine. It takes your logical query plan and transforms it into an efficient physical execution plan through four phases:

1. Analysis: Resolves column references and table names.
2. Logical Optimization: Applies rules like predicate pushdown and constant folding.
3. Physical Planning: Selects join strategies and generates physical plans.
4. Code Generation: Generates Java bytecode for efficient execution (via Tungsten).

Predicate Pushdown

One optimization you must know: predicate pushdown. Spark (and Delta Lake) can push filter conditions down to the data source layer, reading only the relevant data. This is especially powerful with Delta Lake's file skipping based on min/max statistics. This connects directly to Delta Lake exam topics - see the Delta Lake Interview Questions and Exam Prep Guide for deeper coverage.

Temp Views and Global Temp Views

• Temp Views (createOrReplaceTempView): Session-scoped. Only accessible within the same SparkSession.
• Global Temp Views (createOrReplaceGlobalTempView): Application-scoped. Accessible across sessions but accessed with the global_temp database prefix.

Partitioning and Shuffling

Partitioning is one of the most performance-critical concepts in Spark and one that appears frequently in databricks certification study guide materials and actual exam questions.

Default Partitioning

By default, Spark creates partitions based on the data source. For HDFS and cloud storage, Spark typically creates one partition per 128 MB block. After a shuffle, the number of partitions is controlled by spark.sql.shuffle.partitions (default: 200).

Repartition vs. Coalesce

• repartition(n): Performs a full shuffle to create exactly n partitions. Can increase or decrease partition count. Used when you need evenly distributed partitions.
• coalesce(n): Reduces the number of partitions without a full shuffle by combining existing partitions. More efficient for reducing partitions, but can result in uneven partition sizes.

Shuffle Operations and Why They're Expensive

A shuffle involves redistributing data across the cluster - data moves between Executors over the network. This is expensive because it involves disk I/O, serialization, and network transfer. Wide transformations (groupBy, join, distinct) always trigger a shuffle. Minimizing unnecessary shuffles is a core performance tuning strategy.

Structured Streaming with Spark

Domain 2 (Development and Ingestion) and Domain 4 (Productionizing Data Pipelines) both test Structured Streaming heavily. This is Spark's primary API for processing real-time data streams.

Core Concepts

• Micro-batch processing: By default, Structured Streaming processes data in small batches triggered at regular intervals.
• Continuous processing: A lower-latency mode (experimental) that processes records as they arrive.
• Trigger: Controls how often data is processed. Options include Trigger.ProcessingTime, Trigger.Once, and Trigger.AvailableNow.
• Checkpointing: Saves the state of a streaming query to fault-tolerant storage, enabling recovery after failures.
• Watermarking: Handles late-arriving data by defining how late data is tolerated in event-time windows.

Output Modes

Output Mode	Description	Use Case
Append	Only new rows are written to sink	Immutable event data
Complete	Entire result table written each trigger	Aggregations, counts
Update	Only changed rows since last trigger	Aggregations with updates

Auto Loader

Auto Loader (cloudFiles source) is a Databricks-specific feature built on top of Structured Streaming. It incrementally and efficiently ingests new data files as they arrive in cloud storage. It uses file notifications (cloud provider event systems) or directory listing to detect new files. Auto Loader is a high-frequency exam topic - know that it supports schema inference and schema evolution.

Performance Tuning Cheat Sheet

Performance optimization questions appear throughout Domain 3 (Data Processing and Transformations) and Domain 4. Here's what to know:

Mapping Spark Concepts to Exam Domains

Understanding which Spark concepts map to which exam domain helps you prioritize your study time. Here's a practical breakdown for your databricks exam prep:

Spark Concept	Primary Domain	Weight
Cluster architecture, SparkSession	Domain 1: Intelligence Platform	10%
DataFrames, Auto Loader, Structured Streaming	Domain 2: Development & Ingestion	30%
Transformations, SQL, Joins, UDFs, Aggregations	Domain 3: Data Processing	30%
Streaming triggers, checkpointing, pipelines	Domain 4: Productionizing	20%
Data quality checks, schema enforcement	Domain 5: Governance	10%

Domains 2 and 3 together account for 60% of the exam - and both are Spark-heavy. This is where your study time should be concentrated. To understand how the exam difficulty compares to other certifications, read our article Is the Databricks Certification Exam Hard? Real Pass Rates and Difficulty.

If you're wondering how this certification stacks up against alternatives, the article on Databricks vs Snowflake Certification: Which Should You Get First? provides an excellent comparison of both learning paths and career payoffs.

Common Mistakes Candidates Make on Spark Questions

Based on patterns seen across thousands of practice attempts on our Databricks DEA practice test platform, here are the most common Spark-related mistakes that cost candidates the exam:

Other frequent mistakes include:

• Misidentifying when lazy evaluation kicks in: Candidates sometimes think createOrReplaceTempView() triggers execution - it doesn't. Only actions do.
• Confusing spark.sql.shuffle.partitions with input partitions: This setting only affects post-shuffle partitions, not initial read partitions.
• Forgetting that cache() is lazy too: Calling cache() doesn't immediately cache the data - it only caches on the first action that triggers computation.
• Misunderstanding checkpointing scope in Structured Streaming: Checkpoints track processing state, not the raw data. Clearing a checkpoint resets the stream's offset tracking.
• Assuming UDFs are automatically optimized: User-Defined Functions (UDFs) are black boxes to the Catalyst optimizer. They break optimization chains, which is why Spark built-in functions are always preferred.

Practice identifying these traps with a databricks certified data engineer associate practice test. Our full practice exam platform includes hundreds of scenario-based Spark questions designed to simulate the real exam experience. You can also start with our Free Databricks Practice Questions: 25 Sample Questions With Answers to gauge where you stand before diving deeper.

One final area worth reviewing before exam day: make sure you understand how Spark concepts extend into the Professional-level exam if you're planning to advance your certification path. The Databricks Data Engineer Associate vs Professional: Which Level? article breaks down exactly how Spark depth requirements scale between the two levels.

Frequently Asked Questions