grok14ENGINEERING FIELD SCHOOL
Curriculum/Module 05
Build / Week 5

Spark & Delta engineering

Build repeatable transformations and understand the execution decisions behind reliable incremental pipelines.

5 lesson sections16-hour study & practice planModule 04 or equivalent experience

See the system

Build repeatable transformations and understand the execution decisions behind reliable incremental pipelines.

Source records pass through validation and curated tables. Transformations establish usable entities before search indexing. Reconciliation checks completeness, freshness, updates, and deletions.
From source records to reliable evidence. Source records pass through validation and curated tables. Transformations establish usable entities before search indexing. Reconciliation checks completeness, freshness, updates, and deletions.
Module 05 / Lesson 01

Reason about distributed execution

Spark builds an execution plan from transformations and runs it when an action requires a result. Distributed processing is useful when data and computation justify its overhead; it does not automatically make every small task faster.

Shuffles move data between partitions and often dominate cost. Read the plan to understand joins, exchanges, filters, and scans. Avoid collecting a large distributed result into one driver process just because the API makes it easy.

Apply the ideaIdentify which operations in a sample pipeline require moving data between partitions.
Module 05 / Lesson 02

Express transformations clearly

Use DataFrame or SQL operations that the engine can reason about. Select needed columns, apply appropriate filters, and state schemas when inference could be ambiguous. Nulls, malformed records, and timestamp zones require deliberate handling.

Keep business transformations testable on a small fixture. A synthetic dataset with duplicates, late records, and invalid values is often more useful for correctness than a large happy-path sample.

Apply the ideaCreate a six-row fixture containing a duplicate, null, update, and malformed timestamp.
Module 05 / Lesson 03

Use Delta changes deliberately

Transactional tables support reliable updates and history under documented semantics. A merge needs a clear business key and a policy for multiple source records matching the same target. Schema evolution is a decision, not a reason to accept every unexpected field silently.

Think about deletes as well as inserts. A policy removed from a source must not remain indefinitely retrievable downstream. Retention and maintenance operations also affect what history remains available.

Apply the ideaDefine insert, update, duplicate, and delete behavior for policy documents.
Module 05 / Lesson 04

Make ingestion safe to repeat

Pipelines rerun after failure. Track stable source identifiers and use an explicit ingestion strategy so retries do not duplicate logical records. A watermark alone can miss updates arriving late or with equal timestamps.

Record batch identity, source range, ingestion time, and reconciliation results. Separate rejected data for investigation when appropriate. A successful job status does not prove that all expected business records arrived.

Apply the ideaDescribe how your pipeline recovers after data is written but before a checkpoint is recorded.
Module 05 / Lesson 05

Tune after measuring

Start with correctness and a measured bottleneck. Examine skew, partition sizes, repeated scans, join strategies, and unnecessary serialization. Avoid adding cache everywhere: cached data consumes resources and can become stale relative to the intended workload.

Report input size, compute configuration, runtime, and repeated measurements. An optimization that improves one tiny fixture may not generalize. Preserve a correct baseline so performance work cannot silently change results.

Apply the ideaCompare two equivalent transformations and verify identical results before timing them.

Worked scenario

A nightly append loads the same policy file twice after an interrupted run. Introduce stable document/revision identifiers and an ingestion policy that recognizes already processed revisions. Then test an update and a deletion separately; deduplication alone does not establish correct change handling.

Practical assignment

This is a practical design or implementation assignment. Use synthetic data. Where managed services are required, verify account access, costs, supported features, and cleanup before provisioning.
  1. Build a synthetic ticket and policy fixture.
  2. Define keys and change semantics.
  3. Implement a transformation in Spark or a clearly labeled local equivalent.
  4. Run it twice and verify the logical result.
  5. Inject an update, delete, and malformed row.
  6. Record reconciliation and one measured performance observation.

What to submit

Submit the artifacts named above, a short explanation of your decisions, and evidence of the checks you performed. Distinguish measured results from estimates and designs from executed integrations.

Review dimensionSubmission evidence
CorrectnessShow the expected behavior and a meaningful counterexample.
ReproducibilityState setup, inputs, versions, and what was actually executed.
Delivery judgmentExplain the client impact, alternative, and unresolved assumption.
Operational boundaryIdentify permissions, failure behavior, and any resource cleanup.
Open the lab workbenchGet delivery templates

Knowledge check

1. When does Spark normally execute a lazy transformation plan?
2. What makes a rerunnable ingestion reliable?
3. Should schema evolution accept any unexpected field automatically?

Answer guide
  1. When an action requires results. Transformations describe work; actions trigger evaluation.
  2. Stable identity and explicit change semantics. Retries must preserve the intended logical records.
  3. Only as a deliberate contract decision. Unexpected changes may indicate upstream defects or incompatible meaning.

References & next step

Platform examples are environment-dependent. Start with the official documentation in the reference library and verify the exact cloud, region, privileges, and versions you use.

Open the official reference library

Editorial edition: 5 October 2026. The local reference lab is executed locally; this course does not claim a live Databricks deployment.