Building a Spark-Style Execution Engine in Go: From RDD Lineage to Local Task Execution
Spark is used daily in data engineering for large-scale distributed data processing. I’m building Spark’s core execution architecture in Go to explore the systems behind distributed computation and learn through implementation. The work will be divided into three sessions, each with a different focus. Before starting the project, it helps to understand Spark’s core components and how they relate to one another. RDDs are Spark’s fundamental abstraction. We can think of an RDD as a metadata container carrying three main pieces of information: ...