Spark

Github Link: Sparkcore-Go I’m building Spark’s core execution architecture in Go to explore the systems behind distributed computation and learn through implementation. The work will be divided into three sessions, each with a different focus. Let’s look at how a submitted job becomes executable tasks. Session one focuses on building an MVP Spark execution engine while exploring RDD lineage, lazy evaluation, narrow and shuffle dependencies, and partition-level tasks. Execution focuses on narrow transformations within a single process, without distributed execution. ...

Github Link: Sparkcore-Go In session one, I successfully implemented RDD lineage building, stage planning, and partition-level tasks in a single Go process. Goroutines provide concurrency, and channels coordinate that concurrency at the task execution level. But all goroutines share the same memory, function registry, filesystem access, and process lifecycle. An unrecovered panic in one goroutine can bring down the entire application. In this session, I replace the single process with distributed execution to establish process and network boundaries. I introduce a physical scheduler and multiple independent worker processes that communicate with the coordinator over HTTP. ...

Github Link: Sparkcore-Go Now comes the last part of this Spark-core project: implementing shuffle execution and worker failure recovery, built on top of Session 2’s distributed executor and HTTP coordinator. Shuffle is often one of the most expensive and resource-intensive operations in Spark. It’s interesting to dig into the actual process of redistributing data across partitions and executor nodes in a “cluster.” Along with shuffle execution, we also bring in worker lifecycle management and bounded shuffle-stage recovery to demonstrate fault tolerance through recomputation. ...