The dev sandbox

Github Link: Mini Feature Store Today I’m writing about a project I did a year ago called mini feature store. Feature engineering has grown alongside machine learning. In short, feature engineering covers transforming raw data into variables (features) for machine learning. A feature store helps store, catalog, and serve those features. I want to start with the project setup and discuss it from a data engineering perspective to help developers and data engineers understand feature engineering through familiar concepts and setups. Essentially, the data processing side of feature engineering is data engineering tailored for machine learning. They share lots of things in common, like using Spark for data transformations and some base infrastructure (e.g., Kubernetes for cluster management, S3 as storage, and Iceberg or another open table format). ...

Github Link: Sparkcore-Go I’m building Spark’s core execution architecture in Go to explore the systems behind distributed computation and learn through implementation. The work will be divided into three sessions, each with a different focus. Let’s look at how a submitted job becomes executable tasks. Session one focuses on building an MVP Spark execution engine while exploring RDD lineage, lazy evaluation, narrow and shuffle dependencies, and partition-level tasks. Execution focuses on narrow transformations within a single process, without distributed execution. ...

Github Link: Sparkcore-Go In session one, I successfully implemented RDD lineage building, stage planning, and partition-level tasks in a single Go process. Goroutines provide concurrency, and channels coordinate that concurrency at the task execution level. But all goroutines share the same memory, function registry, filesystem access, and process lifecycle. An unrecovered panic in one goroutine can bring down the entire application. In this session, I replace the single process with distributed execution to establish process and network boundaries. I introduce a physical scheduler and multiple independent worker processes that communicate with the coordinator over HTTP. ...

Github Link: Sparkcore-Go Now comes the last part of this Spark-core project: implementing shuffle execution and worker failure recovery, built on top of Session 2’s distributed executor and HTTP coordinator. Shuffle is often one of the most expensive and resource-intensive operations in Spark. It’s interesting to dig into the actual process of redistributing data across partitions and executor nodes in a “cluster.” Along with shuffle execution, we also bring in worker lifecycle management and bounded shuffle-stage recovery to demonstrate fault tolerance through recomputation. ...

Introduction Hi, I’m Wen. I’m a software engineer working on data engineering and large data systems. I work on data platforms, including Spark, data lakehouses, real-time streaming, and data serving. I’m especially interested in distributed systems, Spark internals, and the infrastructure behind modern data and AI platforms. I use this blog to write about what I’m learning at work and through my own programming projects. More about Me When I’m not debugging data platform issues, you’ll find me exploring new destinations 🗺️, hitting the slopes ⛷️, or pushing my limits in CrossFit training 💪. I thrive on challenges, whether I’m trying a harder ski run, pushing through a CrossFit workout, or working through a tough engineering problem.