Mini Feature Store: Structuring the Offline Store
Github Link: Mini Feature Store Today I’m writing about a project I did a year ago called mini feature store. Feature engineering has grown alongside machine learning. In short, feature engineering covers transforming raw data into variables (features) for machine learning. A feature store helps store, catalog, and serve those features. I want to start with the project setup and discuss it from a data engineering perspective to help developers and data engineers understand feature engineering through familiar concepts and setups. Essentially, the data processing side of feature engineering is data engineering tailored for machine learning. They share lots of things in common, like using Spark for data transformations and some base infrastructure (e.g., Kubernetes for cluster management, S3 as storage, and Iceberg or another open table format). ...