Design a Machine Learning Feature Store

Feature registry and discovery: central catalog of all ML features across the organisation; each feature registered with metadata — name, description, data type, owner team, source (raw data table/stream), computation logic, freshness SLA, entity (user_id, item_id, order_id); data scientists search and discover existing features before creating new ones

Offline feature store (batch): store historical feature values for model training; features computed via batch pipelines (Spark/BigQuery) from data warehouse; point-in-time correct retrievals — for a training example at time T, return feature values as they existed at time T (prevent data leakage / lookahead bias); store in columnar format (Parquet/Delta Lake) for efficient training data generation

Online feature store (serving): serve feature values at low latency (< 10ms p99) for real-time model inference; when a model receives a prediction request → fetch features for the entity (e.g., user_id=123) from the online store → feed to model; backed by a low-latency key-value store (Redis, DynamoDB, Bigtable)

Feature materialisation: populate the online store from batch or streaming sources; batch materialisation: scheduled Spark job reads features from offline store → writes to online KV store (periodic refresh, e.g., hourly/daily); stream materialisation: Flink/Spark Streaming job computes features from event streams → writes to online store in real-time (seconds-fresh)

Training-serving consistency: the features used during model training MUST match the features served during inference (training-serving skew is a top cause of ML system failures); the same feature definition produces values for both offline training and online serving; transformations defined once, executed in both contexts

Point-in-time joins: when generating training datasets, join features to training examples at the correct timestamp — for each (entity_id, event_time) in the training set, retrieve the latest feature value that was available BEFORE event_time; prevents future data from leaking into training; critical for temporal correctness

Feature transformations: support defining transformations on raw data — aggregations (count, sum, avg over sliding windows), encodings (one-hot, embedding lookups), derived features (ratio of two features), time-based (days since last purchase, time of day); transformations versioned and tracked

Feature freshness and SLAs: different features have different freshness requirements — user_lifetime_purchases: daily refresh OK; user_last_click_item: must be real-time (< 1 minute); define freshness SLAs per feature; monitoring alerts when SLA is violated (feature staleness exceeds threshold)

Feature versioning and lineage: track feature versions (v1, v2 with changed computation logic); lineage: trace from a feature back to its raw data sources and transformation pipelines; when a raw data source changes → identify all downstream features affected; essential for debugging, auditing, and compliance

Access control and sharing: features owned by teams; fine-grained access control (team A can read features owned by team B but not modify); feature sharing across teams reduces duplication (e.g., 'user_age' used by 10+ models across 5 teams); usage tracking (which models use which features)