Feature Stores - Learning Module

Loading content...

0/245

Feast Architecture

The Open-Source Standard

When Gojek, Southeast Asia's leading super-app, faced the challenge of managing features across hundreds of ML models, they built a feature store. That feature store became Feast (Feature Store), and after being open-sourced in 2019, it has become the de facto standard for open-source feature management.

Feast's adoption by Google Cloud, its integration into major MLOps platforms, and its active community have established it as the reference implementation for feature store concepts. Understanding Feast's architecture provides both practical skills and a template for understanding other feature stores.

What You Will Learn

This page provides a comprehensive deep-dive into Feast's architecture. You'll understand its core abstractions (entities, features, feature views), its dual-store pattern, deployment options, and how to build production-ready feature pipelines. By the end, you'll be able to design and implement Feast-based feature infrastructure.

Feast Overview and Philosophy

Feast is an open-source feature store that helps organizations manage and serve ML features to production models. Unlike monolithic platforms, Feast follows a minimalist, composable philosophy—it focuses on the core feature store responsibilities while integrating with existing infrastructure.

Feast Design Principles

•Minimal Footprint — Feast doesn't require a dedicated cluster or complex infrastructure. It can run as a Python library, leveraging existing data stores.
•Pluggable Architecture — Storage backends, compute engines, and registries are pluggable. Use Redis, DynamoDB, Snowflake, or custom solutions interchangeably.
•Python-Native — Built for Python-first workflows. Feature definitions are Python code, not YAML or GUI configurations.
•Infrastructure-Agnostic — Works with on-premises data centers, any cloud provider, or hybrid environments.
•Training-Serving Consistency — Core focus on eliminating training-serving skew through unified feature definitions.

Feast Evolution

Feast has evolved significantly. Feast 0.10+ introduced a simpler, file-based architecture replacing the earlier Kubernetes-native approach. Modern Feast is lightweight enough to run in a Jupyter notebook while scaling to production workloads.

What Feast Is and Isn't:

Feast IS	Feast IS NOT
A feature registry and serving layer	A complete ML platform
A bridge between offline and online stores	A data warehouse or database
A Python SDK for feature retrieval	A feature transformation engine (primarily)
Infrastructure for consistent serving	A model training framework
An integration layer with existing tools	A replacement for data engineering

Core Abstractions

Feast organizes features using a clear hierarchy of abstractions. Understanding these abstractions is essential for effective feature store design.

Converting Mermaid diagram...

An Entity represents the real-world object for which features are computed. It defines the join key that links features to the business domain.

Key Concepts:

Entities are the primary keys for feature tables
Common entities: user_id, product_id, merchant_id, session_id
Features are always associated with one or more entities
Multi-entity features (composite keys) are supported

entities.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
from feast import Entity
 
# Simple entity - single key
user = Entity(
    name="user",
    description="A registered user of the platform",
    join_keys=["user_id"],
    # Optional: specify value type for validation
    # value_type=ValueType.INT64
)
 
# Another common entity
product = Entity(
    name="product",
    description="A product in the catalog",
    join_keys=["product_id"],
)
 
# Composite entity - multiple keys
user_product = Entity(
    name="user_product",
    description="User-product interaction entity",
    join_keys=["user_id", "product_id"],
)
 
# Session-based entity for real-time features
session = Entity(
    name="session",
    description="A user session",
    join_keys=["session_id"],
)

Architecture Deep Dive

Feast's architecture is designed for flexibility and simplicity. Understanding its components helps in optimizing deployments and troubleshooting issues.

Converting Mermaid diagram...

Architectural Components

•Feature Repository — A Git repository containing feature definitions (Python files) and configuration (feast.yaml). Enables version control, code review, and GitOps workflows.
•Feast SDK — The Python client that parses feature definitions, interacts with registries, and retrieves features. This is the primary interface for both configuration and runtime.
•Registry — Stores metadata about features, entities, data sources, and feature services. Backends include file-based (SQLite, Parquet), SQL databases, and cloud-native options (S3, GCS).
•Offline Store — The storage layer for historical feature data used in training. Typically a data warehouse (BigQuery, Snowflake) or file system (S3 + Parquet).
•Online Store — The low-latency storage layer for real-time feature serving. Typically Redis, DynamoDB, or similar key-value stores.
•Materialization Engine — The component that computes feature values and populates stores. Can use Spark, local processing, or serverless engines.
•Feature Server — An optional REST/gRPC service for serving features without embedding the SDK. Useful for non-Python services.

Registry and Metadata

The Registry is Feast's metadata backbone. It stores all information about features, making discovery, governance, and consistency possible. Understanding the registry is key to operating Feast effectively.

Registry Storage Options
Backend	Best For	Pros	Cons
Local File (SQLite)	Development, testing	Zero setup, portable	Single user, no sharing
S3/GCS File	Small teams, simple deployments	Easy sharing, versioned	No concurrent writes
SQL (PostgreSQL)	Production, multi-team	ACID, concurrent access	Requires database management
AWS Registry (DynamoDB)	AWS-native deployments	Serverless, scalable	AWS lock-in
Snowflake Registry	Snowflake-centric orgs	Unified with data warehouse	Snowflake lock-in

feast_yaml_examples.yaml
YAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# Development configuration - local file registry
project: my_ml_project
registry: data/registry.db
provider: local
online_store:
  type: sqlite
  path: data/online_store.db
offline_store:
  type: file
 
---
# Production configuration - distributed registries and stores
project: my_ml_project
registry:
  registry_type: sql
  path: postgresql://user:pass@host:5432/feast_registry
  cache_ttl_seconds: 60  # Cache registry for performance
 
provider: gcp  # or aws, azure
 
online_store:
  type: redis
  connection_string: redis://redis-cluster:6379
  # Alternative: DynamoDB
  # type: dynamodb
  # region: us-west-2
 
offline_store:
  type: bigquery
  project: my-gcp-project
  dataset: feast_features
  # Alternative: Snowflake
  # type: snowflake
  # account: myaccount
  # database: ANALYTICS
 
entity_key_serialization_version: 2
flags:
  alpha_features: true  # Enable experimental features

Registry Operations:

The registry supports several key operations:

feast apply — Registers or updates feature definitions from your feature repository.
feast registry-dump — Exports the current registry state for backup or analysis.
feast teardown — Removes all registered objects (use with caution!).

The registry also caches locally for performance. In production, configure appropriate cache TTLs to balance freshness and performance.

Registry Best Practices

Use SQL-based registries for production deployments with multiple users. Implement GitOps workflows where CI/CD pipelines run 'feast apply' on merge, ensuring the registry always reflects the repository state. Version your feature definitions alongside your model code.

Offline Store Patterns

The Offline Store provides large-scale historical feature retrieval for model training. Feast supports multiple backends, each with distinct performance characteristics and cost profiles.

File-based offline stores use Parquet files on local filesystems or cloud storage. Ideal for development and small-scale production.

file_offline_store.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
# feast.yaml for file-based offline store
"""
project: my_project
registry: s3://my-bucket/registry.db
provider: local
offline_store:
  type: file
"""
 
# Usage - feature retrieval from file sources
from feast import FeatureStore
import pandas as pd
 
store = FeatureStore(repo_path="./feature_repo")
 
# Define entities with timestamps for point-in-time joins
entity_df = pd.DataFrame({
    "user_id": [1001, 1002, 1003],
    "event_timestamp": pd.to_datetime([
        "2024-01-01", "2024-01-02", "2024-01-03"
    ]),
})
 
# Retrieve historical features (triggers Parquet file reads)
training_df = store.get_historical_features(
    entity_df=entity_df,
    features=[
        "user_statistics:total_purchases_30d",
        "user_statistics:avg_purchase_amount_30d",
    ],
).to_df()
 
# For larger datasets, use to_arrow() for memory efficiency
training_arrow = store.get_historical_features(
    entity_df=entity_df,
    features=["user_statistics:total_purchases_30d"],
).to_arrow()

Offline Store Selection

Choose your offline store based on where your data already lives. Data movement is expensive. If your feature source data is in BigQuery, use BigQuery as your offline store. The same applies to Snowflake, Redshift, and other warehouses.

Online Store Patterns

The Online Store provides low-latency feature serving for real-time inference. The choice of online store backend significantly impacts serving performance. Understanding the tradeoffs is critical for production deployments.

Online Store Comparison
Backend	Latency (p99)	Scalability	Cost Model	Best For
SQLite	~10-50ms	Single machine	Free	Development, testing
Redis	~1-5ms	Cluster scaling	Memory-based	Low-latency production
DynamoDB	~5-15ms	Auto-scaling	Request-based	AWS, serverless
Bigtable	~5-10ms	Massive scale	Row/storage	GCP, very high throughput
PostgreSQL	~10-30ms	Moderate	Compute-based	Simple production, SQL familiarity
Cassandra	~5-15ms	Linear scaling	Node-based	Multi-region, high availability

online_store_configs.yaml
YAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# SQLite - Development/Testing
online_store:
  type: sqlite
  path: data/online_store.db
 
---
# Redis - Low-latency production
online_store:
  type: redis
  connection_string: redis://redis-cluster:6379
  # Redis cluster configuration
  # connection_string: redis://host1:6379,host2:6379,host3:6379
  # With authentication
  # connection_string: redis://:password@host:6379
 
---
# Redis with TLS and connection pooling
online_store:
  type: redis
  connection_string: rediss://redis-cluster:6379  # 'rediss' for TLS
  key_ttl_seconds: 86400  # Optional: Key expiration
  redis_type: redis_cluster  # or 'redis' for single node
 
---
# DynamoDB - AWS serverless
online_store:
  type: dynamodb
  region: us-west-2
  table_name_template: "{project}_{table_name}"
  # Optional: Use on-demand capacity for variable workloads
  # billing_mode: PAY_PER_REQUEST
 
---
# Bigtable - GCP high-scale
online_store:
  type: bigtable
  project: my-gcp-project
  instance: feast-instance
  # Table naming
  table_name_template: "{project}_{table_name}"

Online Store Performance Tuning:

Achieving sub-millisecond latencies requires attention to several factors:

Network Topology — Deploy online stores in the same region/zone as serving infrastructure
Connection Pooling — Reuse connections; don't create new connections per request
Batch Retrieval — Fetch all features for an entity in one request
Key Design — Minimize key size for faster lookups
TTL Configuration — Set appropriate TTLs to bound storage costs

Redis for Low Latency

For latency-critical applications (< 5ms p99), Redis remains the top choice. Redis Cluster provides horizontal scaling, and Redis Sentinel provides high availability. Consider AWS ElastiCache or GCP Memorystore for managed Redis deployments.

Materialization Deep Dive

Materialization is the process of computing feature values from sources and populating the online store. Understanding materialization is essential for maintaining fresh features and optimizing costs.

materialization_patterns.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
from feast import FeatureStore
from datetime import datetime, timedelta
 
store = FeatureStore(repo_path="./feature_repo")
 
# Basic materialization - specify time range
store.materialize(
    start_date=datetime(2024, 1, 1),
    end_date=datetime(2024, 1, 31),
    feature_views=["user_statistics", "product_features"],
)
 
# Incremental materialization - from last materialized point to now
store.materialize_incremental(
    end_date=datetime.now(),
    feature_views=["user_statistics"],
)
 
# CLI-based materialization (often used in pipelines)
# feast materialize 2024-01-01T00:00:00 2024-01-31T00:00:00
# feast materialize-incremental $(date -u +"%Y-%m-%dT%H:%M:%S")
 
# Production pattern: Scheduled incremental materialization
# Using Airflow, Prefect, or similar orchestrators
from airflow.decorators import dag, task
from datetime import datetime
 
@dag(schedule_interval="@hourly", start_date=datetime(2024, 1, 1))
def materialize_features():
    @task
    def run_materialization():
        from feast import FeatureStore
        store = FeatureStore(repo_path="./feature_repo")
        store.materialize_incremental(
            end_date=datetime.now(),
            feature_views=["user_statistics", "realtime_features"],
        )
    run_materialization()

Materialization Best Practices

•Incremental Over Full — Use materialize_incremental() for regular updates. Full materialization is expensive and should be rare.
•Appropriate Frequency — Match materialization frequency to feature freshness requirements. Not all features need minute-level freshness.
•Parallel Processing — For large feature sets, materialize different feature views in parallel.
•Error Handling — Implement retry logic and alerting for failed materialization jobs.
•Monitoring — Track materialization latency, row counts, and errors. Alert on anomalies.
•TTL Management — Set appropriate TTLs in feature views to automatically expire stale data.

Materialization Costs

Materialization incurs compute and storage costs. Over-frequent materialization wastes resources; under-frequent materialization serves stale features. Analyze your use case to determine the right balance between freshness and cost.

Feature Server Deployment

The Feature Server is an optional component that provides HTTP/gRPC APIs for feature retrieval. It's essential for non-Python services and high-performance serving scenarios.

Converting Mermaid diagram...

feature_server_deployment.sh
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# Start feature server locally (development)
feast serve --port 6566 --host 0.0.0.0
 
# Start with specific feature repository
feast serve --repo-path /path/to/feature_repo
 
# Docker deployment
docker run -d \
  -p 6566:6566 \
  -v $(pwd)/feature_repo:/feature_repo \
  -e FEAST_REPO_PATH=/feature_repo \
  feastdev/feature-server:latest
 
# Kubernetes deployment (Helm)
helm repo add feast-helm-charts https://feast-helm-charts.storage.googleapis.com
helm install feast-server feast-helm-charts/feast-feature-server \
  --set feast_repo_path=/feature_repo \
  --set replicaCount=3 \
  --set resources.requests.memory=1Gi

feature_server_client.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
import requests
 
# HTTP REST API - Feature retrieval
response = requests.post(
    "http://feature-server:6566/get-online-features",
    json={
        "features": [
            "user_statistics:total_purchases_30d",
            "user_statistics:avg_purchase_amount_30d",
        ],
        "entities": {
            "user_id": [1001, 1002, 1003]
        }
    }
)
features = response.json()
 
# Feature Service retrieval
response = requests.post(
    "http://feature-server:6566/get-online-features",
    json={
        "feature_service": "fraud_detection_v2",
        "entities": {"user_id": [1001]}
    }
)
 
# gRPC client (higher performance)
from feast.protos.feast.serving.ServingService_pb2_grpc import ServingServiceStub
from feast.protos.feast.serving.ServingService_pb2 import GetOnlineFeaturesRequest
import grpc
 
channel = grpc.insecure_channel("feature-server:6567")
stub = ServingServiceStub(channel)
 
request = GetOnlineFeaturesRequest(
    feature_service="fraud_detection_v2",
    # ... entity configuration
)
response = stub.GetOnlineFeatures(request)

Production Feature Server

For production deployments, run multiple feature server replicas behind a load balancer. Use health checks and readiness probes. Consider using the Go-based feature server for lower latency and memory footprint.

Summary: Feast Architecture

We've comprehensively explored Feast's architecture. Let's consolidate the key concepts:

Key Takeaways

•Feast follows a minimalist, composable philosophy — It focuses on core feature store responsibilities while integrating with existing infrastructure through pluggable backends.
•Core abstractions (Entity, Data Source, Feature View, Feature Service) — These building blocks organize features logically and enable consistent retrieval across training and serving.
•The Registry is the metadata backbone — It stores all feature definitions and enables discovery, governance, and versioning. Choose SQL-based registries for production.
•Offline stores optimize for training — Use your existing data warehouse (BigQuery, Snowflake) to avoid data movement. Point-in-time joins are handled automatically.
•Online stores optimize for serving — Redis provides lowest latency; DynamoDB offers serverless simplicity. Match your online store to your latency requirements.
•Materialization bridges offline and online — Incremental materialization keeps online stores fresh. Schedule appropriately based on freshness requirements.
•Feature Server enables non-Python clients — Deploy for polyglot environments. Use gRPC for lowest latency.

What's Next:

Now that we understand Feast's architecture, we'll explore the critical distinction between online and offline feature stores. You'll learn when to use each, how to optimize for their distinct requirements, and patterns for keeping them synchronized.

Page Complete

You now have a comprehensive understanding of Feast's architecture—from core abstractions through deployment patterns. This knowledge provides the foundation for building production-ready feature infrastructure with Feast.

Feast Architecture

The Open-Source Standard

What You Will Learn

Feast Overview and Philosophy

Feast Design Principles

•Minimal Footprint — Feast doesn't require a dedicated cluster or complex infrastructure. It can run as a Python library, leveraging existing data stores.
•Pluggable Architecture — Storage backends, compute engines, and registries are pluggable. Use Redis, DynamoDB, Snowflake, or custom solutions interchangeably.
•Python-Native — Built for Python-first workflows. Feature definitions are Python code, not YAML or GUI configurations.
•Infrastructure-Agnostic — Works with on-premises data centers, any cloud provider, or hybrid environments.
•Training-Serving Consistency — Core focus on eliminating training-serving skew through unified feature definitions.

Feast Evolution

What Feast Is and Isn't:

Feast IS	Feast IS NOT
A feature registry and serving layer	A complete ML platform
A bridge between offline and online stores	A data warehouse or database
A Python SDK for feature retrieval	A feature transformation engine (primarily)
Infrastructure for consistent serving	A model training framework
An integration layer with existing tools	A replacement for data engineering

Core Abstractions

Feast organizes features using a clear hierarchy of abstractions. Understanding these abstractions is essential for effective feature store design.

Converting Mermaid diagram...

An Entity represents the real-world object for which features are computed. It defines the join key that links features to the business domain.

Key Concepts:

Entities are the primary keys for feature tables
Common entities: user_id, product_id, merchant_id, session_id
Features are always associated with one or more entities
Multi-entity features (composite keys) are supported

entities.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
from feast import Entity
 
# Simple entity - single key
user = Entity(
    name="user",
    description="A registered user of the platform",
    join_keys=["user_id"],
    # Optional: specify value type for validation
    # value_type=ValueType.INT64
)
 
# Another common entity
product = Entity(
    name="product",
    description="A product in the catalog",
    join_keys=["product_id"],
)
 
# Composite entity - multiple keys
user_product = Entity(
    name="user_product",
    description="User-product interaction entity",
    join_keys=["user_id", "product_id"],
)
 
# Session-based entity for real-time features
session = Entity(
    name="session",
    description="A user session",
    join_keys=["session_id"],
)

Architecture Deep Dive

Feast's architecture is designed for flexibility and simplicity. Understanding its components helps in optimizing deployments and troubleshooting issues.

Converting Mermaid diagram...

Architectural Components

•Feature Repository — A Git repository containing feature definitions (Python files) and configuration (feast.yaml). Enables version control, code review, and GitOps workflows.
•Feast SDK — The Python client that parses feature definitions, interacts with registries, and retrieves features. This is the primary interface for both configuration and runtime.
•Registry — Stores metadata about features, entities, data sources, and feature services. Backends include file-based (SQLite, Parquet), SQL databases, and cloud-native options (S3, GCS).
•Offline Store — The storage layer for historical feature data used in training. Typically a data warehouse (BigQuery, Snowflake) or file system (S3 + Parquet).
•Online Store — The low-latency storage layer for real-time feature serving. Typically Redis, DynamoDB, or similar key-value stores.
•Materialization Engine — The component that computes feature values and populates stores. Can use Spark, local processing, or serverless engines.
•Feature Server — An optional REST/gRPC service for serving features without embedding the SDK. Useful for non-Python services.

Registry and Metadata

Registry Storage Options
Backend	Best For	Pros	Cons
Local File (SQLite)	Development, testing	Zero setup, portable	Single user, no sharing
S3/GCS File	Small teams, simple deployments	Easy sharing, versioned	No concurrent writes
SQL (PostgreSQL)	Production, multi-team	ACID, concurrent access	Requires database management
AWS Registry (DynamoDB)	AWS-native deployments	Serverless, scalable	AWS lock-in
Snowflake Registry	Snowflake-centric orgs	Unified with data warehouse	Snowflake lock-in

feast_yaml_examples.yaml
YAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# Development configuration - local file registry
project: my_ml_project
registry: data/registry.db
provider: local
online_store:
  type: sqlite
  path: data/online_store.db
offline_store:
  type: file
 
---
# Production configuration - distributed registries and stores
project: my_ml_project
registry:
  registry_type: sql
  path: postgresql://user:pass@host:5432/feast_registry
  cache_ttl_seconds: 60  # Cache registry for performance
 
provider: gcp  # or aws, azure
 
online_store:
  type: redis
  connection_string: redis://redis-cluster:6379
  # Alternative: DynamoDB
  # type: dynamodb
  # region: us-west-2
 
offline_store:
  type: bigquery
  project: my-gcp-project
  dataset: feast_features
  # Alternative: Snowflake
  # type: snowflake
  # account: myaccount
  # database: ANALYTICS
 
entity_key_serialization_version: 2
flags:
  alpha_features: true  # Enable experimental features

Registry Operations:

The registry supports several key operations:

feast apply — Registers or updates feature definitions from your feature repository.
feast registry-dump — Exports the current registry state for backup or analysis.
feast teardown — Removes all registered objects (use with caution!).

The registry also caches locally for performance. In production, configure appropriate cache TTLs to balance freshness and performance.

Registry Best Practices

Offline Store Patterns

The Offline Store provides large-scale historical feature retrieval for model training. Feast supports multiple backends, each with distinct performance characteristics and cost profiles.

File-based offline stores use Parquet files on local filesystems or cloud storage. Ideal for development and small-scale production.

file_offline_store.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
# feast.yaml for file-based offline store
"""
project: my_project
registry: s3://my-bucket/registry.db
provider: local
offline_store:
  type: file
"""
 
# Usage - feature retrieval from file sources
from feast import FeatureStore
import pandas as pd
 
store = FeatureStore(repo_path="./feature_repo")
 
# Define entities with timestamps for point-in-time joins
entity_df = pd.DataFrame({
    "user_id": [1001, 1002, 1003],
    "event_timestamp": pd.to_datetime([
        "2024-01-01", "2024-01-02", "2024-01-03"
    ]),
})
 
# Retrieve historical features (triggers Parquet file reads)
training_df = store.get_historical_features(
    entity_df=entity_df,
    features=[
        "user_statistics:total_purchases_30d",
        "user_statistics:avg_purchase_amount_30d",
    ],
).to_df()
 
# For larger datasets, use to_arrow() for memory efficiency
training_arrow = store.get_historical_features(
    entity_df=entity_df,
    features=["user_statistics:total_purchases_30d"],
).to_arrow()

Offline Store Selection

Online Store Patterns

Online Store Comparison
Backend	Latency (p99)	Scalability	Cost Model	Best For
SQLite	~10-50ms	Single machine	Free	Development, testing
Redis	~1-5ms	Cluster scaling	Memory-based	Low-latency production
DynamoDB	~5-15ms	Auto-scaling	Request-based	AWS, serverless
Bigtable	~5-10ms	Massive scale	Row/storage	GCP, very high throughput
PostgreSQL	~10-30ms	Moderate	Compute-based	Simple production, SQL familiarity
Cassandra	~5-15ms	Linear scaling	Node-based	Multi-region, high availability

online_store_configs.yaml
YAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# SQLite - Development/Testing
online_store:
  type: sqlite
  path: data/online_store.db
 
---
# Redis - Low-latency production
online_store:
  type: redis
  connection_string: redis://redis-cluster:6379
  # Redis cluster configuration
  # connection_string: redis://host1:6379,host2:6379,host3:6379
  # With authentication
  # connection_string: redis://:password@host:6379
 
---
# Redis with TLS and connection pooling
online_store:
  type: redis
  connection_string: rediss://redis-cluster:6379  # 'rediss' for TLS
  key_ttl_seconds: 86400  # Optional: Key expiration
  redis_type: redis_cluster  # or 'redis' for single node
 
---
# DynamoDB - AWS serverless
online_store:
  type: dynamodb
  region: us-west-2
  table_name_template: "{project}_{table_name}"
  # Optional: Use on-demand capacity for variable workloads
  # billing_mode: PAY_PER_REQUEST
 
---
# Bigtable - GCP high-scale
online_store:
  type: bigtable
  project: my-gcp-project
  instance: feast-instance
  # Table naming
  table_name_template: "{project}_{table_name}"

Online Store Performance Tuning:

Achieving sub-millisecond latencies requires attention to several factors:

Network Topology — Deploy online stores in the same region/zone as serving infrastructure
Connection Pooling — Reuse connections; don't create new connections per request
Batch Retrieval — Fetch all features for an entity in one request
Key Design — Minimize key size for faster lookups
TTL Configuration — Set appropriate TTLs to bound storage costs

Redis for Low Latency

Materialization Deep Dive

materialization_patterns.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
from feast import FeatureStore
from datetime import datetime, timedelta
 
store = FeatureStore(repo_path="./feature_repo")
 
# Basic materialization - specify time range
store.materialize(
    start_date=datetime(2024, 1, 1),
    end_date=datetime(2024, 1, 31),
    feature_views=["user_statistics", "product_features"],
)
 
# Incremental materialization - from last materialized point to now
store.materialize_incremental(
    end_date=datetime.now(),
    feature_views=["user_statistics"],
)
 
# CLI-based materialization (often used in pipelines)
# feast materialize 2024-01-01T00:00:00 2024-01-31T00:00:00
# feast materialize-incremental $(date -u +"%Y-%m-%dT%H:%M:%S")
 
# Production pattern: Scheduled incremental materialization
# Using Airflow, Prefect, or similar orchestrators
from airflow.decorators import dag, task
from datetime import datetime
 
@dag(schedule_interval="@hourly", start_date=datetime(2024, 1, 1))
def materialize_features():
    @task
    def run_materialization():
        from feast import FeatureStore
        store = FeatureStore(repo_path="./feature_repo")
        store.materialize_incremental(
            end_date=datetime.now(),
            feature_views=["user_statistics", "realtime_features"],
        )
    run_materialization()

Materialization Best Practices

•Incremental Over Full — Use materialize_incremental() for regular updates. Full materialization is expensive and should be rare.
•Appropriate Frequency — Match materialization frequency to feature freshness requirements. Not all features need minute-level freshness.
•Parallel Processing — For large feature sets, materialize different feature views in parallel.
•Error Handling — Implement retry logic and alerting for failed materialization jobs.
•Monitoring — Track materialization latency, row counts, and errors. Alert on anomalies.
•TTL Management — Set appropriate TTLs in feature views to automatically expire stale data.

Materialization Costs

Feature Server Deployment

The Feature Server is an optional component that provides HTTP/gRPC APIs for feature retrieval. It's essential for non-Python services and high-performance serving scenarios.

Converting Mermaid diagram...

feature_server_deployment.sh
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# Start feature server locally (development)
feast serve --port 6566 --host 0.0.0.0
 
# Start with specific feature repository
feast serve --repo-path /path/to/feature_repo
 
# Docker deployment
docker run -d \
  -p 6566:6566 \
  -v $(pwd)/feature_repo:/feature_repo \
  -e FEAST_REPO_PATH=/feature_repo \
  feastdev/feature-server:latest
 
# Kubernetes deployment (Helm)
helm repo add feast-helm-charts https://feast-helm-charts.storage.googleapis.com
helm install feast-server feast-helm-charts/feast-feature-server \
  --set feast_repo_path=/feature_repo \
  --set replicaCount=3 \
  --set resources.requests.memory=1Gi

feature_server_client.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
import requests
 
# HTTP REST API - Feature retrieval
response = requests.post(
    "http://feature-server:6566/get-online-features",
    json={
        "features": [
            "user_statistics:total_purchases_30d",
            "user_statistics:avg_purchase_amount_30d",
        ],
        "entities": {
            "user_id": [1001, 1002, 1003]
        }
    }
)
features = response.json()
 
# Feature Service retrieval
response = requests.post(
    "http://feature-server:6566/get-online-features",
    json={
        "feature_service": "fraud_detection_v2",
        "entities": {"user_id": [1001]}
    }
)
 
# gRPC client (higher performance)
from feast.protos.feast.serving.ServingService_pb2_grpc import ServingServiceStub
from feast.protos.feast.serving.ServingService_pb2 import GetOnlineFeaturesRequest
import grpc
 
channel = grpc.insecure_channel("feature-server:6567")
stub = ServingServiceStub(channel)
 
request = GetOnlineFeaturesRequest(
    feature_service="fraud_detection_v2",
    # ... entity configuration
)
response = stub.GetOnlineFeatures(request)

Production Feature Server

Summary: Feast Architecture

We've comprehensively explored Feast's architecture. Let's consolidate the key concepts:

Key Takeaways

•Feast follows a minimalist, composable philosophy — It focuses on core feature store responsibilities while integrating with existing infrastructure through pluggable backends.
•Core abstractions (Entity, Data Source, Feature View, Feature Service) — These building blocks organize features logically and enable consistent retrieval across training and serving.
•The Registry is the metadata backbone — It stores all feature definitions and enables discovery, governance, and versioning. Choose SQL-based registries for production.
•Offline stores optimize for training — Use your existing data warehouse (BigQuery, Snowflake) to avoid data movement. Point-in-time joins are handled automatically.
•Online stores optimize for serving — Redis provides lowest latency; DynamoDB offers serverless simplicity. Match your online store to your latency requirements.
•Materialization bridges offline and online — Incremental materialization keeps online stores fresh. Schedule appropriately based on freshness requirements.
•Feature Server enables non-Python clients — Deploy for polyglot environments. Use gRPC for lowest latency.

What's Next:

Page Complete