At the heart of Featuretools lies Deep Feature Synthesis (DFS)—a deterministic algorithm that systematically generates features by traversing relational structures and applying composable primitives. Unlike black-box AutoML systems, DFS is fully transparent: every generated feature has an explicit, traceable derivation.

The term "deep" in DFS refers not to deep learning, but to the depth of relationship traversal. Just as deep neural networks compose simple functions across many layers, DFS composes simple primitives across relationship paths.

The Core Insight

DFS is built on a profound observation:

Every feature that a human engineer would manually create can be expressed as a composition of primitive operations applied along a path through the relational schema.

This means that instead of writing custom code for each feature, we can:

1. Define the relational structure (EntitySet)
2. Define the primitive operations (transforms and aggregations)
3. Let DFS systematically explore all valid compositions

The result is exhaustive coverage of the feature space bounded only by computational constraints.

Deep Feature Synthesis can be understood as a graph traversal algorithm that operates on the EntitySet schema. Here's the formal structure:

Input

• EntitySet E: A collection of dataframes D₁, D₂, ..., Dₙ with defined relationships
• Target Dataframe D_target: The entity for which we want features
• Transform Primitives T: Set of single-row operations
• Aggregation Primitives A: Set of aggregation operations
• Max Depth d: Maximum relationship hops to traverse

Output

• Feature Definitions F: Set of symbolic feature definitions
• Feature Matrix M: Computed values for each feature across target instances

Algorithm Pseudocode

Key Observations

1. Recursive structure: DFS naturally recurses through the relationship graph
2. Type matching: Only valid primitive-column combinations are generated
3. Depth control: max_depth prevents infinite recursion and controls complexity
4. Stacking: Aggregations can be applied over other aggregations at depth > 1

Every feature generated by DFS can be represented as a feature tree—a hierarchical structure that encodes the complete derivation of the feature from raw data.

Anatomy of a Feature Tree

Consider the feature: MEAN(orders.SUM(order_items.quantity))

This feature has the following tree structure:

        MEAN (aggregation)
          │
          ▼
        orders
          │
     SUM (aggregation)
          │
          ▼
     order_items
          │
      quantity (direct)

Tree Properties

Feature Definition Serialization

Featuretools can serialize feature definitions to JSON, enabling:

• Versioning: Store feature definitions in version control
• Sharing: Exchange features between team members
• Production deployment: Load feature definitions without rerunning DFS
• Documentation: Auto-generate feature documentation

import featuretools as ft

# Save features to file
ft.save_features(features, "features.json")

# Load features later
loaded_features = ft.load_features("features.json")

# Compute new data using loaded definitions
new_feature_matrix = ft.calculate_feature_matrix(
    features=loaded_features,
    entityset=new_es
)

Not all primitive combinations are valid. DFS enforces strict composition rules based on input/output type compatibility and semantic constraints.

Type Compatibility Matrix

Primitives declare their input and output types. A primitive can only be applied if its input type matches the column's type:

Aggregation Stacking Rules

When max_depth > 1, aggregations can be stacked—applied over the results of other aggregations. This creates the powerful composed features that capture complex patterns.

Valid Stacking Patterns:

1. Aggregate → Aggregate: Most common at depth 2

   • MEAN(orders.SUM(order_items.quantity)) ✓
   • MAX(orders.COUNT(order_items)) ✓
   • STD(orders.MEAN(order_items.unit_price)) ✓

2. Transform → Aggregate: Apply transform, then aggregate result

   • SUM(orders.YEAR(order_date)) — Sum of years (rarely useful)
   • MEAN(orders.IS_WEEKEND(order_date)) — Fraction of weekend orders ✓

3. Aggregate → Transform: Aggregate, then transform (only at target level)

   • LOG(SUM(orders.total_amount)) — Log of total spend ✓
   • YEAR(MAX(orders.order_date)) — Year of most recent order ✓

Invalid Stacking Patterns:

• Type mismatches: MEAN(MODE(...)) where MODE returns categorical
• Transform → Transform over aggregation: Generally not meaningful

DFS navigates the EntitySet by following relationship paths—sequences of relationships that connect entities in the schema graph.

Path Types

Forward Paths (Transform Direction) Traverse from parent to child, bringing parent attributes to child rows:

customers → orders
Result: Each order gets the customer's country, age, etc.

Backward Paths (Aggregation Direction) Traverse from child to parent, aggregating child data:

orders → customers
Result: Each customer gets SUM, MEAN, COUNT of their orders

Deep Paths (Multi-Hop) Traverse multiple relationships:

customers → orders → order_items
Result: Aggregations span two relationship hops

Path Pruning Strategies

Not all paths are equally valuable. DFS provides mechanisms to control traversal:

DFS rests on solid mathematical foundations that guarantee valid, interpretable features. Understanding these foundations helps in debugging and extending the algorithm.

Feature Space as a Lattice

The set of all possible features forms a lattice structure under the composition operation:

• Base elements: Direct columns {x₁, x₂, ..., xₙ}
• Operators: Transform primitives T = {t₁, t₂, ...} and aggregation primitives A = {a₁, a₂, ...}
• Composition: f ∘ g applies f to the output of g

The lattice is bounded:

• Bottom: Direct columns (depth 0)
• Top: Features at max_depth with all primitives applied

Feature Complexity Bounds

Let:

• n = number of columns in target entity
• m = number of child relationships
• |T| = number of transform primitives
• |A| = number of aggregation primitives
• d = max_depth

The number of features grows as:

|F| ≈ n + n·|T| + m·c·|A| · (1 + |A|)^(d-1)

where c is the average number of columns per child entity.

This exponential growth in depth explains why depth control is critical.

Feature Independence Properties

DFS-generated features have important mathematical properties:

1. Determinism: Same inputs → same outputs, always

2. Type Safety: Output type is fully determined by the composition

3. Temporal Consistency: With cutoff times, features only use valid historical data

4. Compositional Semantics: The meaning of f(g(x)) is the composition of meanings of f and g

Null Handling

Featuretools handles nulls consistently:

• Transform on null: Propagates null (unless primitive defines otherwise)
• Aggregation ignoring nulls: Most aggregations skip nulls (like SQL)
• All-null aggregation: Returns null (not 0)

This preserves statistical validity and prevents silent corruption.

When entities have time indices, DFS becomes time-aware, ensuring that features only use data available at the prediction time. This is crucial for preventing data leakage.

The Cutoff Time Contract

When you provide a cutoff_time parameter, DFS guarantees:

For each row in the feature matrix, all aggregated data has a timestamp before that row's cutoff time.

This is enforced at the database/dataframe level before aggregation, ensuring temporal validity by construction.

Multiple Cutoff Times

Often you need features at different points in time for the same entity:

Training Windows

Beyond cutoff times, you can limit the lookback window for aggregations:

feature_matrix, features = ft.dfs(
    entityset=es,
    target_dataframe_name="customers",
    cutoff_time=cutoff_df,
    training_window="30 days"  # Only use last 30 days
)

Use Cases for Training Windows:

Beyond the basic parameters, DFS offers fine-grained control over the feature generation process. Mastering these options enables efficient, targeted feature synthesis.

Primitive Options

Customize primitive behavior per-column or per-entity:

Custom Primitives

When built-in primitives don't suffice, you can define custom primitives:

from featuretools.primitives import AggregationPrimitive
from woodwork.column_schema import ColumnSchema
from woodwork.logical_types import Double
import numpy as np

class GeometricMean(AggregationPrimitive):
    """Computes the geometric mean of a numeric column."""
    
    name = "geometric_mean"
    input_types = [ColumnSchema(logical_type=Double)]
    return_type = ColumnSchema(logical_type=Double)
    
    def get_function(self):
        def geometric_mean(x):
            x = x[x > 0]  # Only positive values
            if len(x) == 0:
                return np.nan
            return np.exp(np.log(x).mean())
        return geometric_mean

# Use the custom primitive
feature_matrix, features = ft.dfs(
    entityset=es,
    target_dataframe_name="customers",
    agg_primitives=["mean", GeometricMean],
    max_depth=2
)

Deep Feature Synthesis is a powerful algorithm that transforms the tedious, error-prone process of manual feature engineering into a systematic, reproducible operation. Let's consolidate the key concepts:

What's Next:

Now that we understand how DFS generates features, we turn to the critical question: Which features are actually useful? The next page covers Feature Evaluation—methods for assessing feature quality, selecting the most predictive features, and managing the feature explosion that DFS can produce.