Data Debugging — Debugging ML Models | OneNoughtOne

0/106

Machine LearningML Engineering & Career

Debugging ML Models

LevelAdvanced

Duration90 mins

TopicML Engineering & Career

2 / 5

Data Debugging

Data: The Foundation of ML Success and Failure

In machine learning, data is destiny . No amount of architectural innovation, hyperparameter tuning, or training tricks can compensate for fundamentally broken data. Yet data debugging is often the most neglected aspect of ML practice—engineers spend hours tweaking model architectures while obvious data issues silently sabotage their efforts. Data debugging is challenging because: Data issues are silent : Unlike code bugs that raise exceptions, data problems often produce models that simply underperform Data is high-dimensional : With millions of examples across dozens of features, manual inspection is impossible Ground truth may be wrong : Labels themselves can be corrupted, mislabeled, or inconsistent Issues compound : Small data problems early in the pipeline cascade into large problems downstream

What You Will Master

This page teaches you to systematically diagnose data problems: detecting corrupted samples, identifying label noise, uncovering data leakage, handling distribution shift, and validating data pipelines. You'll develop the discipline to treat data debugging as the first priority, not an afterthought.

Understanding Data Quality Dimensions

Data quality is multi-dimensional. A dataset can be excellent on some dimensions while catastrophically broken on others. Understanding these dimensions provides a framework for systematic data debugging. The six dimensions of data quality for ML: Completeness : Are all expected values present, or are there missing entries? Correctness : Do values accurately represent reality? Are labels accurate? Consistency : Are similar entities represented the same way across the dataset? Timeliness : Is the data current enough for your use case? Representativeness : Does training data reflect production distribution? Uniqueness : Are there unwanted duplicates that bias training?

Data Quality Issues and Their ML Impact
Quality Dimension	Common Issues	ML Impact	Detection Method
Completeness	Missing values, truncated records	Biased predictions, training failure	Null counts, schema validation
Correctness	Wrong labels, sensor errors, typos	Model learns wrong patterns	Label audit, outlier detection
Consistency	Same entity, different representations	Model treats same thing as different	Deduplication analysis, entity resolution
Timeliness	Stale data, temporal mismatch	Model learns outdated patterns	Timestamp analysis, freshness metrics
Representativeness	Sampling bias, covariate shift	Poor generalization to production	Distribution comparison, stratification analysis
Uniqueness	Duplicate records, data leakage	Overfitting, inflated metrics	Hash-based deduplication, train/test overlap check

The 80/20 Rule of ML Debugging

Experienced ML engineers estimate that 80% of ML project time should go to data—understanding it, cleaning it, and ensuring quality. Yet most practitioners spend 80% on models. Flip this ratio for dramatically better outcomes.

Detecting Data Corruption and Anomalies

Data corruption can occur at any stage: during collection, storage, transfer, or preprocessing. Corrupted data might manifest as: Impossible values : Negative ages, dates in the future, prices of zero Encoding errors : Garbled text from charset mismatches, byte-order issues Truncation : Fields cut off mid-value, file read failures Type mismatches : Strings where numbers expected, or vice versa Structural corruption : Misaligned columns, merged records, dropped fields

data_corruption_detection.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
import pandas as pd
import numpy as np
from typing import Dict, List, Any
 
class DataCorruptionDetector:
    """Systematic detection of data corruption patterns."""
    
    def __init__(self, df: pd.DataFrame):
        self.df = df
        self.issues = []
    
    def check_all(self) -> List[Dict[str, Any]]:
        """Run all corruption checks."""
        self.check_missing_values()
        self.check_type_consistency()
        self.check_value_ranges()
        self.check_duplicates()
        self.check_cardinality()
        return self.issues
    
    def check_missing_values(self):
        """Detect missing value patterns."""
        for col in self.df.columns:
            missing_rate = self.df[col].isna().mean()
            if missing_rate > 0:
                severity = "critical" if missing_rate > 0.5 else \
                          "warning" if missing_rate > 0.1 else "info"
                self.issues.append({
                    'type': 'missing_values',
                    'column': col,
                    'missing_rate': f"{missing_rate:.1%}",
                    'severity': severity
                })
    
    def check_type_consistency(self):
        """Check for mixed types within columns."""
        for col in self.df.select_dtypes(include=['object']).columns:
            # Sample types in the column
            type_counts = self.df[col].dropna().apply(
                lambda x: type(x).__name__
            ).value_counts()
            if len(type_counts) > 1:
                self.issues.append({
                    'type': 'mixed_types',
                    'column': col,
                    'types_found': type_counts.to_dict(),
                    'severity': 'warning'
                })
    
    def check_value_ranges(self):
        """Detect statistical outliers."""
        for col in self.df.select_dtypes(include=[np.number]).columns:
            q1, q3 = self.df[col].quantile([0.25, 0.75])
            iqr = q3 - q1
            lower, upper = q1 - 3*iqr, q3 + 3*iqr
            outliers = ((self.df[col] < lower) | (self.df[col] > upper)).sum()
            if outliers > 0:
                self.issues.append({
                    'type': 'outliers',
                    'column': col,
                    'outlier_count': outliers,
                    'bounds': (lower, upper),
                    'severity': 'warning' if outliers < len(self.df)*0.01 else 'critical'
                })
    
    def check_duplicates(self):
        """Detect duplicate records."""
        dup_count = self.df.duplicated().sum()
        if dup_count > 0:
            self.issues.append({
                'type': 'duplicates',
                'duplicate_count': dup_count,
                'duplicate_rate': f"{dup_count/len(self.df):.1%}",
                'severity': 'warning'
            })
    
    def generate_report(self) -> str:
        """Generate human-readable report."""
        if not self.issues:
            return "✓ No data corruption issues detected"
        
        report = "=== Data Corruption Report ===
 
"
        for issue in sorted(self.issues, key=lambda x: 
                          {'critical': 0, 'warning': 1, 'info': 2}[x['severity']]):
            icon = {'critical': '🚨', 'warning': '⚠️', 'info': 'ℹ️'}[issue['severity']]
            report += f"{icon} {issue['type'].upper()}: {issue}
"
        return report

Label Quality and Label Noise

Label noise —incorrect or inconsistent ground truth labels—is endemic in real-world datasets. Studies show that even carefully curated datasets like ImageNet contain 5-10% label errors. For datasets labeled by crowdworkers or automated systems, error rates can exceed 20%. Sources of label noise: Annotator error : Human mistakes, fatigue, ambiguous guidelines Subjective tasks : Sentiment, quality ratings, aesthetic judgments vary by annotator Systematic bias : Annotators from one demographic mislabel examples from another Automation errors : Weak supervision, programmatic labeling, proxy labels Temporal drift : Labels correct at collection time become wrong as reality changes

label_noise_detection.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
import numpy as np
from sklearn.model_selection import cross_val_predict
from cleanlab.filter import find_label_issues
 
def detect_label_errors(X, y, model, method='confident_learning'):
    """
    Detect potentially mislabeled examples using confident learning.
    
    Confident learning identifies labels that are likely wrong by
    analyzing the disagreement between model predictions and labels.
    """
    if method == 'confident_learning':
        # Get out-of-fold predicted probabilities
        pred_probs = cross_val_predict(
            model, X, y, 
            cv=5, 
            method='predict_proba'
        )
        
        # Find label issues using cleanlab
        label_issues = find_label_issues(
            labels=y,
            pred_probs=pred_probs,
            return_indices_ranked_by='self_confidence'
        )
        return label_issues
    
    elif method == 'loss_based':
        # High-loss examples after training are suspicious
        model.fit(X, y)
        pred_probs = model.predict_proba(X)
        
        # Cross-entropy loss per sample
        losses = -np.log(pred_probs[np.arange(len(y)), y] + 1e-10)
        
        # Top 5% highest loss are suspicious
        threshold = np.percentile(losses, 95)
        suspicious_indices = np.where(losses > threshold)[0]
        return suspicious_indices
 
def compute_label_consistency(annotations_matrix):
    """
    Compute inter-annotator agreement for multi-annotated data.
    
    annotations_matrix: shape (n_samples, n_annotators)
    Values are labels from each annotator, NaN if annotator didn't label.
    """
    n_samples = len(annotations_matrix)
    consistencies = []
    
    for i in range(n_samples):
        labels = annotations_matrix[i][~np.isnan(annotations_matrix[i])]
        if len(labels) > 1:
            # Fraction of annotators who agree with majority
            majority = np.argmax(np.bincount(labels.astype(int)))
            agreement = (labels == majority).mean()
            consistencies.append(agreement)
    
    return np.array(consistencies)

The Label Noise Trap

Models trained on noisy labels often memorize the noise, especially when trained too long. Early stopping helps, but better is to identify and correct label errors. Even removing 2% of the noisiest labels can improve model accuracy more than any hyperparameter tuning.

Data Leakage: The Silent Performance Killer

Data leakage occurs when information from outside the training set improperly influences model training or evaluation. It's the most dangerous data bug because it makes models appear excellent during development while failing catastrophically in production. Types of data leakage: Target leakage : Features that encode or proxy for the target variable Train-test contamination : Test examples appearing in training (directly or via related records) Temporal leakage : Using future information to predict the past Feature engineering leakage : Computing features using global statistics (including test data)

Common Data Leakage Patterns
Leakage Type	Example	Why It's Dangerous	Prevention
Target leakage	Using 'days_in_hospital' to predict 'will_be_hospitalized'	Feature directly encodes outcome	Audit feature creation logic
Train-test contamination	Same user in both train and test sets	Model memorizes user patterns	Split by user ID, not records
Temporal leakage	Using 2024 data to predict 2023 events	Future info unavailable at prediction time	Strict temporal splits
Preprocessing leakage	Fitting StandardScaler on all data before split	Test statistics influence training	Fit only on training data

leakage_detection.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
 
def check_train_test_overlap(train_df, test_df, key_columns):
    """Check for record overlap between train and test sets."""
    train_keys = set(train_df[key_columns].apply(tuple, axis=1))
    test_keys = set(test_df[key_columns].apply(tuple, axis=1))
    
    overlap = train_keys & test_keys
    if overlap:
        print(f"🚨 LEAKAGE: {len(overlap)} records in both train and test!")
        return list(overlap)[:10]  # Return sample
    print("✓ No train-test overlap detected")
    return None
 
def check_feature_target_correlation(df, target_col, threshold=0.95):
    """Detect features suspiciously correlated with target."""
    suspicious = []
    for col in df.columns:
        if col == target_col:
            continue
        if df[col].dtype in ['int64', 'float64']:
            corr = df[col].corr(df[target_col])
            if abs(corr) > threshold:
                suspicious.append((col, corr))
                print(f"⚠️ POTENTIAL LEAKAGE: {col} correlation = {corr:.3f}")
    return suspicious
 
def check_temporal_leakage(df, timestamp_col, feature_cols):
    """Check if features use future information."""
    issues = []
    # Features derived from timestamps should not reference future
    for col in feature_cols:
        if 'future' in col.lower() or 'next' in col.lower():
            issues.append(f"Suspicious feature name: {col}")
    
    # Check if any feature creation uses forward-looking windows
    # (This requires domain knowledge to implement fully)
    return issues

Distribution Shift and Covariate Drift

Distribution shift occurs when the data distribution at prediction time differs from training time. This is perhaps the most common cause of ML model degradation in production. Types of distribution shift: Covariate shift : P(X) changes but P(Y|X) stays the same Label shift : P(Y) changes but P(X|Y) stays the same Concept drift : P(Y|X) itself changes over time Domain shift : Related but distinct domains have different distributions

Shift Warning Signs

•Model accuracy degrades over time
•Prediction distribution differs from training targets
•Feature distributions differ between train and serve
•Model confidence drops on recent data
•Known ground truth disagrees with predictions

Mitigation Strategies

•Continuous monitoring of feature distributions
•Regular retraining on fresh data
•Sample weighting for distribution matching
•Domain adaptation techniques
•Ensemble of models from different time periods

Monitoring is Prevention

The best defense against distribution shift is proactive monitoring. Log feature distributions at training time, then compare against production data continuously. Statistical tests (KS test, PSI) can automatically alert you to significant drift before model accuracy suffers.

Data Pipeline Validation

Data pipelines are complex systems that can fail in subtle ways. A pipeline that worked yesterday may silently break today due to upstream changes, schema drift, or infrastructure issues. Robust validation catches issues before they corrupt your models.

Data Pipeline Validation Checklist

•Schema validation: Verify column names, types, and constraints match expectations
•Completeness checks: Assert no unexpected nulls, verify row counts are in expected range
•Freshness validation: Confirm data timestamps are recent, not stale
•Distribution checks: Statistical tests comparing current batch to historical baseline
•Referential integrity: Foreign keys resolve, no orphaned records
•Business rule validation: Domain-specific constraints (prices > 0, ages in valid range)
•End-to-end tests: Known inputs should produce known outputs

Key Takeaways

Data debugging is the highest-leverage activity in ML. Before tuning hyperparameters or adding model complexity, ensure your data is clean, correctly labeled, free of leakage, and representative of production. Invest in automated data validation—it pays dividends every time your pipeline runs.

Machine LearningML Engineering & Career

Debugging ML Models

LevelAdvanced

Duration90 mins

TopicML Engineering & Career

2 / 5

Data Debugging

Data: The Foundation of ML Success and Failure

In machine learning, data is destiny . No amount of architectural innovation, hyperparameter tuning, or training tricks can compensate for fundamentally broken data. Yet data debugging is often the most neglected aspect of ML practice—engineers spend hours tweaking model architectures while obvious data issues silently sabotage their efforts. Data debugging is challenging because: Data issues are silent : Unlike code bugs that raise exceptions, data problems often produce models that simply underperform Data is high-dimensional : With millions of examples across dozens of features, manual inspection is impossible Ground truth may be wrong : Labels themselves can be corrupted, mislabeled, or inconsistent Issues compound : Small data problems early in the pipeline cascade into large problems downstream

What You Will Master

This page teaches you to systematically diagnose data problems: detecting corrupted samples, identifying label noise, uncovering data leakage, handling distribution shift, and validating data pipelines. You'll develop the discipline to treat data debugging as the first priority, not an afterthought.

Understanding Data Quality Dimensions

Data quality is multi-dimensional. A dataset can be excellent on some dimensions while catastrophically broken on others. Understanding these dimensions provides a framework for systematic data debugging. The six dimensions of data quality for ML: Completeness : Are all expected values present, or are there missing entries? Correctness : Do values accurately represent reality? Are labels accurate? Consistency : Are similar entities represented the same way across the dataset? Timeliness : Is the data current enough for your use case? Representativeness : Does training data reflect production distribution? Uniqueness : Are there unwanted duplicates that bias training?

Data Quality Issues and Their ML Impact
Quality Dimension	Common Issues	ML Impact	Detection Method
Completeness	Missing values, truncated records	Biased predictions, training failure	Null counts, schema validation
Correctness	Wrong labels, sensor errors, typos	Model learns wrong patterns	Label audit, outlier detection
Consistency	Same entity, different representations	Model treats same thing as different	Deduplication analysis, entity resolution
Timeliness	Stale data, temporal mismatch	Model learns outdated patterns	Timestamp analysis, freshness metrics
Representativeness	Sampling bias, covariate shift	Poor generalization to production	Distribution comparison, stratification analysis
Uniqueness	Duplicate records, data leakage	Overfitting, inflated metrics	Hash-based deduplication, train/test overlap check

The 80/20 Rule of ML Debugging

Experienced ML engineers estimate that 80% of ML project time should go to data—understanding it, cleaning it, and ensuring quality. Yet most practitioners spend 80% on models. Flip this ratio for dramatically better outcomes.

Detecting Data Corruption and Anomalies

Data corruption can occur at any stage: during collection, storage, transfer, or preprocessing. Corrupted data might manifest as: Impossible values : Negative ages, dates in the future, prices of zero Encoding errors : Garbled text from charset mismatches, byte-order issues Truncation : Fields cut off mid-value, file read failures Type mismatches : Strings where numbers expected, or vice versa Structural corruption : Misaligned columns, merged records, dropped fields

data_corruption_detection.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
import pandas as pd
import numpy as np
from typing import Dict, List, Any
 
class DataCorruptionDetector:
    """Systematic detection of data corruption patterns."""
    
    def __init__(self, df: pd.DataFrame):
        self.df = df
        self.issues = []
    
    def check_all(self) -> List[Dict[str, Any]]:
        """Run all corruption checks."""
        self.check_missing_values()
        self.check_type_consistency()
        self.check_value_ranges()
        self.check_duplicates()
        self.check_cardinality()
        return self.issues
    
    def check_missing_values(self):
        """Detect missing value patterns."""
        for col in self.df.columns:
            missing_rate = self.df[col].isna().mean()
            if missing_rate > 0:
                severity = "critical" if missing_rate > 0.5 else \
                          "warning" if missing_rate > 0.1 else "info"
                self.issues.append({
                    'type': 'missing_values',
                    'column': col,
                    'missing_rate': f"{missing_rate:.1%}",
                    'severity': severity
                })
    
    def check_type_consistency(self):
        """Check for mixed types within columns."""
        for col in self.df.select_dtypes(include=['object']).columns:
            # Sample types in the column
            type_counts = self.df[col].dropna().apply(
                lambda x: type(x).__name__
            ).value_counts()
            if len(type_counts) > 1:
                self.issues.append({
                    'type': 'mixed_types',
                    'column': col,
                    'types_found': type_counts.to_dict(),
                    'severity': 'warning'
                })
    
    def check_value_ranges(self):
        """Detect statistical outliers."""
        for col in self.df.select_dtypes(include=[np.number]).columns:
            q1, q3 = self.df[col].quantile([0.25, 0.75])
            iqr = q3 - q1
            lower, upper = q1 - 3*iqr, q3 + 3*iqr
            outliers = ((self.df[col] < lower) | (self.df[col] > upper)).sum()
            if outliers > 0:
                self.issues.append({
                    'type': 'outliers',
                    'column': col,
                    'outlier_count': outliers,
                    'bounds': (lower, upper),
                    'severity': 'warning' if outliers < len(self.df)*0.01 else 'critical'
                })
    
    def check_duplicates(self):
        """Detect duplicate records."""
        dup_count = self.df.duplicated().sum()
        if dup_count > 0:
            self.issues.append({
                'type': 'duplicates',
                'duplicate_count': dup_count,
                'duplicate_rate': f"{dup_count/len(self.df):.1%}",
                'severity': 'warning'
            })
    
    def generate_report(self) -> str:
        """Generate human-readable report."""
        if not self.issues:
            return "✓ No data corruption issues detected"
        
        report = "=== Data Corruption Report ===
 
"
        for issue in sorted(self.issues, key=lambda x: 
                          {'critical': 0, 'warning': 1, 'info': 2}[x['severity']]):
            icon = {'critical': '🚨', 'warning': '⚠️', 'info': 'ℹ️'}[issue['severity']]
            report += f"{icon} {issue['type'].upper()}: {issue}
"
        return report

Label Quality and Label Noise

Label noise —incorrect or inconsistent ground truth labels—is endemic in real-world datasets. Studies show that even carefully curated datasets like ImageNet contain 5-10% label errors. For datasets labeled by crowdworkers or automated systems, error rates can exceed 20%. Sources of label noise: Annotator error : Human mistakes, fatigue, ambiguous guidelines Subjective tasks : Sentiment, quality ratings, aesthetic judgments vary by annotator Systematic bias : Annotators from one demographic mislabel examples from another Automation errors : Weak supervision, programmatic labeling, proxy labels Temporal drift : Labels correct at collection time become wrong as reality changes

label_noise_detection.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
import numpy as np
from sklearn.model_selection import cross_val_predict
from cleanlab.filter import find_label_issues
 
def detect_label_errors(X, y, model, method='confident_learning'):
    """
    Detect potentially mislabeled examples using confident learning.
    
    Confident learning identifies labels that are likely wrong by
    analyzing the disagreement between model predictions and labels.
    """
    if method == 'confident_learning':
        # Get out-of-fold predicted probabilities
        pred_probs = cross_val_predict(
            model, X, y, 
            cv=5, 
            method='predict_proba'
        )
        
        # Find label issues using cleanlab
        label_issues = find_label_issues(
            labels=y,
            pred_probs=pred_probs,
            return_indices_ranked_by='self_confidence'
        )
        return label_issues
    
    elif method == 'loss_based':
        # High-loss examples after training are suspicious
        model.fit(X, y)
        pred_probs = model.predict_proba(X)
        
        # Cross-entropy loss per sample
        losses = -np.log(pred_probs[np.arange(len(y)), y] + 1e-10)
        
        # Top 5% highest loss are suspicious
        threshold = np.percentile(losses, 95)
        suspicious_indices = np.where(losses > threshold)[0]
        return suspicious_indices
 
def compute_label_consistency(annotations_matrix):
    """
    Compute inter-annotator agreement for multi-annotated data.
    
    annotations_matrix: shape (n_samples, n_annotators)
    Values are labels from each annotator, NaN if annotator didn't label.
    """
    n_samples = len(annotations_matrix)
    consistencies = []
    
    for i in range(n_samples):
        labels = annotations_matrix[i][~np.isnan(annotations_matrix[i])]
        if len(labels) > 1:
            # Fraction of annotators who agree with majority
            majority = np.argmax(np.bincount(labels.astype(int)))
            agreement = (labels == majority).mean()
            consistencies.append(agreement)
    
    return np.array(consistencies)

The Label Noise Trap

Models trained on noisy labels often memorize the noise, especially when trained too long. Early stopping helps, but better is to identify and correct label errors. Even removing 2% of the noisiest labels can improve model accuracy more than any hyperparameter tuning.

Data Leakage: The Silent Performance Killer

Data leakage occurs when information from outside the training set improperly influences model training or evaluation. It's the most dangerous data bug because it makes models appear excellent during development while failing catastrophically in production. Types of data leakage: Target leakage : Features that encode or proxy for the target variable Train-test contamination : Test examples appearing in training (directly or via related records) Temporal leakage : Using future information to predict the past Feature engineering leakage : Computing features using global statistics (including test data)

Common Data Leakage Patterns
Leakage Type	Example	Why It's Dangerous	Prevention
Target leakage	Using 'days_in_hospital' to predict 'will_be_hospitalized'	Feature directly encodes outcome	Audit feature creation logic
Train-test contamination	Same user in both train and test sets	Model memorizes user patterns	Split by user ID, not records
Temporal leakage	Using 2024 data to predict 2023 events	Future info unavailable at prediction time	Strict temporal splits
Preprocessing leakage	Fitting StandardScaler on all data before split	Test statistics influence training	Fit only on training data

leakage_detection.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
 
def check_train_test_overlap(train_df, test_df, key_columns):
    """Check for record overlap between train and test sets."""
    train_keys = set(train_df[key_columns].apply(tuple, axis=1))
    test_keys = set(test_df[key_columns].apply(tuple, axis=1))
    
    overlap = train_keys & test_keys
    if overlap:
        print(f"🚨 LEAKAGE: {len(overlap)} records in both train and test!")
        return list(overlap)[:10]  # Return sample
    print("✓ No train-test overlap detected")
    return None
 
def check_feature_target_correlation(df, target_col, threshold=0.95):
    """Detect features suspiciously correlated with target."""
    suspicious = []
    for col in df.columns:
        if col == target_col:
            continue
        if df[col].dtype in ['int64', 'float64']:
            corr = df[col].corr(df[target_col])
            if abs(corr) > threshold:
                suspicious.append((col, corr))
                print(f"⚠️ POTENTIAL LEAKAGE: {col} correlation = {corr:.3f}")
    return suspicious
 
def check_temporal_leakage(df, timestamp_col, feature_cols):
    """Check if features use future information."""
    issues = []
    # Features derived from timestamps should not reference future
    for col in feature_cols:
        if 'future' in col.lower() or 'next' in col.lower():
            issues.append(f"Suspicious feature name: {col}")
    
    # Check if any feature creation uses forward-looking windows
    # (This requires domain knowledge to implement fully)
    return issues

Distribution Shift and Covariate Drift

Distribution shift occurs when the data distribution at prediction time differs from training time. This is perhaps the most common cause of ML model degradation in production. Types of distribution shift: Covariate shift : P(X) changes but P(Y|X) stays the same Label shift : P(Y) changes but P(X|Y) stays the same Concept drift : P(Y|X) itself changes over time Domain shift : Related but distinct domains have different distributions

Shift Warning Signs

•Model accuracy degrades over time
•Prediction distribution differs from training targets
•Feature distributions differ between train and serve
•Model confidence drops on recent data
•Known ground truth disagrees with predictions

Mitigation Strategies

•Continuous monitoring of feature distributions
•Regular retraining on fresh data
•Sample weighting for distribution matching
•Domain adaptation techniques
•Ensemble of models from different time periods

Monitoring is Prevention

The best defense against distribution shift is proactive monitoring. Log feature distributions at training time, then compare against production data continuously. Statistical tests (KS test, PSI) can automatically alert you to significant drift before model accuracy suffers.

Data Pipeline Validation

Data pipelines are complex systems that can fail in subtle ways. A pipeline that worked yesterday may silently break today due to upstream changes, schema drift, or infrastructure issues. Robust validation catches issues before they corrupt your models.

Data Pipeline Validation Checklist

•Schema validation: Verify column names, types, and constraints match expectations
•Completeness checks: Assert no unexpected nulls, verify row counts are in expected range
•Freshness validation: Confirm data timestamps are recent, not stale
•Distribution checks: Statistical tests comparing current batch to historical baseline
•Referential integrity: Foreign keys resolve, no orphaned records
•Business rule validation: Domain-specific constraints (prices > 0, ages in valid range)
•End-to-end tests: Known inputs should produce known outputs

Key Takeaways

Data debugging is the highest-leverage activity in ML. Before tuning hyperparameters or adding model complexity, ensure your data is clean, correctly labeled, free of leakage, and representative of production. Invest in automated data validation—it pays dividends every time your pipeline runs.

Data Debugging

Data: The Foundation of ML Success and Failure

Understanding Data Quality Dimensions

Detecting Data Corruption and Anomalies

Label Quality and Label Noise

Data Leakage: The Silent Performance Killer

Distribution Shift and Covariate Drift

Data Pipeline Validation

There’s more to understand.

Data Debugging

Data: The Foundation of ML Success and Failure

Understanding Data Quality Dimensions

Detecting Data Corruption and Anomalies

Label Quality and Label Noise

Data Leakage: The Silent Performance Killer

Distribution Shift and Covariate Drift

Data Pipeline Validation

There’s more to understand.