Documentation claims, stakeholder promises, and regulatory commitments are only as good as the mechanisms that verify them. Auditing provides systematic, evidence-based evaluation of ML systems against requirements, standards, and expectations. It transforms aspirational statements into verified facts.

ML auditing has emerged as a critical practice for several reasons:

• Regulatory mandates: Laws increasingly require third-party assessments of AI systems
• Trust requirements: Stakeholders demand verification beyond self-reporting
• Complexity: ML systems are difficult to assess through traditional testing
• Evolving behavior: Models that passed initial testing may degrade or develop problems over time
• High stakes: Errors in ML systems can harm individuals and organizations at scale

This page examines auditing comprehensively: frameworks and methodologies, internal and external audit practices, technical auditing approaches, fairness-specific auditing, and the role of continuous monitoring as ongoing audit.

What is ML Auditing?

ML auditing is the systematic evaluation of machine learning systems to assess:

• Compliance: Does the system meet legal and regulatory requirements?
• Performance: Does the system meet stated performance claims?
• Fairness: Does the system treat different groups equitably?
• Safety: Does the system avoid unacceptable harms?
• Documentation: Is the system adequately documented?
• Governance: Are appropriate controls and oversight in place?

Types of AI/ML Audits:

The Audit Cycle:

Effective auditing follows a systematic cycle:

1. PLAN → Define scope, criteria, methodology
     ↓
2. GATHER → Collect evidence (documentation, testing, interviews)
     ↓
3. ANALYZE → Evaluate evidence against criteria
     ↓
4. REPORT → Document findings, severity, recommendations
     ↓
5. REMEDIATE → Address identified issues
     ↓
6. FOLLOW-UP → Verify remediation effectiveness
     ↓
[Return to 1. PLAN for next cycle]

Audit Independence:

Audit value depends heavily on independence:

Several frameworks provide structure for ML auditing. Understanding these enables systematic assessment and alignment with industry practices.

Whether conducting or supporting an audit, understanding the audit process enables effective participation.

Pre-Audit Phase:

Evidence Gathering:

ML audits rely on multiple evidence types:

The Evidence Triangle:

Strong audits triangulate evidence—claims should be verified through multiple sources:

                Documentation
                    /  \\
                   /    \\
                  /      \\
                 /        \\
          Interviews ---- Testing

If documentation says one thing, testing shows another, and interviews reveal a third, something is wrong. Convergence increases confidence; divergence demands investigation.

Technical auditing evaluates the actual behavior of ML systems through systematic testing, analysis, and review.

Performance Auditing:

Fairness Auditing (Detailed):

Fairness auditing requires systematic analysis across protected groups:

Step 1: Define Protected Groups

• Legal protected classes (race, sex, age, etc.)
• Groups relevant to specific domain
• Intersections of groups (race × sex)

Step 2: Obtain Group Information

• Direct data (if legally collected)
• Proxy estimation (BISG for race, first name for sex)
• Synthetic testing (create matched test cases)

Step 3: Calculate Disparities

Step 4: Investigate Causes

• Feature analysis: Which features drive group differences?
• Proxy feature investigation: Are features proxies for protected attributes?
• Data bias analysis: Does training data reflect historical discrimination?
• Labeling bias analysis: Were labels created fairly across groups?

Step 5: Document and Report

• Metrics and statistical significance
• Comparison to industry benchmarks
• Root cause hypotheses
• Remediation recommendations
• Residual risk assessment

Code Review Auditing:

Code review catches implementation errors that testing might miss:

• Preprocessing Logic: Is preprocessing applied consistently to training and inference data?
• Feature Engineering: Are features calculated correctly? Any leakage?
• Model Loading: Is the deployed model the validated version?
• Threshold Implementation: Is the decision threshold applied correctly?
• Error Handling: What happens when inputs are unexpected?
• Logging: Is sufficient information logged for monitoring and audit?

Audit findings must be clearly communicated, appropriately classified, and systematically remediated.

Finding Classification:

Finding Report Structure:

## Finding F-2024-042: Disparate Impact in Model Scoring

**Severity:** High
**System:** Credit scoring model v3.2
**Criteria:** ECOA/Reg B adverse impact ratio > 0.80

### Condition (What We Found)
Analysis of Q4 2023 decisions shows approval rate for Hispanic applicants 
(67.2%) is 76.3% of approval rate for non-Hispanic white applicants (88.1%), 
yielding an adverse impact ratio of 0.76. This falls below the 0.80 threshold 
generally considered evidence of disparate impact.

### Cause (Why It Exists)
Feature analysis indicates that geographic features (ZIP code density, 
median home value) contribute 23% of model score variance and correlate 
strongly with ethnicity due to historical residential segregation patterns.

### Effect (Why It Matters)
- Potential violation of ECOA/Reg B fair lending requirements
- Risk of regulatory enforcement action
- Harm to Hispanic applicants receiving disproportionate denials
- Reputational risk if disparity becomes public

### Recommendation
1. Immediate: Conduct legal review of disparate impact finding
2. Short-term: Evaluate removal or modification of geographic features
3. Medium-term: Retrain model with fairness constraints
4. Ongoing: Implement continuous fair lending monitoring

### Management Response
[To be completed by management]

Remediation Tracking:

remediation should be tracked to closure:

External audits provide independence and credibility that internal audits cannot. They're increasingly required by regulation and expected by stakeholders.

When External Audits Are Required/Appropriate:

Engaging External Auditors:

Selection Criteria:

Due Diligence Questions:

1. What is your experience auditing similar systems/industries?
2. What methodology do you follow? Is it documented?
3. Who will perform the audit (specific individuals and their qualifications)?
4. What access and data do you require?
5. What assurances/certifications do your findings provide?
6. What is your policy on conflicts of interest?
7. What professional liability coverage do you carry?

Managing External Audits:

Point-in-time audits are insufficient for ML systems that evolve and operate continuously. Continuous monitoring extends audit into day-to-day operations.

The Monitoring Audit Loop:

                    ┌─────────────┐
      ┌────────────>│  Deploy     │
      │             └──────┬──────┘
      │                    │
      │                    v
      │             ┌─────────────┐
      │             │   Monitor   │───> Dashboards, Alerts
      │             └──────┬──────┘
      │                    │
  Remediate               v
      │             ┌─────────────┐
      │             │   Analyze   │───> Drift, Performance, Fairness
      │             └──────┬──────┘
      │                    │
      │                    v
      │             ┌─────────────┐
      └─────────────│   Report    │───> Periodic Review, Escalation
                    └─────────────┘

What to Monitor:

Drift Detection:

Models can degrade through drift even without code changes:

Data Drift: Input distributions shift from training distribution

• Population changes (customer base evolves)
• Feature meaning changes (economy affects income patterns)
• Data collection changes (new forms, new sources)

Detection: Statistical tests (KS test, PSI), distance metrics

Concept Drift: Relationship between inputs and outputs changes

• Environment changes (pandemic changes fraud patterns)
• User behavior changes (new shopping habits)
• Regulatory changes (new lending rules)

Detection: Performance monitoring on labeled samples, challenge datasets

Upstream Data Drift: Changes in systems feeding model inputs

• Pipeline changes (new ETL, different preprocessing)
• Source system changes (new database schema)
• Data quality changes (vendor data quality issues)

Detection: Schema checks, value range checks, freshness checks

Fairness Drift: Disparities emerge or worsen over time

• Population shifts (demographic changes in applicants)
• Economic changes (differential impact across groups)
• Model degradation differs by group

Detection: Regular disaggregated analysis, automated fairness metric calculation

Organizations that prepare for audits continuously—not just when audits are imminent—experience smoother audits with better outcomes.

Audit Readiness Checklist:

Developing Audit Muscle:

Audit readiness improves with practice:

1. Periodic Self-Assessment

• Conduct internal mock audits using external frameworks
• Identify gaps before external auditors do
• Build familiarity with audit process

2. Lessons Learned Integration

• After each audit, document what worked and what didn't
• Update processes based on auditor feedback
• Share learning across teams

3. Continuous Evidence Collection

• Automate collection of audit-relevant data
• Maintain running logs rather than reconstructing
• Index and catalog artifacts as they're created

4. Designated Audit Liaisons

• Trained individuals who understand audit processes
• Single point of contact for auditor requests
• Coordinate across teams efficiently

Common Audit Readiness Failures:

Auditing provides systematic, evidence-based assurance that ML systems meet requirements, perform as claimed, and operate fairly. It transforms interpretability commitments from aspirations to verified facts. Let's consolidate the key insights:

Module Complete:

This concludes Module 6: Practical Interpretability. You have learned how to communicate with diverse stakeholders, navigate regulatory requirements, create Model Cards and comprehensive documentation, and conduct rigorous audits. Together, these practices transform interpretability from a technical capability into an organizational competency that builds trust, ensures compliance, and enables responsible AI deployment.

The Practical Interpretability Toolkit: