Profiling and Monitoring

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

# Python Benchmark Example with pytest-benchmark
 
import pytest
from myapp.serializers import serialize_user, serialize_user_optimized
 
 
def test_serialize_user_performance(benchmark):
    """
    Benchmark user serialization performance.
    Runs multiple iterations and reports statistics.
    """
    user = create_test_user()  # Sample data
    
    result = benchmark(serialize_user, user)
    
    # Optional: assert performance bounds
    # Fails test if mean exceeds threshold
    assert benchmark.stats['mean'] < 0.001  # < 1ms
    
 
def test_serialize_user_optimized_performance(benchmark):
    """
    Benchmark optimized implementation for comparison.
    """
    user = create_test_user()
    
    result = benchmark(serialize_user_optimized, user)
    
    assert benchmark.stats['mean'] < 0.0001  # < 0.1ms (10x improvement)
 
 
# Output example:
# ------------------- benchmark: 2 tests -------------------
# Name                                   Min      Max     Mean    StdDev   Median
# test_serialize_user_performance       0.8ms    1.2ms   0.95ms   0.08ms   0.93ms
# test_serialize_user_optimized         0.05ms   0.12ms  0.08ms   0.01ms   0.07ms
 
# CI Integration: pytest-benchmark can save results to JSON
# Later runs can compare against baseline:
# pytest --benchmark-autosave --benchmark-compare

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75

// k6 Load Test Example
// k6 is a modern load testing tool with excellent CI/CD integration
 
import http from 'k6/http';
import { check, sleep } from 'k6';
import { Rate, Trend } from 'k6/metrics';
 
// Custom metrics
const errorRate = new Rate('errors');
const latency = new Trend('latency');
 
// Test configuration
export const options = {
    // Ramp-up pattern simulating realistic traffic
    stages: [
        { duration: '2m', target: 100 },   // Ramp up to 100 users
        { duration: '5m', target: 100 },   // Sustain 100 users
        { duration: '2m', target: 200 },   // Ramp up to 200 users
        { duration: '5m', target: 200 },   // Sustain 200 users
        { duration: '2m', target: 0 },     // Ramp down
    ],
    
    // Thresholds for pass/fail in CI
    thresholds: {
        // 95th percentile response time < 500ms
        'http_req_duration': ['p(95)<500'],
        
        // 99th percentile response time < 1500ms
        'http_req_duration': ['p(99)<1500'],
        
        // Error rate < 1%
        'errors': ['rate<0.01'],
        
        // Latency custom metric
        'latency': ['p(95)<400'],
    },
};
 
// Main test scenario
export default function () {
    // Simulate user flow
    const baseUrl = __ENV.BASE_URL || 'https://staging.example.com';
    
    // 1. Homepage load
    let res = http.get(`${baseUrl}/`);
    check(res, {
        'homepage status is 200': (r) => r.status === 200,
    }) || errorRate.add(1);
    latency.add(res.timings.duration);
    
    sleep(1);
    
    // 2. API call - list products
    res = http.get(`${baseUrl}/api/products?limit=20`);
    check(res, {
        'products API status is 200': (r) => r.status === 200,
        'products returned': (r) => JSON.parse(r.body).length > 0,
    }) || errorRate.add(1);
    latency.add(res.timings.duration);
    
    sleep(2);
    
    // 3. API call - product detail (with database lookup)
    const productId = Math.floor(Math.random() * 1000) + 1;
    res = http.get(`${baseUrl}/api/products/${productId}`);
    check(res, {
        'product detail status is 200': (r) => r.status === 200,
    }) || errorRate.add(1);
    latency.add(res.timings.duration);
    
    sleep(1);
}
 
// Run with: k6 run --out json=results.json load_test.js
// CI integration: Parse results.json for pass/fail

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157

# GitHub Actions: Performance Testing Pipeline
 
name: Performance Tests
 
on:
  pull_request:
    branches: [main, develop]
  push:
    branches: [main]
  schedule:
    # Full load tests nightly
    - cron: '0 2 * * *'
 
jobs:
  # ============================================
  # Stage 1: Fast benchmarks (every PR)
  # ============================================
  benchmark:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Run benchmarks
        run: npm run benchmark -- --json > benchmark-results.json
      
      # Compare against baseline from main branch
      - name: Download baseline
        uses: actions/download-artifact@v4
        with:
          name: benchmark-baseline
          path: baseline/
        continue-on-error: true  # First run won't have baseline
      
      - name: Compare benchmarks
        id: benchmark-compare
        run: |
          node scripts/compare-benchmarks.js             --current benchmark-results.json             --baseline baseline/benchmark-results.json             --threshold 10  # Fail if >10% regression
      
      # Save results as new baseline on main
      - name: Upload benchmark results
        if: github.ref == 'refs/heads/main'
        uses: actions/upload-artifact@v4
        with:
          name: benchmark-baseline
          path: benchmark-results.json
      
      # Comment on PR with results
      - name: Comment PR
        if: github.event_name == 'pull_request'
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const comparison = fs.readFileSync('benchmark-comparison.md', 'utf8');
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: comparison
            });
 
  # ============================================
  # Stage 2: Integration performance tests
  # ============================================
  performance-integration:
    runs-on: ubuntu-latest
    needs: benchmark
    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_PASSWORD: test
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 5432:5432
      redis:
        image: redis:7
        ports:
          - 6379:6379
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup environment
        run: |
          npm ci
          npm run db:migrate
          npm run db:seed
      
      - name: Start application
        run: |
          npm run start:test &
          sleep 10  # Wait for startup
          curl --retry 10 --retry-delay 2 http://localhost:3000/health
      
      - name: Run performance integration tests
        run: |
          k6 run --out json=k6-results.json             --env BASE_URL=http://localhost:3000             tests/performance/integration.js
      
      - name: Analyze results
        run: |
          node scripts/analyze-k6-results.js k6-results.json
      
      - name: Upload results
        uses: actions/upload-artifact@v4
        with:
          name: performance-results
          path: k6-results.json
 
  # ============================================
  # Stage 3: Full load test (nightly)
  # ============================================
  load-test:
    if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - uses: actions/checkout@v4
      
      - name: Run full load test against staging
        run: |
          k6 run --out influxdb=http://metrics.internal:8086/k6             --env BASE_URL=${{ secrets.STAGING_URL }}             tests/performance/full-load.js
      
      - name: Generate report
        run: |
          node scripts/generate-perf-report.js > report.md
      
      - name: Notify on regression
        if: failure()
        uses: slackapi/slack-github-action@v1
        with:
          payload: |
            {
              "text": "⚠️ Performance regression detected in nightly load test",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "*Performance Regression Detected*\nNightly load test failed thresholds."
                  }
                }
              ]
            }

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142

# Statistical Regression Detection
 
import numpy as np
from scipy import stats
from typing import Optional
from dataclasses import dataclass
 
 
@dataclass
class RegressionResult:
    is_regression: bool
    baseline_mean: float
    current_mean: float
    percent_change: float
    p_value: float
    confidence: float
    message: str
 
 
def detect_regression(
    baseline_samples: list[float],
    current_samples: list[float],
    regression_threshold: float = 0.10,  # 10% increase = regression
    confidence_level: float = 0.95,
) -> RegressionResult:
    """
    Statistically detect if performance has regressed.
    
    Uses Welch's t-test to compare means, accounting for
    potentially different variances between runs.
    
    Args:
        baseline_samples: Performance measurements from baseline (e.g., main branch)
        current_samples: Performance measurements from current (e.g., PR)
        regression_threshold: Minimum % increase to consider a regression
        confidence_level: Required confidence for regression detection
        
    Returns:
        RegressionResult with analysis details
    """
    baseline = np.array(baseline_samples)
    current = np.array(current_samples)
    
    baseline_mean = np.mean(baseline)
    current_mean = np.mean(current)
    
    # Calculate percent change
    percent_change = (current_mean - baseline_mean) / baseline_mean
    
    # Welch's t-test (doesn't assume equal variance)
    t_stat, p_value = stats.ttest_ind(current, baseline, equal_var=False)
    
    # Is the change statistically significant AND exceeds threshold?
    significant = p_value < (1 - confidence_level)
    exceeds_threshold = percent_change > regression_threshold
    is_regression = significant and exceeds_threshold
    
    # Generate human-readable message
    if is_regression:
        message = (
            f"🔴 REGRESSION DETECTED: {percent_change:+.1%} increase "
            f"(p={p_value:.4f}, {baseline_mean:.2f}ms → {current_mean:.2f}ms)"
        )
    elif percent_change > regression_threshold:
        message = (
            f"🟡 WARNING: {percent_change:+.1%} increase, but not statistically "
            f"significant (p={p_value:.4f}). Consider more samples."
        )
    elif percent_change < -0.05:  # 5% improvement
        message = (
            f"🟢 IMPROVEMENT: {percent_change:+.1%} decrease "
            f"(p={p_value:.4f}, {baseline_mean:.2f}ms → {current_mean:.2f}ms)"
        )
    else:
        message = (
            f"✅ No significant change: {percent_change:+.1%} "
            f"({baseline_mean:.2f}ms → {current_mean:.2f}ms)"
        )
    
    return RegressionResult(
        is_regression=is_regression,
        baseline_mean=baseline_mean,
        current_mean=current_mean,
        percent_change=percent_change,
        p_value=p_value,
        confidence=confidence_level,
        message=message,
    )
 
 
def detect_regression_with_history(
    historical_runs: list[list[float]],
    current_samples: list[float],
    regression_threshold: float = 0.10,
) -> RegressionResult:
    """
    Compare against a moving baseline of recent runs.
    
    More robust than single-run comparison. Uses the median
    of recent runs to establish baseline.
    """
    # Flatten historical runs and compute robust baseline
    all_historical = np.concatenate(historical_runs)
    
    # Use median and IQR for robustness against outliers
    baseline_median = np.median(all_historical)
    baseline_iqr = stats.iqr(all_historical)
    
    current_median = np.median(current_samples)
    percent_change = (current_median - baseline_median) / baseline_median
    
    # Use Mann-Whitney U test (non-parametric, more robust)
    u_stat, p_value = stats.mannwhitneyu(
        current_samples, all_historical, alternative='greater'
    )
    
    is_regression = (p_value < 0.05) and (percent_change > regression_threshold)
    
    message = f"Median: {baseline_median:.2f}ms → {current_median:.2f}ms ({percent_change:+.1%})"
    
    return RegressionResult(
        is_regression=is_regression,
        baseline_mean=float(baseline_median),
        current_mean=float(current_median),
        percent_change=percent_change,
        p_value=p_value,
        confidence=0.95,
        message=message,
    )
 
 
# Usage example:
# 
# baseline = [102, 98, 105, 101, 99, 103, 100, 97, 104, 101]  # Previous runs
# current = [115, 118, 112, 120, 116, 114, 119, 117, 113, 118]  # Current PR
# 
# result = detect_regression(baseline, current)
# print(result.message)
# # 🔴 REGRESSION DETECTED: +14.2% increase (p=0.0001, 101.00ms → 115.20ms)
#
# if result.is_regression:
#     sys.exit(1)  # Fail CI pipeline

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146

// k6: Realistic Load Test with User Flows
 
import http from 'k6/http';
import { check, group, sleep } from 'k6';
import { SharedArray } from 'k6/data';
 
// Load test data from file (shared across VUs)
const users = new SharedArray('users', function () {
    return JSON.parse(open('./test-data/users.json'));
});
 
const products = new SharedArray('products', function () {
    return JSON.parse(open('./test-data/products.json'));
});
 
export const options = {
    // Scenario-based testing: Different user types
    scenarios: {
        // 70% of traffic: Browsing users (just looking)
        browsers: {
            executor: 'ramping-vus',
            startVUs: 0,
            stages: [
                { duration: '5m', target: 70 },
                { duration: '20m', target: 70 },
                { duration: '5m', target: 0 },
            ],
            exec: 'browserFlow',
        },
        
        // 25% of traffic: Shoppers (add to cart, maybe checkout)
        shoppers: {
            executor: 'ramping-vus',
            startVUs: 0,
            stages: [
                { duration: '5m', target: 25 },
                { duration: '20m', target: 25 },
                { duration: '5m', target: 0 },
            ],
            exec: 'shopperFlow',
        },
        
        // 5% of traffic: Buyers (complete purchase)
        buyers: {
            executor: 'ramping-vus',
            startVUs: 0,
            stages: [
                { duration: '5m', target: 5 },
                { duration: '20m', target: 5 },
                { duration: '5m', target: 0 },
            ],
            exec: 'buyerFlow',
        },
    },
    
    thresholds: {
        'http_req_duration{scenario:browsers}': ['p(95)<300'],
        'http_req_duration{scenario:shoppers}': ['p(95)<500'],
        'http_req_duration{scenario:buyers}': ['p(95)<1000'],  // Checkout can be slower
        'http_req_failed{scenario:buyers}': ['rate<0.01'],     // Buyers must not fail
    },
};
 
// Browser flow: View pages, search, leave
export function browserFlow() {
    group('homepage', function () {
        const res = http.get(`${__ENV.BASE_URL}/`);
        check(res, { 'homepage 2xx': (r) => r.status < 300 });
    });
    
    sleep(randomBetween(2, 5));
    
    group('search', function () {
        const query = ['shoes', 'shirts', 'pants', 'accessories'][Math.floor(Math.random() * 4)];
        const res = http.get(`${__ENV.BASE_URL}/search?q=${query}`);
        check(res, { 'search 2xx': (r) => r.status < 300 });
    });
    
    sleep(randomBetween(3, 8));
    
    group('product_view', function () {
        const product = products[Math.floor(Math.random() * products.length)];
        const res = http.get(`${__ENV.BASE_URL}/products/${product.id}`);
        check(res, { 'product 2xx': (r) => r.status < 300 });
    });
    
    sleep(randomBetween(5, 15));  // Think time
}
 
// Shopper flow: Browse + add to cart
export function shopperFlow() {
    browserFlow();  // Start like a browser
    
    group('add_to_cart', function () {
        const product = products[Math.floor(Math.random() * products.length)];
        const res = http.post(`${__ENV.BASE_URL}/cart/add`, {
            product_id: product.id,
            quantity: 1,
        });
        check(res, { 'add to cart 2xx': (r) => r.status < 300 });
    });
    
    sleep(randomBetween(5, 30));  // Decide whether to buy
}
 
// Buyer flow: Complete purchase
export function buyerFlow() {
    const user = users[__VU % users.length];  // Assign user to VU
    
    // Login
    group('login', function () {
        const res = http.post(`${__ENV.BASE_URL}/auth/login`, {
            email: user.email,
            password: user.password,
        });
        check(res, { 'login 2xx': (r) => r.status < 300 });
    });
    
    sleep(2);
    
    // Add items to cart
    for (let i = 0; i < randomBetween(1, 3); i++) {
        const product = products[Math.floor(Math.random() * products.length)];
        http.post(`${__ENV.BASE_URL}/cart/add`, {
            product_id: product.id,
            quantity: 1,
        });
        sleep(1);
    }
    
    // Checkout
    group('checkout', function () {
        const res = http.post(`${__ENV.BASE_URL}/checkout`, {
            payment_method: 'test_card',
            shipping_address: user.address,
        });
        check(res, {
            'checkout 2xx': (r) => r.status < 300,
            'checkout completed': (r) => r.json('status') === 'completed',
        });
    });
}
 
function randomBetween(min, max) {
    return Math.random() * (max - min) + min;
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77

# Terraform: Dedicated Performance Testing Environment
 
# Performance testing cluster - matches production ratios
resource "aws_ecs_cluster" "perf_test" {
  name = "perf-test-cluster"
  
  setting {
    name  = "containerInsights"
    value = "enabled"
  }
}
 
# Application service - scaled to match production ratio
resource "aws_ecs_service" "app_perf" {
  name            = "app-perf-test"
  cluster         = aws_ecs_cluster.perf_test.id
  task_definition = aws_ecs_task_definition.app.arn
  
  # Production runs 10 tasks; perf test runs 2 (1:5 ratio)
  # Load generation should also be 1:5 of expected production
  desired_count   = 2
  
  # Match production CPU/memory configuration
  # This is critical for meaningful results
}
 
# Database - production-like sizing
resource "aws_db_instance" "perf_db" {
  identifier     = "perf-test-db"
  
  # Use same instance class as production
  instance_class = "db.r6g.large"  # Match production
  
  # Use production-like data
  # Restore from sanitized production snapshot
  snapshot_identifier = data.aws_db_snapshot.prod_sanitized.id
  
  # Performance monitoring enabled
  performance_insights_enabled = true
}
 
# Load generator - dedicated instances
resource "aws_instance" "load_generator" {
  count         = 3  # Distributed load generation
  
  ami           = data.aws_ami.k6.id
  instance_type = "c5.2xlarge"  # CPU-optimized for load gen
  
  # Ensure network capacity for load generation
  associate_public_ip_address = true
  
  tags = {
    Name = "k6-load-generator-${count.index}"
    Role = "performance-testing"
  }
}
 
# Metrics collection
resource "aws_prometheus_workspace" "perf_metrics" {
  alias = "perf-test-metrics"
}
 
# Grafana for visualization
resource "aws_grafana_workspace" "perf_dashboard" {
  name                     = "perf-test-dashboards"
  account_access_type      = "CURRENT_ACCOUNT"
  authentication_providers = ["SAML"]
  permission_type          = "SERVICE_MANAGED"
  
  data_sources = ["PROMETHEUS", "CLOUDWATCH"]
}
 
# Automated cleanup - don't leave resources running
resource "aws_cloudwatch_event_rule" "cleanup" {
  name                = "perf-env-cleanup"
  schedule_expression = "cron(0 6 * * ? *)"  # Daily at 6 AM
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

# Performance Test Report: Release v2.5.0
 
## Executive Summary
✅ **PASSED** - All critical thresholds met. Minor regression in product search (investigate recommended).
 
## Key Metrics vs. Previous Release
 
| Metric | v2.4.0 | v2.5.0 | Change | Status |
|--------|--------|--------|--------|--------|
| Homepage P95 | 145ms | 148ms | +2.1% | ✅ |
| Search P95 | 280ms | 340ms | +21.4% | ⚠️ |
| Checkout P95 | 890ms | 875ms | -1.7% | ✅ |
| Error Rate | 0.02% | 0.01% | -50% | ✅ |
| Max Throughput | 1,450 RPS | 1,520 RPS | +4.8% | ✅ |
 
## Thresholds Status
 
| Threshold | Requirement | Actual | Status |
|-----------|-------------|--------|--------|
| Homepage P95 | < 200ms | 148ms | ✅ Pass |
| Search P95 | < 400ms | 340ms | ✅ Pass |
| Checkout P95 | < 1000ms | 875ms | ✅ Pass |
| Error Rate | < 0.1% | 0.01% | ✅ Pass |
| Checkout Errors | < 0.01% | 0.00% | ✅ Pass |
 
## Identified Issues
 
### ⚠️ Product Search Regression (+21.4%)
 
**Severity:** Medium (within threshold but notable regression)
 
**Observation:** Search latency increased from 280ms to 340ms P95.
 
**Potential Causes:**
- New full-text search feature added in v2.5.0
- Index structure changed for product catalog
- Additional fields returned in search results
 
**Recommendation:** 
- Review search query execution plan
- Consider Elasticsearch query optimization
- Evaluate if additional fields are necessary
 
### ✅ Checkout Improvement (-1.7%)
 
Payment processing optimization appears successful. No action required.
 
## Test Configuration
 
- **Environment:** Performance Testing (perf-cluster-01)
- **Duration:** 30 minutes sustained load
- **Load Profile:** Peak production traffic pattern
- **Virtual Users:** 200 concurrent (matching weekday peak)
- **Test Data:** sanitized production snapshot (2024-01-10)
 
## Appendix
 
- [Full k6 Report](./k6-report.html)
- [Grafana Dashboard](https://grafana.internal/d/perf-test)
- [Trace Analysis](./traces/summary.json)