Rate Limiting - Learning Module

Loading content...

0/273

Client Identification

The Identity Problem

Rate limiting is only as good as your ability to identify who is making requests. A sophisticated rate limiting algorithm is useless if attackers can simply assume new identities at will. Conversely, a crude identifier that groups unrelated users together will penalize innocent parties for each other's behavior.

The fundamental question: When a request arrives, how do you determine which 'bucket' it belongs to for rate limiting purposes?

This deceptively simple question has profound implications. The answer changes based on whether the user is authenticated, whether they're behind a corporate proxy, whether they're using a VPN, whether they're a bot with rotating signatures, or whether they're a mobile user switching between WiFi and cellular networks.

What You Will Learn

By the end of this page, you will understand the full spectrum of client identification strategies—from simple IP-based identification to sophisticated multi-factor fingerprinting. You'll learn how to handle edge cases like NAT, proxies, and IP rotation, and how to combine identifiers for robust client distinction.

The Identification Hierarchy

Different identifiers provide different levels of precision and trust. Understanding this hierarchy helps you choose the right identifier for each rate limiting scenario.

Client Identification Methods Compared
Identifier	Precision	Spoofability	Availability	Best For
User Account (post-login)	Very High	Low (requires auth)	Authenticated only	Per-user limits, premium features
API Key	High	Medium (if leaked)	API integrations only	Developer/integration limits
Session/Token	High	Medium (if stolen)	After session creation	Anonymous user tracking
Device Fingerprint	Medium-High	Medium-High	Browser/app only	Bot detection supplement
IP Address	Low-Medium	High (rotation easy)	Always	Basic abuse prevention
User-Agent	Very Low	Trivial	Usually	Bot classification only

The golden rule: Use the most precise identifier available for each request. Authenticated requests should be rate limited by user account. API requests should use API keys. Anonymous requests fall back to IP + device fingerprinting.

Composite identification:

In practice, you often combine identifiers:

IP + User-Agent + Session — For anonymous web users
API Key + IP — Catch leaked keys being abused from unexpected locations
User Account + Device ID — Identify session hijacking (legitimate user's device doesn't match)
IP + Endpoint + Time Pattern — Behavioral fingerprinting for sophisticated detection

Defense in Depth for Identification

No single identifier is perfect. Layer multiple identifiers: IP can be rotated, but rotating IP + device fingerprint + behavior pattern simultaneously is much harder. Each layer an attacker must spoof increases their cost and complexity.

IP Address Identification

IP address is the most fundamental client identifier—available for every request, requiring no authentication, and identifying the network location of the client. However, it comes with significant limitations that every engineer must understand.

ip-extraction.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
/**
 * Extracting Client IP Address
 * 
 * The 'real' client IP isn't always in the socket address.
 * Requests often pass through proxies, load balancers, and CDNs.
 */
import { Request } from 'express';
 
interface IPExtractionConfig {
  // Headers to check in order of preference
  headers: string[];
  // Trusted proxy CIDRs (only trust headers from these)
  trustedProxies: string[];
  // Maximum number of hops to trace back
  maxHops: number;
}
 
const DEFAULT_CONFIG: IPExtractionConfig = {
  headers: [
    'CF-Connecting-IP',        // Cloudflare
    'True-Client-IP',          // Akamai, Cloudflare Enterprise
    'X-Real-IP',               // Nginx
    'X-Forwarded-For',         // Standard (comma-separated list)
    'X-Client-IP',             // Apache
    'Forwarded',               // RFC 7239 standard
  ],
  trustedProxies: [
    '10.0.0.0/8',              // Private networks (internal LBs)
    '172.16.0.0/12',           // Private networks
    '192.168.0.0/16',          // Private networks
    '127.0.0.0/8',             // Localhost
    // Add your CDN/proxy IP ranges here
  ],
  maxHops: 5,
};
 
function extractClientIP(req: Request, config = DEFAULT_CONFIG): string {
  // Try each header in order
  for (const header of config.headers) {
    const value = req.headers[header.toLowerCase()];
    if (!value) continue;
 
    const headerValue = Array.isArray(value) ? value[0] : value;
 
    // X-Forwarded-For is comma-separated: client, proxy1, proxy2, ...
    if (header === 'X-Forwarded-For') {
      const ips = headerValue.split(',').map(ip => ip.trim());
      
      // Find rightmost non-trusted proxy
      // Walk from right (closest to us) to left (closest to client)
      for (let i = ips.length - 1; i >= 0; i--) {
        const ip = ips[i];
        if (!isTrustedProxy(ip, config.trustedProxies)) {
          // First non-trusted IP is likely the real client
          // But verify it's not spoofed by checking previous hop
          if (i > 0 || isValidPublicIP(ip)) {
            return ip;
          }
        }
      }
      // All IPs are trusted? Use leftmost (original client)
      return ips[0];
    }
 
    // Single-value headers
    if (isValidIP(headerValue)) {
      return headerValue;
    }
  }
 
  // Fallback to socket address
  return req.socket.remoteAddress || req.ip || 'unknown';
}
 
function isTrustedProxy(ip: string, trustedRanges: string[]): boolean {
  // Implementation would check if IP is in any trusted CIDR
  // Using a library like 'ip-range-check' or 'ipaddr.js'
  return trustedRanges.some(range => isIPInRange(ip, range));
}
 
function isValidIP(ip: string): boolean {
  // Basic IPv4/IPv6 validation
  const ipv4Regex = /^(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$/;
  const ipv6Regex = /^([0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}$/;
  return ipv4Regex.test(ip) || ipv6Regex.test(ip);
}
 
function isValidPublicIP(ip: string): boolean {
  // Reject private, loopback, and special-use IPs
  const privateRanges = [
    '10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16',
    '127.0.0.0/8', '169.254.0.0/16', '0.0.0.0/8',
  ];
  return !privateRanges.some(range => isIPInRange(ip, range));
}
 
// Example Express middleware
function ipRateLimitMiddleware(req: Request, res: Response, next: NextFunction) {
  const clientIP = extractClientIP(req);
  
  // Normalize IPv6 to handle variations
  const normalizedIP = normalizeIP(clientIP);
  
  // Use as rate limit key
  req.rateLimitKey = `ip:${normalizedIP}`;
  
  next();
}

The X-Forwarded-For Trust Problem:

X-Forwarded-For is trivially spoofable. A malicious client can send:

X-Forwarded-For: 1.2.3.4, 5.6.7.8, 9.10.11.12

If you naively take the leftmost IP (1.2.3.4), the client can claim any identity. Your load balancer/proxy then appends the real IP to the end.

Correct parsing: Walk the list from right to left. Count hops from trusted proxies. The first untrusted IP is your best guess at the real client.

X-Forwarded-For: [spoofed], [spoofed], [real_client], [your_proxy], [your_lb]
                                           ↑
                              First non-trusted = real client

Critical: Configure Trusted Proxies

Many applications trust X-Forwarded-For by default, creating a trivial bypass. ALWAYS configure the exact IPs/CIDRs of your proxies and only trust headers from those sources. Most frameworks support this: Express (trust proxy), Django (TRUSTED_PROXIES), Rails (trusted_proxies).

NAT and Shared IP Challenges

One of the most significant challenges with IP-based rate limiting is Network Address Translation (NAT)—where multiple devices share a single public IP address. This is not an edge case; it's the norm for much of the internet.

NAT Scenarios and Scale
Scenario	Users Per IP	Risk of Collateral Damage
Home router (typical)	1-10 devices	Low - family usually acts as unit
Small office	10-50 users	Medium - rate limits may affect team
Large enterprise proxy	1,000-50,000 users	High - aggressive limits block many innocents
University campus	10,000-100,000 students	Very High - one bad actor blocks everyone
Mobile carrier (CGNAT)	100,000+ users	Extreme - entire carrier shares IP blocks
Public WiFi (café, airport)	Variable, many ephemeral	High - limits affect all customers

Carrier-Grade NAT (CGNAT):

ISPs running low on IPv4 addresses deploy CGNAT, where thousands of customers share a small pool of public IP addresses. An IP address like 100.64.x.x might represent 50,000+ different household connections.

Practical implications:

Loose IP limits — Set IP limits high enough that normal users behind NAT aren't affected (e.g., 1000/minute vs 100/minute)
Combined identifiers — Use IP + other signals to distinguish users behind the same NAT (session, device fingerprint)
Graduated enforcement — Light limits on IP alone, stricter limits with more signals
Challenge-response — When IP limit triggers, present CAPTCHA rather than hard block
Monitor false positives — Track when legitimate users hit limits and adjust accordingly

nat-aware-limiting.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
/**
 * NAT-Aware Rate Limiting Strategy
 * 
 * Use tiered limits: loose per-IP, strict per-session/user
 */
interface RateLimitResult {
  allowed: boolean;
  triggered: 'ip' | 'session' | 'user' | 'none';
  action: 'allow' | 'captcha' | 'block' | 'delay';
  retryAfter?: number;
}
 
class NATAwareRateLimiter {
  private ipLimiter: RateLimiter;      // Loose: 1000/min
  private sessionLimiter: RateLimiter; // Medium: 200/min
  private userLimiter: RateLimiter;    // Strict: 100/min
 
  constructor() {
    // IP limits are loose due to NAT
    this.ipLimiter = new SlidingWindowCounter(1000, 60);
    
    // Session limits are tighter
    this.sessionLimiter = new SlidingWindowCounter(200, 60);
    
    // Authenticated user limits are strictest
    this.userLimiter = new SlidingWindowCounter(100, 60);
  }
 
  async checkRequest(
    ip: string,
    sessionId: string | null,
    userId: string | null
  ): Promise<RateLimitResult> {
    
    // Check from most specific to least specific
    
    // 1. User limit (if authenticated)
    if (userId) {
      const userResult = await this.userLimiter.check(`user:${userId}`);
      if (!userResult.allowed) {
        return {
          allowed: false,
          triggered: 'user',
          action: 'block',
          retryAfter: userResult.retryAfter,
        };
      }
    }
    
    // 2. Session limit (if session exists)
    if (sessionId) {
      const sessionResult = await this.sessionLimiter.check(`session:${sessionId}`);
      if (!sessionResult.allowed) {
        // Session limit hit - could be shared browser, so captcha
        return {
          allowed: false,
          triggered: 'session',
          action: 'captcha',
        };
      }
    }
    
    // 3. IP limit (always check, but loose)
    const ipResult = await this.ipLimiter.check(`ip:${ip}`);
    if (!ipResult.allowed) {
      // IP limit hit - might be NAT, so captcha not block
      // Unless this IP has history of abuse
      const ipAbusiveHistory = await this.checkIPReputation(ip);
      
      if (ipAbusiveHistory) {
        return {
          allowed: false,
          triggered: 'ip',
          action: 'block',
          retryAfter: ipResult.retryAfter,
        };
      }
      
      return {
        allowed: false,
        triggered: 'ip',
        action: 'captcha',
      };
    }
    
    return {
      allowed: true,
      triggered: 'none',
      action: 'allow',
    };
  }
 
  private async checkIPReputation(ip: string): Promise<boolean> {
    // Check abuse history, IP reputation services, etc.
    // Return true if IP has been previously identified as abusive
    return false;
  }
}

IPv6 Changes the Equation

IPv6 allocates /64 or larger blocks to end users, meaning each user effectively has billions of unique IPs. Rate limit on /64 or /56 prefixes, not individual IPv6 addresses. Otherwise, attackers can easily generate new addresses within their allocation.

API Key Identification

API keys provide a much better identifier than IP addresses for programmatic access. They're issued to specific developers/applications, can be revoked individually, and allow for differentiated service tiers.

api-key-limiting.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
/**
 * API Key-Based Rate Limiting
 * 
 * Keys can have different limits based on tier, and can be
 * tracked for abuse patterns independent of IP.
 */
interface APIKey {
  id: string;
  key: string;
  tier: 'free' | 'basic' | 'professional' | 'enterprise';
  ownerId: string;
  createdAt: Date;
  lastUsed: Date;
  allowedIPs?: string[];  // Optional IP allowlist for security
  rateLimit: {
    requestsPerMinute: number;
    requestsPerDay: number;
    burstSize: number;
  };
}
 
const TIER_LIMITS: Record<string, APIKey['rateLimit']> = {
  free: { requestsPerMinute: 60, requestsPerDay: 1000, burstSize: 10 },
  basic: { requestsPerMinute: 600, requestsPerDay: 50000, burstSize: 50 },
  professional: { requestsPerMinute: 3000, requestsPerDay: 500000, burstSize: 200 },
  enterprise: { requestsPerMinute: 10000, requestsPerDay: 10000000, burstSize: 1000 },
};
 
class APIKeyRateLimiter {
  private keyCache: Map<string, APIKey> = new Map();
  private minuteLimiter: RateLimiter;
  private dailyLimiter: RateLimiter;
  private burstLimiter: TokenBucketRateLimiter;
 
  async validateAndLimit(
    apiKey: string,
    clientIP: string
  ): Promise<{ allowed: boolean; key?: APIKey; error?: string }> {
    
    // 1. Validate key exists and is active
    const key = await this.getKey(apiKey);
    if (!key) {
      return { allowed: false, error: 'Invalid API key' };
    }
 
    // 2. Check IP allowlist if configured
    if (key.allowedIPs && key.allowedIPs.length > 0) {
      if (!key.allowedIPs.includes(clientIP)) {
        // Log potential key compromise
        await this.logSuspiciousActivity(key.id, clientIP, 'ip_not_allowed');
        return { 
          allowed: false, 
          error: 'Request from unauthorized IP address',
          key 
        };
      }
    }
 
    // 3. Check all rate limits
    const limits = key.rateLimit;
    
    // Burst protection (token bucket)
    const burstAllowed = await this.burstLimiter.check(
      `burst:${key.id}`,
      limits.burstSize,
      limits.requestsPerMinute / 60  // refill rate
    );
    if (!burstAllowed.allowed) {
      return { 
        allowed: false, 
        error: 'Burst limit exceeded. Please slow down.',
        key 
      };
    }
 
    // Minute limit (sliding window)
    const minuteAllowed = await this.minuteLimiter.check(
      `minute:${key.id}`,
      limits.requestsPerMinute,
      60
    );
    if (!minuteAllowed.allowed) {
      return { 
        allowed: false, 
        error: 'Rate limit exceeded. Retry after ' + minuteAllowed.retryAfter + 's',
        key 
      };
    }
 
    // Daily limit (fixed window aligned to UTC midnight)
    const dailyAllowed = await this.dailyLimiter.check(
      `daily:${key.id}`,
      limits.requestsPerDay,
      86400
    );
    if (!dailyAllowed.allowed) {
      return { 
        allowed: false, 
        error: 'Daily quota exceeded. Resets at UTC midnight.',
        key 
      };
    }
 
    // 4. Update last used timestamp
    this.updateLastUsed(key.id);
 
    return { allowed: true, key };
  }
 
  private async getKey(apiKey: string): Promise<APIKey | null> {
    // Check cache first
    let key = this.keyCache.get(apiKey);
    if (key) return key;
 
    // Lookup in database
    key = await this.lookupKeyFromDatabase(apiKey);
    if (key) {
      // Cache for performance (short TTL due to revocation possibility)
      this.keyCache.set(apiKey, key);
      setTimeout(() => this.keyCache.delete(apiKey), 60000);
    }
 
    return key;
  }
}

API Key Best Practices

•Prefix for visibility — Use prefixes like sk_live_, pk_test_ so keys are identifiable
•Sufficient entropy — 32+ bytes of cryptographic randomness
•Hash for storage — Store hash(key), never plaintext
•Rotation support — Allow multiple active keys for zero-downtime rotation
•Scope limitation — Keys should have minimal required permissions
•Audit logging — Log all key usage for security review

Key Compromise Indicators

•Geographic anomalies — Key used from unexpected countries
•IP diversity — Sudden use from many IPs (distributed abuse)
•Usage pattern shifts — 10x normal volume, different endpoints
•Off-hours activity — Business key used at 3 AM
•Error rate spike — Many 401/403s suggest credential testing
•Concurrent users — Key used from multiple IPs simultaneously

Key + IP for Defense in Depth

A leaked API key can be abused from anywhere. Combine key-based limits with IP-based limits: even with a valid key, unusual IPs should face stricter scrutiny. Consider requiring IP allowlisting for high-privilege keys or implementing step-up verification for sensitive operations from new IPs.

Session and User Account Identification

For web applications, sessions provide a stable identifier that persists across requests, while user accounts (post-authentication) provide the most reliable identification possible.

Session-based identification:

Sessions bridge the gap between anonymous IP-based identification and authenticated user-based identification:

Anonymous session — Assign session ID on first visit (cookie or token)
Pre-login tracking — Track behavior before user authenticates
Session binding — After login, bind session to user account
Cross-session continuity — User limits follow them across devices after login

Session token best practices:

Generate with cryptographic randomness (128+ bits entropy)
Set HttpOnly, Secure, SameSite=Strict flags for cookies
Rotate on authentication state changes
Store server-side (Redis/DB), not just in JWT

user-rate-limiting.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
/**
 * User-Centric Rate Limiting
 * 
 * Once authenticated, rate limiting follows the user
 * regardless of IP, device, or session.
 */
interface UserRateLimits {
  // Standard API usage
  apiRequestsPerMinute: number;
  apiRequestsPerDay: number;
  
  // Specific action limits (stricter)
  passwordChangesPerDay: number;
  emailChangesPerWeek: number;
  paymentAttemptsPerHour: number;
  
  // Resource creation limits
  newProjectsPerDay: number;
  fileUploadsPerHour: number;
  invitationsPerDay: number;
}
 
const DEFAULT_USER_LIMITS: UserRateLimits = {
  apiRequestsPerMinute: 120,
  apiRequestsPerDay: 10000,
  passwordChangesPerDay: 3,
  emailChangesPerWeek: 2,
  paymentAttemptsPerHour: 5,
  newProjectsPerDay: 10,
  fileUploadsPerHour: 100,
  invitationsPerDay: 50,
};
 
class UserRateLimiter {
  /**
   * Check if a specific action is allowed for a user
   */
  async checkAction(
    userId: string,
    action: keyof UserRateLimits,
    userTier: 'free' | 'paid' | 'enterprise' = 'free'
  ): Promise<{ allowed: boolean; remaining: number; resetIn: number }> {
    
    // Get user's limits (may vary by tier)
    const limits = this.getLimitsForTier(userTier);
    const limit = limits[action];
    
    // Determine window based on action name
    const window = this.getWindowForAction(action);
    
    const key = `user:${userId}:action:${action}`;
    
    return this.limiter.check(key, limit, window);
  }
 
  /**
   * Check multiple limits atomically (for complex operations)
   */
  async checkMultipleActions(
    userId: string,
    actions: (keyof UserRateLimits)[],
    userTier: 'free' | 'paid' | 'enterprise' = 'free'
  ): Promise<{
    allowed: boolean;
    blockedBy?: keyof UserRateLimits;
    results: Record<keyof UserRateLimits, { allowed: boolean; remaining: number }>;
  }> {
    const results: any = {};
    let blockedBy: keyof UserRateLimits | undefined;
 
    for (const action of actions) {
      const result = await this.checkAction(userId, action, userTier);
      results[action] = {
        allowed: result.allowed,
        remaining: result.remaining,
      };
      
      if (!result.allowed && !blockedBy) {
        blockedBy = action;
      }
    }
 
    return {
      allowed: !blockedBy,
      blockedBy,
      results,
    };
  }
 
  private getWindowForAction(action: string): number {
    if (action.includes('PerMinute')) return 60;
    if (action.includes('PerHour')) return 3600;
    if (action.includes('PerDay')) return 86400;
    if (action.includes('PerWeek')) return 604800;
    return 60; // default
  }
 
  private getLimitsForTier(tier: string): UserRateLimits {
    const multipliers = { free: 1, paid: 3, enterprise: 10 };
    const mult = multipliers[tier] || 1;
    
    return {
      apiRequestsPerMinute: DEFAULT_USER_LIMITS.apiRequestsPerMinute * mult,
      apiRequestsPerDay: DEFAULT_USER_LIMITS.apiRequestsPerDay * mult,
      // Security-related limits don't scale with tier
      passwordChangesPerDay: DEFAULT_USER_LIMITS.passwordChangesPerDay,
      emailChangesPerWeek: DEFAULT_USER_LIMITS.emailChangesPerWeek,
      paymentAttemptsPerHour: DEFAULT_USER_LIMITS.paymentAttemptsPerHour,
      newProjectsPerDay: DEFAULT_USER_LIMITS.newProjectsPerDay * mult,
      fileUploadsPerHour: DEFAULT_USER_LIMITS.fileUploadsPerHour * mult,
      invitationsPerDay: DEFAULT_USER_LIMITS.invitationsPerDay * mult,
    };
  }
}

Action-Specific Limits

Different actions warrant different limits. Viewing content might allow 1000/minute. Creating resources might allow 10/minute. Changing security settings might allow 3/day. Password reset requests might allow 5/hour. Apply the principle of least privilege to rate limits.

Device Fingerprinting

Device fingerprinting attempts to uniquely identify a device or browser instance based on a combination of attributes: screen resolution, installed fonts, browser plugins, timezone, language preferences, and dozens of other signals.

Use in rate limiting:

Device fingerprints can distinguish multiple users behind the same NAT, detect when attackers rotate IP addresses, and identify bot behavior that mimics human characteristics. However, they're not foolproof and raise privacy considerations.

Device Fingerprinting Signals
Signal Category	Examples	Entropy	Privacy Concern
Screen & Display	Resolution, color depth, pixel ratio	Medium	Low
Browser	User-Agent, plugins, languages	Medium-High	Medium
System	Timezone, platform, CPU cores	Medium	Low
Fonts	Installed font list	High	High
Canvas/WebGL	Rendering fingerprint	Very High	High
Audio	AudioContext fingerprint	High	High
Behavior	Mouse movement, typing patterns	Very High	Very High

fingerprint-integration.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
/**
 * Integrating Device Fingerprinting with Rate Limiting
 * 
 * Use fingerprints as an additional signal, not sole identifier.
 * FingerprintJS or similar libraries provide stable device IDs.
 */
interface DeviceSignals {
  fingerprintId: string;      // Stable device identifier
  confidence: number;         // 0.0 to 1.0
  signals: {
    timezone: string;
    language: string;
    platform: string;
    screenResolution: string;
    colorDepth: number;
    cpuCores: number;
    // ... other signals
  };
  bot: {
    probability: number;
    signals: string[];        // ['headless_browser', 'automation_tool', ...]
  };
}
 
class FingerprintAwareRateLimiter {
  /**
   * Enhanced rate limiting using device fingerprint
   */
  async checkWithFingerprint(
    ip: string,
    sessionId: string | null,
    userId: string | null,
    fingerprint: DeviceSignals | null
  ): Promise<RateLimitResult> {
    
    // High bot probability = stricter limits immediately
    if (fingerprint?.bot.probability > 0.8) {
      // Log for review but might block or challenge
      await this.logBotDetection(ip, fingerprint);
      
      return {
        allowed: false,
        action: 'captcha',
        reason: 'Suspected automated access',
      };
    }
 
    // Use fingerprint to augment/replace IP for anonymous users
    const identifier = this.selectBestIdentifier(ip, sessionId, userId, fingerprint);
    
    // Check limits using selected identifier
    return this.checkLimit(identifier);
  }
 
  private selectBestIdentifier(
    ip: string,
    sessionId: string | null,
    userId: string | null,
    fingerprint: DeviceSignals | null
  ): string {
    // Prefer most specific/trustworthy identifier
    
    if (userId) {
      return `user:${userId}`;
    }
    
    if (sessionId) {
      // Combine session with IP to prevent session hijacking abuse
      return `session:${sessionId}:ip:${this.hashIP(ip)}`;
    }
    
    if (fingerprint && fingerprint.confidence > 0.9) {
      // High-confidence fingerprint can replace IP
      return `device:${fingerprint.fingerprintId}`;
    }
    
    if (fingerprint) {
      // Lower confidence: combine fingerprint with IP
      return `ip:${ip}:device:${fingerprint.fingerprintId}`;
    }
    
    // Fallback to IP only
    return `ip:${ip}`;
  }
 
  /**
   * Detect device switching that may indicate abuse
   */
  async detectDeviceAnomaly(
    userId: string,
    newFingerprint: DeviceSignals
  ): Promise<{ anomaly: boolean; risk: 'low' | 'medium' | 'high' }> {
    // Get user's known devices
    const knownDevices = await this.getUserDevices(userId);
    
    // Check if this fingerprint is known
    const isKnown = knownDevices.some(
      d => d.fingerprintId === newFingerprint.fingerprintId
    );
    
    if (isKnown) {
      return { anomaly: false, risk: 'low' };
    }
    
    // New device - check other signals
    const knownTimezones = new Set(knownDevices.map(d => d.signals.timezone));
    const knownPlatforms = new Set(knownDevices.map(d => d.signals.platform));
    
    const tzChange = !knownTimezones.has(newFingerprint.signals.timezone);
    const platformChange = !knownPlatforms.has(newFingerprint.signals.platform);
    
    // Multiple signal changes = higher risk
    const risk = (tzChange && platformChange) ? 'high' : 
                 (tzChange || platformChange) ? 'medium' : 'low';
    
    return { anomaly: true, risk };
  }
}

Privacy and Legal Considerations

Device fingerprinting can create unique identifiers that track users without consent, potentially violating GDPR, CCPA, and other privacy regulations. If using fingerprinting: disclose it in your privacy policy, consider consent requirements, limit data retention, and use it for security (fraud prevention) rather than tracking. Some jurisdictions treat fingerprints like cookies and require explicit consent.

Proxy and VPN Detection

Attackers frequently use proxies, VPNs, and Tor to mask their identities and bypass IP-based rate limiting. Detecting and handling these sources is crucial for effective rate limiting at scale.

Types of IP masking:

Datacenter proxies — IPs from cloud providers (AWS, GCP, DigitalOcean). Cheap, fast, but easily detected.
Residential proxies — IPs from real home connections. Harder to distinguish from legitimate users.
Mobile proxies — IPs from mobile carriers. Very difficult to block without impacting real mobile users.
VPN services — Commercial VPNs with known IP ranges. Balance between privacy needs and abuse.
Tor exit nodes — Published list of IPs. Known for both privacy advocates and abuse.

Detection approaches:

IP reputation databases — Services like MaxMind, IPQualityScore, IPHub maintain databases of proxy/VPN IPs
ASN analysis — Datacenter ASNs are distinct from residential ISP ASNs
Latency analysis — VPN traffic often has higher/inconsistent latency
Behavioral anomalies — Traffic patterns differ from normal users
IPv6 leaks — Poorly configured VPNs may leak real IPv6 addresses

proxy-handling.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
/**
 * Proxy-Aware Rate Limiting Strategy
 * 
 * Don't block proxies outright—many have legitimate uses.
 * Apply stricter limits and additional verification.
 */
interface IPIntelligence {
  ip: string;
  classification: 'residential' | 'datacenter' | 'mobile' | 'proxy' | 'vpn' | 'tor';
  asn: number;
  asnOrg: string;
  country: string;
  riskScore: number;  // 0.0 to 1.0
  isKnownBad: boolean;
}
 
class ProxyAwareRateLimiter {
  private ipIntelService: IPIntelligenceService;
 
  /**
   * Apply risk-adjusted rate limits
   */
  async getRateLimits(
    ip: string,
    isAuthenticated: boolean
  ): Promise<{
    requestsPerMinute: number;
    requiresCaptcha: boolean;
    requiresMFA: boolean;
    blocked: boolean;
  }> {
    const intel = await this.ipIntelService.lookup(ip);
    
    // Definitely blocked
    if (intel.isKnownBad && intel.riskScore > 0.95) {
      return {
        requestsPerMinute: 0,
        requiresCaptcha: true,
        requiresMFA: false,
        blocked: true,
      };
    }
 
    // Base limits
    let limits = {
      requestsPerMinute: 100,
      requiresCaptcha: false,
      requiresMFA: false,
      blocked: false,
    };
 
    // Adjust based on classification
    switch (intel.classification) {
      case 'residential':
        // Normal limits
        break;
        
      case 'mobile':
        // Slightly looser - CGNAT common
        limits.requestsPerMinute = 120;
        break;
        
      case 'datacenter':
        // Stricter - usually not real users
        limits.requestsPerMinute = 30;
        if (!isAuthenticated) {
          limits.requiresCaptcha = true;
        }
        break;
        
      case 'proxy':
      case 'vpn':
        // Even stricter
        limits.requestsPerMinute = 20;
        limits.requiresCaptcha = !isAuthenticated;
        // Require additional verification for sensitive actions
        limits.requiresMFA = true;
        break;
        
      case 'tor':
        // Most restrictive
        limits.requestsPerMinute = 10;
        limits.requiresCaptcha = true;
        limits.requiresMFA = true;
        break;
    }
 
    // Adjust based on risk score
    if (intel.riskScore > 0.7) {
      limits.requestsPerMinute = Math.floor(limits.requestsPerMinute * 0.5);
      limits.requiresCaptcha = true;
    }
 
    return limits;
  }
 
  /**
   * Detect behavior inconsistent with claimed location
   */
  detectLocationAnomaly(
    intel: IPIntelligence,
    browserTimezone: string,
    browserLanguage: string
  ): { anomaly: boolean; signals: string[] } {
    const signals: string[] = [];
    
    // Country from IP vs timezone
    const expectedTimezones = this.getTimezonesForCountry(intel.country);
    if (!expectedTimezones.includes(browserTimezone)) {
      signals.push(`Timezone mismatch: ${browserTimezone} not in ${intel.country}`);
    }
    
    // Country from IP vs language
    const expectedLanguages = this.getLanguagesForCountry(intel.country);
    const browserLang = browserLanguage.split('-')[0];
    if (!expectedLanguages.includes(browserLang)) {
      signals.push(`Language mismatch: ${browserLang} unusual for ${intel.country}`);
    }
    
    return {
      anomaly: signals.length > 0,
      signals,
    };
  }
}

Legitimate Proxy Use Cases

Not all proxy/VPN usage is malicious. Privacy-conscious users, journalists in authoritarian countries, corporate employees behind VPNs, and security researchers all have legitimate reasons. Consider: lower limits rather than blocks, CAPTCHA challenges rather than denial, and reputation building over time for consistent users.

Summary: Client Identification

Effective client identification is the foundation of effective rate limiting. Without accurate identification, even the best algorithms can be circumvented. Let's consolidate the key insights:

Key Takeaways

•Use the most precise identifier available — User accounts are better than API keys, which are better than sessions, which are better than IPs
•Combine identifiers for defense in depth — IP + session + device fingerprint is harder to spoof than any single identifier
•Handle IP headers carefully — X-Forwarded-For is trivially spoofable; only trust headers from configured trusted proxies
•Account for NAT and shared IPs — Set IP limits high enough for NAT scenarios; use CAPTCHA challenges rather than hard blocks
•Rate limit IPv6 by prefix — Users have /64 or larger allocations; limiting individual addresses is ineffective
•API keys enable tiered limits — Different tiers get different limits; keys can be revoked individually
•Device fingerprinting supplements other identifiers — Helps distinguish users behind NAT but raises privacy concerns
•Detect and handle proxies/VPNs specially — Don't block outright, but apply stricter limits and verification requirements

What's next:

With algorithms, distributed architectures, and client identification covered, we now address the final piece: Response Handling. How do you communicate rate limits to clients? What HTTP status codes and headers should you use? How do you design graceful degradation? How should clients handle being rate limited?

Page Complete

You now understand the full spectrum of client identification strategies for rate limiting. From simple IP-based identification to sophisticated multi-factor fingerprinting, these techniques enable you to accurately identify and distinguish clients—the foundation for all rate limiting decisions.