System Design (HLD)Google Drive / Dropbox

Designing Cloud File Storage Systems

LevelAdvanced

Duration90 mins

TopicGoogle Drive / Dropbox

4 / 6

Chunked Uploads

The Large File Problem

Uploading large files over the internet presents unique challenges. A 4K video file might be 50GB. A database backup might be 200GB. Uploading such files in a single HTTP request is impractical—any network interruption means starting from scratch.

The Core Problems:

Unreliable Networks — Connections drop. WiFi switches. Laptops close. Any interruption loses all progress.
Timeout Limits — Load balancers, proxies, and servers have timeout limits (often 30-60 seconds). A slow upload might not complete in time.
Memory Constraints — Server can't buffer an entire 50GB file in memory before writing to storage.
Progress Feedback — Users need to see upload progress. Single-request uploads provide poor progress granularity.
Retry Efficiency — If upload fails at 99%, we shouldn't re-upload the entire file.

The Solution: Chunked Uploads

Break large files into smaller chunks (typically 4-16MB each) and upload each chunk independently. This enables resumption, parallel uploads, and progress tracking.

Learning Objectives

By the end of this page, you'll understand: (1) Chunked upload protocols and their design rationale, (2) Resumable upload implementation, (3) Parallel chunk upload for speed, (4) Server-side chunk assembly and verification, and (5) Content-defined chunking for deduplication.

Chunking Strategies

There are two fundamental approaches to dividing files into chunks: fixed-size chunking and content-defined chunking (CDC). Each has distinct trade-offs.

Fixed-Size Chunking

•Divide file at fixed byte boundaries (e.g., every 8MB)
•Simple to implement: chunk N = bytes[N*size : (N+1)*size]
•Predictable chunk count: ceil(fileSize / chunkSize)
•Problem: Insert 1 byte at start, ALL chunks shift
•Poor deduplication for modified files
•Good for: initial upload of new files

Content-Defined Chunking

•Boundaries determined by content pattern
•Uses rolling hash to find 'cut points'
•Insert 1 byte → only nearby chunks affected
•Excellent for deduplication of modified files
•Variable chunk sizes (need min/max bounds)
•Good for: sync, backup, modified file upload

Content-Defined Chunking (CDC) Algorithm:

Rolling Hash Chunking:

1. Initialize rolling hash window (e.g., 48 bytes)
2. Slide window byte-by-byte through file
3. At each position, check if hash matches pattern:
   if (hash & MASK) == TARGET:  // e.g., last 13 bits are zero
       Create chunk boundary here
4. Enforce minimum chunk size (don't cut too often)
5. Enforce maximum chunk size (cut even without pattern match)

Example with average 8KB chunks:
  MASK = 0x1FFF (13 bits)
  TARGET = 0
  Expected avg chunk: 2^13 = 8192 bytes
  Min chunk: 2KB (avoid tiny chunks)
  Max chunk: 64KB (ensure progress on non-matching content)

Chunk Size Trade-offs
Chunk Size	Chunks per 1GB	Metadata Overhead	Dedup Efficiency	Best Use Case
256 KB	4,096	High	Excellent	Text documents, code
1 MB	1,024	Moderate	Very Good	Mixed content
4 MB	256	Low	Good	General purpose
8 MB	128	Very Low	Moderate	Large media files
16 MB	64	Minimal	Lower	Streaming video

Hybrid Approach

Production systems often use a hybrid: fixed-size chunking for the initial upload (simpler, parallel-friendly), and content-defined chunking for subsequent syncs (better deduplication). The storage layer uses content-addressed storage regardless of how chunks were created.

Resumable Upload Protocol

A resumable upload protocol enables clients to continue interrupted uploads without retransmitting already-uploaded chunks. Here's how a well-designed protocol works:

Converting Mermaid diagram...

Protocol Design Principles

•Idempotent Operations — Re-uploading the same chunk with same content has no effect. Safe to retry on any failure.
•Upload Session State — Server tracks which chunks received. Client can query: 'Which chunks do you have?' and resume from there.
•Session Expiry — Upload sessions expire after inactivity (typically 24-72 hours). Prevents orphaned session accumulation.
•Chunk Verification — Each chunk includes MD5/SHA256 hash. Server verifies before acknowledging. Corrupted chunks rejected.
•Final Assembly — After all chunks received, server assembles file and verifies overall hash matches expected.
•Atomic Completion — File doesn't appear to other users until fully uploaded and verified. No partial files visible.

resumable_upload.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
// Resumable upload client implementation
interface UploadSession {
    uploadId: string;
    uploadUrl: string;
    expiry: Date;
    uploadedChunks: Set<number>;
}
 
class ResumableUploader {
    private chunkSize = 4 * 1024 * 1024; // 4MB
    
    async uploadFile(file: File, onProgress: (pct: number) => void): Promise<string> {
        // Phase 1: Initialize or resume session
        const session = await this.getOrCreateSession(file);
        
        // Phase 2: Upload chunks
        const totalChunks = Math.ceil(file.size / this.chunkSize);
        
        for (let i = 0; i < totalChunks; i++) {
            if (session.uploadedChunks.has(i)) continue; // Already uploaded
            
            const start = i * this.chunkSize;
            const end = Math.min(start + this.chunkSize, file.size);
            const chunk = file.slice(start, end);
            
            await this.uploadChunkWithRetry(session, i, chunk);
            onProgress((i + 1) / totalChunks * 100);
            
            this.saveSessionToLocalStorage(session); // Persist progress
        }
        
        // Phase 3: Complete upload
        const fileId = await this.completeUpload(session, file);
        this.clearSession(session.uploadId);
        
        return fileId;
    }
    
    private async uploadChunkWithRetry(
        session: UploadSession, 
        index: number, 
        chunk: Blob,
        maxRetries = 3
    ): Promise<void> {
        const hash = await this.computeHash(chunk);
        
        for (let attempt = 0; attempt < maxRetries; attempt++) {
            try {
                const response = await fetch(
                    `${session.uploadUrl}/chunk/${index}`,
                    {
                        method: 'PUT',
                        headers: {
                            'Content-Type': 'application/octet-stream',
                            'Content-MD5': hash,
                            'Content-Range': `bytes ${index * this.chunkSize}-${index * this.chunkSize + chunk.size - 1}`,
                        },
                        body: chunk,
                    }
                );
                
                if (response.ok) {
                    session.uploadedChunks.add(index);
                    return;
                }
                
                if (response.status === 409) {
                    // Chunk already exists (idempotent retry)
                    session.uploadedChunks.add(index);
                    return;
                }
                
                throw new Error(`Upload failed: ${response.status}`);
            } catch (error) {
                if (attempt === maxRetries - 1) throw error;
                await this.exponentialBackoff(attempt);
            }
        }
    }
    
    private async getOrCreateSession(file: File): Promise<UploadSession> {
        // Check for existing session in localStorage
        const existing = this.loadSessionFromLocalStorage(file.name, file.size);
        if (existing) {
            // Verify session still valid on server
            const status = await this.getSessionStatus(existing.uploadId);
            if (status.valid) {
                existing.uploadedChunks = new Set(status.uploadedChunks);
                return existing;
            }
        }
        
        // Create new session
        const response = await fetch('/api/uploads/init', {
            method: 'POST',
            headers: { 'Content-Type': 'application/json' },
            body: JSON.stringify({
                filename: file.name,
                size: file.size,
                chunkSize: this.chunkSize,
                contentHash: await this.computeFileHash(file),
            }),
        });
        
        return response.json();
    }
}

The tus Protocol

The 'tus' protocol (tus.io) is an open standard for resumable uploads, implemented by many cloud providers. It defines HTTP-based resumable upload semantics that are widely supported. Consider using tus rather than inventing a custom protocol—it handles edge cases you might miss.

Parallel Chunk Upload

Sequential chunk upload is simple but doesn't maximize bandwidth utilization. Parallel uploading multiple chunks simultaneously dramatically improves upload speed, especially on high-latency connections.

Why Parallel Uploads Are Faster:

Sequential Upload (1 chunk at a time):
  Total Time = N × (latency + chunk_transfer_time)
  
  For 10 chunks, 100ms latency, 2s transfer each:
  Total = 10 × (0.1 + 2) = 21 seconds

Parallel Upload (4 concurrent):
  Total Time ≈ ceil(N/4) × (latency + chunk_transfer_time)
  
  For 10 chunks, 100ms latency, 2s transfer each:
  Total ≈ 3 × (0.1 + 2) = 6.3 seconds
  
  3.3x faster!

The key insight: while one chunk is transferring, other connections can be established and chunks can be queued. TCP slow-start is amortized across multiple connections.

parallel_upload.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
// Parallel chunk upload with concurrency control
class ParallelUploader {
    private concurrency = 4;  // Simultaneous uploads
    private chunkSize = 4 * 1024 * 1024;
    
    async uploadFile(file: File, session: UploadSession): Promise<void> {
        const totalChunks = Math.ceil(file.size / this.chunkSize);
        const pendingChunks: number[] = [];
        
        // Build list of chunks that need uploading
        for (let i = 0; i < totalChunks; i++) {
            if (!session.uploadedChunks.has(i)) {
                pendingChunks.push(i);
            }
        }
        
        // Process chunks with limited concurrency
        await this.processWithConcurrency(
            pendingChunks,
            async (chunkIndex) => {
                const start = chunkIndex * this.chunkSize;
                const end = Math.min(start + this.chunkSize, file.size);
                const chunk = file.slice(start, end);
                
                await this.uploadChunk(session, chunkIndex, chunk);
                session.uploadedChunks.add(chunkIndex);
            },
            this.concurrency
        );
    }
    
    private async processWithConcurrency<T>(
        items: T[],
        processor: (item: T) => Promise<void>,
        limit: number
    ): Promise<void> {
        const queue = [...items];
        const inFlight: Promise<void>[] = [];
        
        const processNext = async (): Promise<void> => {
            while (queue.length > 0) {
                const item = queue.shift()!;
                await processor(item);
            }
        };
        
        // Start 'limit' concurrent workers
        const workers = Array(Math.min(limit, items.length))
            .fill(null)
            .map(() => processNext());
        
        await Promise.all(workers);
    }
}
 
// Advanced: Priority queue for smart chunk ordering
class SmartParallelUploader extends ParallelUploader {
    // Upload chunks near current read position first
    // This enables streaming playback during upload
    getChunkPriority(chunkIndex: number, totalChunks: number): number {
        // Priority: first few chunks (for preview), then sequential
        if (chunkIndex < 3) return 0;  // Highest priority
        return chunkIndex;  // Then sequential
    }
    
    // Adaptive concurrency based on bandwidth
    async measureBandwidth(): Promise<number> {
        const testChunk = new Uint8Array(64 * 1024); // 64KB test
        const start = performance.now();
        await this.uploadTestChunk(testChunk);
        const elapsed = performance.now() - start;
        
        const mbps = (64 / 1024) / (elapsed / 1000);
        
        // Adjust concurrency based on observed bandwidth
        if (mbps > 10) return 6;     // Fast connection: more parallel
        if (mbps > 2) return 4;      // Medium: moderate parallel
        return 2;                     // Slow: fewer parallel
    }
}

Optimal Concurrency by Connection Type
Connection	Bandwidth	Latency	Optimal Concurrency
Slow WiFi	1-5 Mbps	50-200ms	2
Home Broadband	10-50 Mbps	20-50ms	4
Fast Fiber	100-500 Mbps	5-20ms	6-8
Enterprise LAN	1+ Gbps	1-5ms	8-16
Same Datacenter	10+ Gbps	<1ms	16-32

Browser Limitations

Browsers limit concurrent HTTP connections per domain to 6 (HTTP/1.1) or effectively unlimited but prioritized (HTTP/2). Exceeding this creates head-of-line blocking. Solutions: use multiple subdomains for uploads, ensure HTTP/2 support, or use WebSocket-based upload protocols.

Server-Side Chunk Handling

The server must efficiently receive, store, and reassemble chunks while handling concurrent uploads from millions of users. This requires careful architecture design.

Converting Mermaid diagram...

Server Architecture Components

•Upload Pods — Stateless services that receive chunks. Any pod can receive any chunk (no affinity needed). Chunks are immediately streamed to storage, not buffered in memory.
•Session State (Redis) — Tracks upload sessions: which chunks received, expiry times, expected hashes. Must survive pod restarts, hence external storage.
•Temporary Chunk Storage — Chunks stored temporarily in object storage (S3) with upload ID prefix. Lifecycle policies auto-delete orphaned chunks after expiry.
•Assembly Queue — When all chunks received, message sent to queue. Decouples upload from assembly for better scaling.
•Assembler Workers — Read chunks from temporary storage, assemble into final file, verify hash, move to permanent storage, cleanup temp chunks.

server_chunk_handler.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
// Server-side chunk handling (simplified)
interface ChunkUploadRequest {
    uploadId: string;
    chunkIndex: number;
    contentMD5: string;
    data: Buffer;
}
 
class ChunkHandler {
    constructor(
        private redis: RedisClient,
        private storage: ObjectStorage,
        private queue: MessageQueue,
    ) {}
    
    async handleChunkUpload(req: ChunkUploadRequest): Promise<void> {
        // 1. Validate session exists and is not expired
        const session = await this.redis.get(`upload:${req.uploadId}`);
        if (!session) throw new UploadExpiredError();
        
        // 2. Verify chunk hash
        const actualHash = md5(req.data);
        if (actualHash !== req.contentMD5) {
            throw new ChunkCorruptedError('Hash mismatch');
        }
        
        // 3. Store chunk (streaming write, not buffered)
        const chunkKey = `chunks/${req.uploadId}/${req.chunkIndex}`;
        await this.storage.put(chunkKey, req.data, {
            contentMD5: req.contentMD5,
            metadata: { uploadId: req.uploadId, index: req.chunkIndex },
        });
        
        // 4. Mark chunk as received in session
        const allReceived = await this.redis.eval(`
            redis.call('SADD', KEYS[1], ARGV[1])
            local count = redis.call('SCARD', KEYS[1])
            local expected = redis.call('HGET', KEYS[2], 'totalChunks')
            return count >= tonumber(expected) and 1 or 0
        `, [`upload:${req.uploadId}:chunks`, `upload:${req.uploadId}`], [req.chunkIndex]);
        
        // 5. If all chunks received, queue assembly
        if (allReceived) {
            await this.queue.send('file-assembly', {
                uploadId: req.uploadId,
                timestamp: Date.now(),
            });
        }
    }
    
    async assembleFile(uploadId: string): Promise<string> {
        const session = await this.redis.hgetall(`upload:${uploadId}`);
        const totalChunks = parseInt(session.totalChunks);
        
        // Stream chunks to final location
        const finalKey = `files/${session.userId}/${session.filename}`;
        const multipartUpload = await this.storage.createMultipartUpload(finalKey);
        
        for (let i = 0; i < totalChunks; i++) {
            const chunkKey = `chunks/${uploadId}/${i}`;
            await this.storage.copyPart(
                multipartUpload, 
                chunkKey, 
                i + 1  // Parts are 1-indexed
            );
        }
        
        const result = await this.storage.completeMultipartUpload(multipartUpload);
        
        // Verify final hash
        if (result.etag !== session.expectedHash) {
            await this.storage.delete(finalKey);
            throw new AssemblyError('Final hash mismatch');
        }
        
        // Cleanup temp chunks
        await this.cleanupTempChunks(uploadId, totalChunks);
        await this.redis.del(`upload:${uploadId}`, `upload:${uploadId}:chunks`);
        
        return result.fileId;
    }
}

S3 Multipart Upload

AWS S3 (and compatible storage) has native multipart upload support. Client can upload parts directly to S3 with pre-signed URLs, bypassing your servers entirely for large files. This dramatically reduces server load and bandwidth costs—the data flows Client → S3 directly, not Client → Server → S3.

Direct-to-Storage Uploads

For maximum efficiency, production systems enable clients to upload directly to object storage (S3, GCS, Azure Blob) using pre-signed URLs. This bypasses your API servers entirely for the bulk data transfer.

Converting Mermaid diagram...

Benefits of Direct Upload

•Reduced Server Load — API servers only handle metadata, not gigabytes of file data. 100x+ reduction in server resources for large file uploads.
•Lower Bandwidth Costs — Data doesn't traverse your infrastructure. Client → S3 direct connection. Egress from S3 to client, not server to S3.
•Better Performance — S3 has edge locations globally. Client may upload to closer S3 endpoint than your servers.
•Simpler Scaling — S3 auto-scales to any upload volume. No need to provision upload servers for peak load.
•Built-in Resumability — S3 multipart upload is inherently resumable. Each part can be retried independently.

presigned_urls.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
// Generate presigned URLs for direct-to-S3 upload
import { S3Client, CreateMultipartUploadCommand, UploadPartCommand } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
 
async function initializeDirectUpload(
    userId: string,
    filename: string,
    fileSize: number,
    chunkSize: number = 10 * 1024 * 1024 // 10MB
): Promise<DirectUploadSession> {
    const s3 = new S3Client({ region: 'us-east-1' });
    const key = `uploads/${userId}/${Date.now()}-${filename}`;
    
    // Create multipart upload
    const createCommand = new CreateMultipartUploadCommand({
        Bucket: 'my-bucket',
        Key: key,
        ContentType: getMimeType(filename),
    });
    const { UploadId } = await s3.send(createCommand);
    
    // Calculate number of parts
    const numParts = Math.ceil(fileSize / chunkSize);
    
    // Generate presigned URLs for each part
    const presignedUrls: PresignedPart[] = [];
    for (let partNumber = 1; partNumber <= numParts; partNumber++) {
        const uploadPartCommand = new UploadPartCommand({
            Bucket: 'my-bucket',
            Key: key,
            UploadId,
            PartNumber: partNumber,
        });
        
        const url = await getSignedUrl(s3, uploadPartCommand, {
            expiresIn: 3600 * 24, // 24 hours
        });
        
        presignedUrls.push({
            partNumber,
            url,
            startByte: (partNumber - 1) * chunkSize,
            endByte: Math.min(partNumber * chunkSize, fileSize),
        });
    }
    
    // Store session in database
    await db.uploadSessions.create({
        uploadId: UploadId,
        userId,
        key,
        filename,
        status: 'in_progress',
        createdAt: new Date(),
        expiresAt: new Date(Date.now() + 24 * 3600 * 1000),
    });
    
    return {
        uploadId: UploadId,
        key,
        presignedUrls,
        chunkSize,
    };
}

Security Considerations

Presigned URLs are bearer tokens—anyone with the URL can upload. Mitigations: (1) Short expiry times (1-24 hours), (2) Include content-type and content-length in signature to prevent misuse, (3) Use separate buckets for uploads with restricted permissions, (4) Validate uploaded content before accepting (virus scan, format validation).

Upload Validation and Security

Before accepting uploaded files into the system, rigorous validation ensures data integrity and security. This is especially critical when clients upload directly to storage.

Validation Layers

•Hash Verification — Compare final file hash against client-provided expected hash. Detects corruption during transfer. Required for data integrity guarantee.
•Size Validation — Verify final size matches declared size. Prevents truncated uploads from being accepted. Also enforces storage quotas.
•Content-Type Verification — Don't trust client-provided MIME type. Use magic bytes to detect actual file type. Prevents disguising executables as images.
•Virus Scanning — Scan uploaded files for malware. ClamAV for open-source, commercial solutions for enterprise. Quarantine suspicious files.
•Format Validation — For known formats (images, documents), validate they're well-formed. Malformed files might exploit viewer vulnerabilities.
•Zip Bomb Detection — Check compressed files for 'zip bombs' (files that expand to enormous sizes). Limit expansion ratio.

upload_validator.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
// Comprehensive upload validation pipeline
class UploadValidator {
    async validateUpload(
        uploadId: string, 
        expectedHash: string,
        expectedSize: number,
        declaredType: string
    ): Promise<ValidationResult> {
        const errors: string[] = [];
        const warnings: string[] = [];
        
        // 1. Size verification
        const actualSize = await this.storage.getSize(`uploads/${uploadId}`);
        if (actualSize !== expectedSize) {
            errors.push(`Size mismatch: expected ${expectedSize}, got ${actualSize}`);
        }
        
        // 2. Hash verification
        const actualHash = await this.storage.computeHash(`uploads/${uploadId}`);
        if (actualHash !== expectedHash) {
            errors.push('Hash mismatch: file may be corrupted');
        }
        
        // 3. Content type verification
        const magicBytes = await this.storage.readBytes(`uploads/${uploadId}`, 0, 8);
        const detectedType = this.detectMimeType(magicBytes);
        
        if (detectedType !== declaredType) {
            if (this.isDangerous(detectedType)) {
                errors.push(`Dangerous file type detected: ${detectedType}`);
            } else {
                warnings.push(`Type mismatch: claimed ${declaredType}, detected ${detectedType}`);
            }
        }
        
        // 4. Virus scan (async, may take time)
        const scanResult = await this.virusScanner.scan(`uploads/${uploadId}`);
        if (scanResult.infected) {
            errors.push(`Malware detected: ${scanResult.threat}`);
            await this.quarantineFile(uploadId);
        }
        
        // 5. Format-specific validation
        if (this.isImage(detectedType)) {
            const imageValid = await this.validateImage(`uploads/${uploadId}`);
            if (!imageValid) errors.push('Invalid image format');
        }
        
        // 6. Zip bomb detection
        if (this.isArchive(detectedType)) {
            const compressionRatio = await this.getCompressionRatio(`uploads/${uploadId}`);
            if (compressionRatio > 100) {  // 100:1 expansion ratio
                errors.push('Suspicious compression ratio (potential zip bomb)');
            }
        }
        
        return {
            valid: errors.length === 0,
            errors,
            warnings,
            metadata: {
                actualSize,
                actualHash,
                detectedType,
            },
        };
    }
    
    // MIME type detection via magic bytes
    detectMimeType(bytes: Buffer): string {
        // PNG: 89 50 4E 47
        if (bytes.slice(0, 4).equals(Buffer.from([0x89, 0x50, 0x4E, 0x47]))) {
            return 'image/png';
        }
        // JPEG: FF D8 FF
        if (bytes.slice(0, 3).equals(Buffer.from([0xFF, 0xD8, 0xFF]))) {
            return 'image/jpeg';
        }
        // PDF: 25 50 44 46
        if (bytes.slice(0, 4).equals(Buffer.from([0x25, 0x50, 0x44, 0x46]))) {
            return 'application/pdf';
        }
        // ... more types
        return 'application/octet-stream';
    }
}

Async Validation Pipeline

Synchronous validation delays upload completion. For better UX, accept the upload immediately with 'processing' status, then validate asynchronously. Notify user if validation fails. This improves perceived performance while maintaining security.

Storage Optimization

Efficient chunk storage requires content-addressed storage with deduplication. When multiple users upload the same file (or file has common chunks), we store each unique chunk only once.

Content-Addressed Storage Model:

Traditional Path-Based Storage:
  /users/alice/report.pdf  →  [file content]
  /users/bob/report.pdf    →  [same file content]  (duplicate!)

Content-Addressed Storage:
  /chunks/sha256-abc123... →  [chunk A]
  /chunks/sha256-def456... →  [chunk B]
  
  /files/alice/report.pdf  →  manifest: [abc123, def456, ghi789]
  /files/bob/report.pdf    →  manifest: [abc123, def456, ghi789]
  
  Same chunks, different manifests. Storage saved!

Deduplication Strategies

•Inline Dedup — Check for duplicate during upload. Before storing chunk, query if hash exists. Skip storage if already present. Immediate space savings.
•Post-Process Dedup — Store all chunks first, then async job identifies and removes duplicates. Simpler upload path but delayed savings.
•Per-User Dedup — Only deduplicate within single user's files. Simpler privacy model. Less effective for shared content.
•Global Dedup — Deduplicate across all users. Maximum storage efficiency. Privacy concerns (can probe for file existence).

Deduplication Effectiveness by File Type
File Type	Typical Dedup Ratio	Reason
Source code repos	60-80%	Many common files (libraries, configs)
Office documents	40-60%	Common templates, shared files
Photos (same camera)	20-30%	Similar headers, embedded profiles
Compressed media	5-15%	Already compressed, unique content
Random data	0-1%	No patterns to deduplicate

dedup_storage.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
// Content-addressed chunk storage with deduplication
interface ChunkManifest {
    fileId: string;
    chunks: string[];  // Array of chunk hashes
    totalSize: number;
    createdAt: Date;
}
 
class DedupStorage {
    // Upload chunk with inline deduplication
    async storeChunk(content: Buffer): Promise<{ hash: string; stored: boolean }> {
        const hash = sha256(content);
        
        // Check if chunk already exists
        const exists = await this.storage.exists(`chunks/${hash}`);
        
        if (exists) {
            // Increment reference count
            await this.incrementRefCount(hash);
            return { hash, stored: false };  // Deduped!
        }
        
        // Store new chunk
        await this.storage.put(`chunks/${hash}`, content, {
            contentType: 'application/octet-stream',
            metadata: { refCount: 1 },
        });
        
        return { hash, stored: true };
    }
    
    // Create file manifest from chunks
    async createManifest(
        fileId: string, 
        chunkHashes: string[], 
        totalSize: number
    ): Promise<void> {
        const manifest: ChunkManifest = {
            fileId,
            chunks: chunkHashes,
            totalSize,
            createdAt: new Date(),
        };
        
        await this.db.manifests.create(manifest);
    }
    
    // Read file by streaming chunks
    async* streamFile(fileId: string): AsyncGenerator<Buffer> {
        const manifest = await this.db.manifests.findById(fileId);
        
        for (const chunkHash of manifest.chunks) {
            const chunk = await this.storage.get(`chunks/${chunkHash}`);
            yield chunk;
        }
    }
    
    // Delete file (decrement refs, garbage collect)
    async deleteFile(fileId: string): Promise<void> {
        const manifest = await this.db.manifests.findById(fileId);
        
        // Decrement reference counts
        for (const chunkHash of manifest.chunks) {
            const refCount = await this.decrementRefCount(chunkHash);
            
            if (refCount === 0) {
                // No more references, chunk can be deleted
                await this.storage.delete(`chunks/${chunkHash}`);
            }
        }
        
        await this.db.manifests.delete(fileId);
    }
}

Reference Counting Complexity

Reference counting seems simple but has edge cases: What if decrement crashes after delete but before updating count? Use transactions or idempotent operations. Some systems use periodic garbage collection instead—scan for unreferenced chunks and delete. Simpler but delayed space recovery.

Summary: Chunked Upload Mastery

Chunked uploads are essential for handling large files reliably. Let's consolidate the key concepts:

Key Takeaways

•Chunking enables reliability — Breaking files into chunks enables resumability, parallel upload, and progress tracking. Essential for files larger than a few megabytes.
•CDC beats fixed-size for sync — Content-defined chunking with rolling hashes enables efficient delta sync. Only changed portions need re-upload.
•Resumable protocols preserve progress — Session-based protocols track upload state. Any chunk can be retried independently. Clients resume from where they left off.
•Parallel upload maximizes speed — Multiple concurrent chunk uploads overcome latency overhead. Adaptive concurrency based on connection quality.
•Direct-to-storage reduces load — Presigned URLs let clients upload directly to S3. API servers only handle metadata. Dramatically reduces infrastructure cost.
•Validation ensures integrity and security — Hash verification, content type validation, virus scanning, and format validation protect against corruption and attacks.
•Content-addressed storage enables dedup — Storing chunks by hash enables deduplication. Multiple files sharing chunks save significant storage.

What's Next:

With upload and sync mechanics covered, the next page explores Version History—how systems track file changes over time, enable rollback, and manage storage costs of keeping historical versions.

Page Complete

You now understand the complete chunked upload pipeline: from client-side chunking through parallel upload to server-side assembly and storage optimization. These techniques enable reliable upload of files of any size over any network condition. Next, we explore version history and its architectural implications.

4 / 6

Loading learning content...

System Design (HLD)Google Drive / Dropbox

Designing Cloud File Storage Systems

LevelAdvanced

Duration90 mins

TopicGoogle Drive / Dropbox

4 / 6

Chunked Uploads

The Large File Problem

The Core Problems:

Unreliable Networks — Connections drop. WiFi switches. Laptops close. Any interruption loses all progress.
Timeout Limits — Load balancers, proxies, and servers have timeout limits (often 30-60 seconds). A slow upload might not complete in time.
Memory Constraints — Server can't buffer an entire 50GB file in memory before writing to storage.
Progress Feedback — Users need to see upload progress. Single-request uploads provide poor progress granularity.
Retry Efficiency — If upload fails at 99%, we shouldn't re-upload the entire file.

The Solution: Chunked Uploads

Break large files into smaller chunks (typically 4-16MB each) and upload each chunk independently. This enables resumption, parallel uploads, and progress tracking.

Learning Objectives

Chunking Strategies

There are two fundamental approaches to dividing files into chunks: fixed-size chunking and content-defined chunking (CDC). Each has distinct trade-offs.

Fixed-Size Chunking

•Divide file at fixed byte boundaries (e.g., every 8MB)
•Simple to implement: chunk N = bytes[N*size : (N+1)*size]
•Predictable chunk count: ceil(fileSize / chunkSize)
•Problem: Insert 1 byte at start, ALL chunks shift
•Poor deduplication for modified files
•Good for: initial upload of new files

Content-Defined Chunking

•Boundaries determined by content pattern
•Uses rolling hash to find 'cut points'
•Insert 1 byte → only nearby chunks affected
•Excellent for deduplication of modified files
•Variable chunk sizes (need min/max bounds)
•Good for: sync, backup, modified file upload

Content-Defined Chunking (CDC) Algorithm:

Rolling Hash Chunking:

1. Initialize rolling hash window (e.g., 48 bytes)
2. Slide window byte-by-byte through file
3. At each position, check if hash matches pattern:
   if (hash & MASK) == TARGET:  // e.g., last 13 bits are zero
       Create chunk boundary here
4. Enforce minimum chunk size (don't cut too often)
5. Enforce maximum chunk size (cut even without pattern match)

Example with average 8KB chunks:
  MASK = 0x1FFF (13 bits)
  TARGET = 0
  Expected avg chunk: 2^13 = 8192 bytes
  Min chunk: 2KB (avoid tiny chunks)
  Max chunk: 64KB (ensure progress on non-matching content)

Chunk Size Trade-offs
Chunk Size	Chunks per 1GB	Metadata Overhead	Dedup Efficiency	Best Use Case
256 KB	4,096	High	Excellent	Text documents, code
1 MB	1,024	Moderate	Very Good	Mixed content
4 MB	256	Low	Good	General purpose
8 MB	128	Very Low	Moderate	Large media files
16 MB	64	Minimal	Lower	Streaming video

Hybrid Approach

Resumable Upload Protocol

A resumable upload protocol enables clients to continue interrupted uploads without retransmitting already-uploaded chunks. Here's how a well-designed protocol works:

Converting Mermaid diagram...

Protocol Design Principles

•Idempotent Operations — Re-uploading the same chunk with same content has no effect. Safe to retry on any failure.
•Upload Session State — Server tracks which chunks received. Client can query: 'Which chunks do you have?' and resume from there.
•Session Expiry — Upload sessions expire after inactivity (typically 24-72 hours). Prevents orphaned session accumulation.
•Chunk Verification — Each chunk includes MD5/SHA256 hash. Server verifies before acknowledging. Corrupted chunks rejected.
•Final Assembly — After all chunks received, server assembles file and verifies overall hash matches expected.
•Atomic Completion — File doesn't appear to other users until fully uploaded and verified. No partial files visible.

resumable_upload.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
// Resumable upload client implementation
interface UploadSession {
    uploadId: string;
    uploadUrl: string;
    expiry: Date;
    uploadedChunks: Set<number>;
}
 
class ResumableUploader {
    private chunkSize = 4 * 1024 * 1024; // 4MB
    
    async uploadFile(file: File, onProgress: (pct: number) => void): Promise<string> {
        // Phase 1: Initialize or resume session
        const session = await this.getOrCreateSession(file);
        
        // Phase 2: Upload chunks
        const totalChunks = Math.ceil(file.size / this.chunkSize);
        
        for (let i = 0; i < totalChunks; i++) {
            if (session.uploadedChunks.has(i)) continue; // Already uploaded
            
            const start = i * this.chunkSize;
            const end = Math.min(start + this.chunkSize, file.size);
            const chunk = file.slice(start, end);
            
            await this.uploadChunkWithRetry(session, i, chunk);
            onProgress((i + 1) / totalChunks * 100);
            
            this.saveSessionToLocalStorage(session); // Persist progress
        }
        
        // Phase 3: Complete upload
        const fileId = await this.completeUpload(session, file);
        this.clearSession(session.uploadId);
        
        return fileId;
    }
    
    private async uploadChunkWithRetry(
        session: UploadSession, 
        index: number, 
        chunk: Blob,
        maxRetries = 3
    ): Promise<void> {
        const hash = await this.computeHash(chunk);
        
        for (let attempt = 0; attempt < maxRetries; attempt++) {
            try {
                const response = await fetch(
                    `${session.uploadUrl}/chunk/${index}`,
                    {
                        method: 'PUT',
                        headers: {
                            'Content-Type': 'application/octet-stream',
                            'Content-MD5': hash,
                            'Content-Range': `bytes ${index * this.chunkSize}-${index * this.chunkSize + chunk.size - 1}`,
                        },
                        body: chunk,
                    }
                );
                
                if (response.ok) {
                    session.uploadedChunks.add(index);
                    return;
                }
                
                if (response.status === 409) {
                    // Chunk already exists (idempotent retry)
                    session.uploadedChunks.add(index);
                    return;
                }
                
                throw new Error(`Upload failed: ${response.status}`);
            } catch (error) {
                if (attempt === maxRetries - 1) throw error;
                await this.exponentialBackoff(attempt);
            }
        }
    }
    
    private async getOrCreateSession(file: File): Promise<UploadSession> {
        // Check for existing session in localStorage
        const existing = this.loadSessionFromLocalStorage(file.name, file.size);
        if (existing) {
            // Verify session still valid on server
            const status = await this.getSessionStatus(existing.uploadId);
            if (status.valid) {
                existing.uploadedChunks = new Set(status.uploadedChunks);
                return existing;
            }
        }
        
        // Create new session
        const response = await fetch('/api/uploads/init', {
            method: 'POST',
            headers: { 'Content-Type': 'application/json' },
            body: JSON.stringify({
                filename: file.name,
                size: file.size,
                chunkSize: this.chunkSize,
                contentHash: await this.computeFileHash(file),
            }),
        });
        
        return response.json();
    }
}

The tus Protocol

Parallel Chunk Upload

Why Parallel Uploads Are Faster:

Sequential Upload (1 chunk at a time):
  Total Time = N × (latency + chunk_transfer_time)
  
  For 10 chunks, 100ms latency, 2s transfer each:
  Total = 10 × (0.1 + 2) = 21 seconds

Parallel Upload (4 concurrent):
  Total Time ≈ ceil(N/4) × (latency + chunk_transfer_time)
  
  For 10 chunks, 100ms latency, 2s transfer each:
  Total ≈ 3 × (0.1 + 2) = 6.3 seconds
  
  3.3x faster!

The key insight: while one chunk is transferring, other connections can be established and chunks can be queued. TCP slow-start is amortized across multiple connections.

parallel_upload.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
// Parallel chunk upload with concurrency control
class ParallelUploader {
    private concurrency = 4;  // Simultaneous uploads
    private chunkSize = 4 * 1024 * 1024;
    
    async uploadFile(file: File, session: UploadSession): Promise<void> {
        const totalChunks = Math.ceil(file.size / this.chunkSize);
        const pendingChunks: number[] = [];
        
        // Build list of chunks that need uploading
        for (let i = 0; i < totalChunks; i++) {
            if (!session.uploadedChunks.has(i)) {
                pendingChunks.push(i);
            }
        }
        
        // Process chunks with limited concurrency
        await this.processWithConcurrency(
            pendingChunks,
            async (chunkIndex) => {
                const start = chunkIndex * this.chunkSize;
                const end = Math.min(start + this.chunkSize, file.size);
                const chunk = file.slice(start, end);
                
                await this.uploadChunk(session, chunkIndex, chunk);
                session.uploadedChunks.add(chunkIndex);
            },
            this.concurrency
        );
    }
    
    private async processWithConcurrency<T>(
        items: T[],
        processor: (item: T) => Promise<void>,
        limit: number
    ): Promise<void> {
        const queue = [...items];
        const inFlight: Promise<void>[] = [];
        
        const processNext = async (): Promise<void> => {
            while (queue.length > 0) {
                const item = queue.shift()!;
                await processor(item);
            }
        };
        
        // Start 'limit' concurrent workers
        const workers = Array(Math.min(limit, items.length))
            .fill(null)
            .map(() => processNext());
        
        await Promise.all(workers);
    }
}
 
// Advanced: Priority queue for smart chunk ordering
class SmartParallelUploader extends ParallelUploader {
    // Upload chunks near current read position first
    // This enables streaming playback during upload
    getChunkPriority(chunkIndex: number, totalChunks: number): number {
        // Priority: first few chunks (for preview), then sequential
        if (chunkIndex < 3) return 0;  // Highest priority
        return chunkIndex;  // Then sequential
    }
    
    // Adaptive concurrency based on bandwidth
    async measureBandwidth(): Promise<number> {
        const testChunk = new Uint8Array(64 * 1024); // 64KB test
        const start = performance.now();
        await this.uploadTestChunk(testChunk);
        const elapsed = performance.now() - start;
        
        const mbps = (64 / 1024) / (elapsed / 1000);
        
        // Adjust concurrency based on observed bandwidth
        if (mbps > 10) return 6;     // Fast connection: more parallel
        if (mbps > 2) return 4;      // Medium: moderate parallel
        return 2;                     // Slow: fewer parallel
    }
}

Optimal Concurrency by Connection Type
Connection	Bandwidth	Latency	Optimal Concurrency
Slow WiFi	1-5 Mbps	50-200ms	2
Home Broadband	10-50 Mbps	20-50ms	4
Fast Fiber	100-500 Mbps	5-20ms	6-8
Enterprise LAN	1+ Gbps	1-5ms	8-16
Same Datacenter	10+ Gbps	<1ms	16-32

Browser Limitations

Server-Side Chunk Handling

The server must efficiently receive, store, and reassemble chunks while handling concurrent uploads from millions of users. This requires careful architecture design.

Converting Mermaid diagram...

Server Architecture Components

•Upload Pods — Stateless services that receive chunks. Any pod can receive any chunk (no affinity needed). Chunks are immediately streamed to storage, not buffered in memory.
•Session State (Redis) — Tracks upload sessions: which chunks received, expiry times, expected hashes. Must survive pod restarts, hence external storage.
•Temporary Chunk Storage — Chunks stored temporarily in object storage (S3) with upload ID prefix. Lifecycle policies auto-delete orphaned chunks after expiry.
•Assembly Queue — When all chunks received, message sent to queue. Decouples upload from assembly for better scaling.
•Assembler Workers — Read chunks from temporary storage, assemble into final file, verify hash, move to permanent storage, cleanup temp chunks.

server_chunk_handler.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
// Server-side chunk handling (simplified)
interface ChunkUploadRequest {
    uploadId: string;
    chunkIndex: number;
    contentMD5: string;
    data: Buffer;
}
 
class ChunkHandler {
    constructor(
        private redis: RedisClient,
        private storage: ObjectStorage,
        private queue: MessageQueue,
    ) {}
    
    async handleChunkUpload(req: ChunkUploadRequest): Promise<void> {
        // 1. Validate session exists and is not expired
        const session = await this.redis.get(`upload:${req.uploadId}`);
        if (!session) throw new UploadExpiredError();
        
        // 2. Verify chunk hash
        const actualHash = md5(req.data);
        if (actualHash !== req.contentMD5) {
            throw new ChunkCorruptedError('Hash mismatch');
        }
        
        // 3. Store chunk (streaming write, not buffered)
        const chunkKey = `chunks/${req.uploadId}/${req.chunkIndex}`;
        await this.storage.put(chunkKey, req.data, {
            contentMD5: req.contentMD5,
            metadata: { uploadId: req.uploadId, index: req.chunkIndex },
        });
        
        // 4. Mark chunk as received in session
        const allReceived = await this.redis.eval(`
            redis.call('SADD', KEYS[1], ARGV[1])
            local count = redis.call('SCARD', KEYS[1])
            local expected = redis.call('HGET', KEYS[2], 'totalChunks')
            return count >= tonumber(expected) and 1 or 0
        `, [`upload:${req.uploadId}:chunks`, `upload:${req.uploadId}`], [req.chunkIndex]);
        
        // 5. If all chunks received, queue assembly
        if (allReceived) {
            await this.queue.send('file-assembly', {
                uploadId: req.uploadId,
                timestamp: Date.now(),
            });
        }
    }
    
    async assembleFile(uploadId: string): Promise<string> {
        const session = await this.redis.hgetall(`upload:${uploadId}`);
        const totalChunks = parseInt(session.totalChunks);
        
        // Stream chunks to final location
        const finalKey = `files/${session.userId}/${session.filename}`;
        const multipartUpload = await this.storage.createMultipartUpload(finalKey);
        
        for (let i = 0; i < totalChunks; i++) {
            const chunkKey = `chunks/${uploadId}/${i}`;
            await this.storage.copyPart(
                multipartUpload, 
                chunkKey, 
                i + 1  // Parts are 1-indexed
            );
        }
        
        const result = await this.storage.completeMultipartUpload(multipartUpload);
        
        // Verify final hash
        if (result.etag !== session.expectedHash) {
            await this.storage.delete(finalKey);
            throw new AssemblyError('Final hash mismatch');
        }
        
        // Cleanup temp chunks
        await this.cleanupTempChunks(uploadId, totalChunks);
        await this.redis.del(`upload:${uploadId}`, `upload:${uploadId}:chunks`);
        
        return result.fileId;
    }
}

S3 Multipart Upload

Direct-to-Storage Uploads

Converting Mermaid diagram...

Benefits of Direct Upload

•Reduced Server Load — API servers only handle metadata, not gigabytes of file data. 100x+ reduction in server resources for large file uploads.
•Lower Bandwidth Costs — Data doesn't traverse your infrastructure. Client → S3 direct connection. Egress from S3 to client, not server to S3.
•Better Performance — S3 has edge locations globally. Client may upload to closer S3 endpoint than your servers.
•Simpler Scaling — S3 auto-scales to any upload volume. No need to provision upload servers for peak load.
•Built-in Resumability — S3 multipart upload is inherently resumable. Each part can be retried independently.

presigned_urls.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
// Generate presigned URLs for direct-to-S3 upload
import { S3Client, CreateMultipartUploadCommand, UploadPartCommand } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
 
async function initializeDirectUpload(
    userId: string,
    filename: string,
    fileSize: number,
    chunkSize: number = 10 * 1024 * 1024 // 10MB
): Promise<DirectUploadSession> {
    const s3 = new S3Client({ region: 'us-east-1' });
    const key = `uploads/${userId}/${Date.now()}-${filename}`;
    
    // Create multipart upload
    const createCommand = new CreateMultipartUploadCommand({
        Bucket: 'my-bucket',
        Key: key,
        ContentType: getMimeType(filename),
    });
    const { UploadId } = await s3.send(createCommand);
    
    // Calculate number of parts
    const numParts = Math.ceil(fileSize / chunkSize);
    
    // Generate presigned URLs for each part
    const presignedUrls: PresignedPart[] = [];
    for (let partNumber = 1; partNumber <= numParts; partNumber++) {
        const uploadPartCommand = new UploadPartCommand({
            Bucket: 'my-bucket',
            Key: key,
            UploadId,
            PartNumber: partNumber,
        });
        
        const url = await getSignedUrl(s3, uploadPartCommand, {
            expiresIn: 3600 * 24, // 24 hours
        });
        
        presignedUrls.push({
            partNumber,
            url,
            startByte: (partNumber - 1) * chunkSize,
            endByte: Math.min(partNumber * chunkSize, fileSize),
        });
    }
    
    // Store session in database
    await db.uploadSessions.create({
        uploadId: UploadId,
        userId,
        key,
        filename,
        status: 'in_progress',
        createdAt: new Date(),
        expiresAt: new Date(Date.now() + 24 * 3600 * 1000),
    });
    
    return {
        uploadId: UploadId,
        key,
        presignedUrls,
        chunkSize,
    };
}

Security Considerations

Upload Validation and Security

Before accepting uploaded files into the system, rigorous validation ensures data integrity and security. This is especially critical when clients upload directly to storage.

Validation Layers

•Hash Verification — Compare final file hash against client-provided expected hash. Detects corruption during transfer. Required for data integrity guarantee.
•Size Validation — Verify final size matches declared size. Prevents truncated uploads from being accepted. Also enforces storage quotas.
•Content-Type Verification — Don't trust client-provided MIME type. Use magic bytes to detect actual file type. Prevents disguising executables as images.
•Virus Scanning — Scan uploaded files for malware. ClamAV for open-source, commercial solutions for enterprise. Quarantine suspicious files.
•Format Validation — For known formats (images, documents), validate they're well-formed. Malformed files might exploit viewer vulnerabilities.
•Zip Bomb Detection — Check compressed files for 'zip bombs' (files that expand to enormous sizes). Limit expansion ratio.

upload_validator.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
// Comprehensive upload validation pipeline
class UploadValidator {
    async validateUpload(
        uploadId: string, 
        expectedHash: string,
        expectedSize: number,
        declaredType: string
    ): Promise<ValidationResult> {
        const errors: string[] = [];
        const warnings: string[] = [];
        
        // 1. Size verification
        const actualSize = await this.storage.getSize(`uploads/${uploadId}`);
        if (actualSize !== expectedSize) {
            errors.push(`Size mismatch: expected ${expectedSize}, got ${actualSize}`);
        }
        
        // 2. Hash verification
        const actualHash = await this.storage.computeHash(`uploads/${uploadId}`);
        if (actualHash !== expectedHash) {
            errors.push('Hash mismatch: file may be corrupted');
        }
        
        // 3. Content type verification
        const magicBytes = await this.storage.readBytes(`uploads/${uploadId}`, 0, 8);
        const detectedType = this.detectMimeType(magicBytes);
        
        if (detectedType !== declaredType) {
            if (this.isDangerous(detectedType)) {
                errors.push(`Dangerous file type detected: ${detectedType}`);
            } else {
                warnings.push(`Type mismatch: claimed ${declaredType}, detected ${detectedType}`);
            }
        }
        
        // 4. Virus scan (async, may take time)
        const scanResult = await this.virusScanner.scan(`uploads/${uploadId}`);
        if (scanResult.infected) {
            errors.push(`Malware detected: ${scanResult.threat}`);
            await this.quarantineFile(uploadId);
        }
        
        // 5. Format-specific validation
        if (this.isImage(detectedType)) {
            const imageValid = await this.validateImage(`uploads/${uploadId}`);
            if (!imageValid) errors.push('Invalid image format');
        }
        
        // 6. Zip bomb detection
        if (this.isArchive(detectedType)) {
            const compressionRatio = await this.getCompressionRatio(`uploads/${uploadId}`);
            if (compressionRatio > 100) {  // 100:1 expansion ratio
                errors.push('Suspicious compression ratio (potential zip bomb)');
            }
        }
        
        return {
            valid: errors.length === 0,
            errors,
            warnings,
            metadata: {
                actualSize,
                actualHash,
                detectedType,
            },
        };
    }
    
    // MIME type detection via magic bytes
    detectMimeType(bytes: Buffer): string {
        // PNG: 89 50 4E 47
        if (bytes.slice(0, 4).equals(Buffer.from([0x89, 0x50, 0x4E, 0x47]))) {
            return 'image/png';
        }
        // JPEG: FF D8 FF
        if (bytes.slice(0, 3).equals(Buffer.from([0xFF, 0xD8, 0xFF]))) {
            return 'image/jpeg';
        }
        // PDF: 25 50 44 46
        if (bytes.slice(0, 4).equals(Buffer.from([0x25, 0x50, 0x44, 0x46]))) {
            return 'application/pdf';
        }
        // ... more types
        return 'application/octet-stream';
    }
}

Async Validation Pipeline

Storage Optimization

Efficient chunk storage requires content-addressed storage with deduplication. When multiple users upload the same file (or file has common chunks), we store each unique chunk only once.

Content-Addressed Storage Model:

Traditional Path-Based Storage:
  /users/alice/report.pdf  →  [file content]
  /users/bob/report.pdf    →  [same file content]  (duplicate!)

Content-Addressed Storage:
  /chunks/sha256-abc123... →  [chunk A]
  /chunks/sha256-def456... →  [chunk B]
  
  /files/alice/report.pdf  →  manifest: [abc123, def456, ghi789]
  /files/bob/report.pdf    →  manifest: [abc123, def456, ghi789]
  
  Same chunks, different manifests. Storage saved!

Deduplication Strategies

•Inline Dedup — Check for duplicate during upload. Before storing chunk, query if hash exists. Skip storage if already present. Immediate space savings.
•Post-Process Dedup — Store all chunks first, then async job identifies and removes duplicates. Simpler upload path but delayed savings.
•Per-User Dedup — Only deduplicate within single user's files. Simpler privacy model. Less effective for shared content.
•Global Dedup — Deduplicate across all users. Maximum storage efficiency. Privacy concerns (can probe for file existence).

Deduplication Effectiveness by File Type
File Type	Typical Dedup Ratio	Reason
Source code repos	60-80%	Many common files (libraries, configs)
Office documents	40-60%	Common templates, shared files
Photos (same camera)	20-30%	Similar headers, embedded profiles
Compressed media	5-15%	Already compressed, unique content
Random data	0-1%	No patterns to deduplicate

dedup_storage.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
// Content-addressed chunk storage with deduplication
interface ChunkManifest {
    fileId: string;
    chunks: string[];  // Array of chunk hashes
    totalSize: number;
    createdAt: Date;
}
 
class DedupStorage {
    // Upload chunk with inline deduplication
    async storeChunk(content: Buffer): Promise<{ hash: string; stored: boolean }> {
        const hash = sha256(content);
        
        // Check if chunk already exists
        const exists = await this.storage.exists(`chunks/${hash}`);
        
        if (exists) {
            // Increment reference count
            await this.incrementRefCount(hash);
            return { hash, stored: false };  // Deduped!
        }
        
        // Store new chunk
        await this.storage.put(`chunks/${hash}`, content, {
            contentType: 'application/octet-stream',
            metadata: { refCount: 1 },
        });
        
        return { hash, stored: true };
    }
    
    // Create file manifest from chunks
    async createManifest(
        fileId: string, 
        chunkHashes: string[], 
        totalSize: number
    ): Promise<void> {
        const manifest: ChunkManifest = {
            fileId,
            chunks: chunkHashes,
            totalSize,
            createdAt: new Date(),
        };
        
        await this.db.manifests.create(manifest);
    }
    
    // Read file by streaming chunks
    async* streamFile(fileId: string): AsyncGenerator<Buffer> {
        const manifest = await this.db.manifests.findById(fileId);
        
        for (const chunkHash of manifest.chunks) {
            const chunk = await this.storage.get(`chunks/${chunkHash}`);
            yield chunk;
        }
    }
    
    // Delete file (decrement refs, garbage collect)
    async deleteFile(fileId: string): Promise<void> {
        const manifest = await this.db.manifests.findById(fileId);
        
        // Decrement reference counts
        for (const chunkHash of manifest.chunks) {
            const refCount = await this.decrementRefCount(chunkHash);
            
            if (refCount === 0) {
                // No more references, chunk can be deleted
                await this.storage.delete(`chunks/${chunkHash}`);
            }
        }
        
        await this.db.manifests.delete(fileId);
    }
}

Reference Counting Complexity

Summary: Chunked Upload Mastery

Chunked uploads are essential for handling large files reliably. Let's consolidate the key concepts:

Key Takeaways

•Chunking enables reliability — Breaking files into chunks enables resumability, parallel upload, and progress tracking. Essential for files larger than a few megabytes.
•CDC beats fixed-size for sync — Content-defined chunking with rolling hashes enables efficient delta sync. Only changed portions need re-upload.
•Resumable protocols preserve progress — Session-based protocols track upload state. Any chunk can be retried independently. Clients resume from where they left off.
•Parallel upload maximizes speed — Multiple concurrent chunk uploads overcome latency overhead. Adaptive concurrency based on connection quality.
•Direct-to-storage reduces load — Presigned URLs let clients upload directly to S3. API servers only handle metadata. Dramatically reduces infrastructure cost.
•Validation ensures integrity and security — Hash verification, content type validation, virus scanning, and format validation protect against corruption and attacks.
•Content-addressed storage enables dedup — Storing chunks by hash enables deduplication. Multiple files sharing chunks save significant storage.

What's Next:

Page Complete

4 / 6