Hot, Warm, and Cold Storage

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

AWS S3 Monthly Cost Breakdown - Enterprise Data Lake
═══════════════════════════════════════════════════════════════════════════
 
STORAGE COSTS                                                    MONTHLY COST
─────────────────────────────────────────────────────────────────────────────
S3 Standard                     10 TB  × $0.023/GB          =    $235.52
S3 Standard-IA                  30 TB  × $0.0125/GB         =    $384.00
S3 Glacier Instant Retrieval    40 TB  × $0.004/GB          =    $163.84
S3 Glacier Flexible Retrieval   15 TB  × $0.0036/GB         =     $55.30
S3 Glacier Deep Archive          5 TB  × $0.00099/GB        =      $5.07
                                                            ────────────────
                                Storage Subtotal:                 $843.73
 
REQUEST COSTS
─────────────────────────────────────────────────────────────────────────────
S3 Standard PUT/POST (5M requests)      × $0.005/1K         =     $25.00
S3 Standard GET (50M requests)          × $0.0004/1K        =     $20.00
S3 Standard-IA GET (2M requests)        × $0.001/1K         =      $2.00
Glacier IR retrieval requests (10K)     × $0.01/1K          =      $0.10
Lifecycle transitions (100K)            × $0.01/1K          =      $1.00
                                                            ────────────────
                                Request Subtotal:                  $48.10
 
DATA RETRIEVAL COSTS
─────────────────────────────────────────────────────────────────────────────
S3 Standard-IA retrieval        500 GB  × $0.01/GB          =      $5.00
Glacier IR retrieval            200 GB  × $0.03/GB          =      $6.00
Glacier Flexible retrieval       50 GB  × $0.03/GB (expedited) =   $1.50
                                                            ────────────────
                                Retrieval Subtotal:                $12.50
 
DATA TRANSFER COSTS
─────────────────────────────────────────────────────────────────────────────
Data Transfer OUT to Internet   500 GB  × $0.09/GB          =     $45.00
Data Transfer to CloudFront    1000 GB  × $0.00/GB          =      $0.00
Cross-region replication       100 GB   × $0.02/GB          =      $2.00
                                                            ────────────────
                                Transfer Subtotal:                 $47.00
 
═══════════════════════════════════════════════════════════════════════════
TOTAL MONTHLY S3 COST:                                           $951.33
═══════════════════════════════════════════════════════════════════════════
 
COST ATTRIBUTION BY WORKLOAD:
  - Analytics Pipeline (40TB IA, high GET):                      $420.00 (44%)
  - Log Archive (20TB Glacier, low access):                      $180.00 (19%)
  - Application Media (8TB Standard, high access):               $210.00 (22%)
  - Compliance Archive (32TB Deep Archive):                      $141.33 (15%)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124

interface StorageTCOInputs {
  // Data profile
  initialDataVolumeGB: number;
  monthlyDataGrowthRate: number;  // e.g., 0.05 for 5% monthly growth
  dataRetentionMonths: number;
  
  // Access patterns
  monthlyReadGB: number;
  monthlyWriteGB: number;
  monthlyListOperations: number;
  monthlyGetOperations: number;
  monthlyPutOperations: number;
  
  // Transfer patterns
  monthlyEgressGB: number;
  crossRegionReplicationGB: number;
  
  // Tier distribution (must sum to 1.0)
  tierDistribution: {
    hot: number;
    warm: number;
    cool: number;
    cold: number;
    archive: number;
  };
  
  // Operational overhead
  engineeringHoursPerMonth: number;
  hourlyEngineeringCost: number;
  
  // Risk factors
  estimatedComplianceIncidentCost: number;
  incidentProbabilityPerYear: number;
}
 
interface StorageTCOOutput {
  monthlyDirectStorageCost: number;
  monthlyOperationsCost: number;
  monthlyTransferCost: number;
  monthlyRetrievalCost: number;
  monthlyManagementCost: number;
  annualizedRiskCost: number;
  totalMonthlyTCO: number;
  costPerGBMonth: number;
  totalLifetimeCost: number;
}
 
function calculateStorageTCO(inputs: StorageTCOInputs): StorageTCOOutput {
  // Storage tier pricing (AWS S3 us-east-1 as reference)
  const tierPricing = {
    hot:     { storagePerGB: 0.023, retrievalPerGB: 0.00, getRequestPer1K: 0.0004 },
    warm:    { storagePerGB: 0.0125, retrievalPerGB: 0.01, getRequestPer1K: 0.001 },
    cool:    { storagePerGB: 0.004, retrievalPerGB: 0.03, getRequestPer1K: 0.01 },
    cold:    { storagePerGB: 0.0036, retrievalPerGB: 0.03, getRequestPer1K: 0.03 },
    archive: { storagePerGB: 0.00099, retrievalPerGB: 0.05, getRequestPer1K: 0.05 }
  };
  
  const { tierDistribution } = inputs;
  
  // Calculate weighted average storage cost
  const storagePerGB = 
    tierDistribution.hot * tierPricing.hot.storagePerGB +
    tierDistribution.warm * tierPricing.warm.storagePerGB +
    tierDistribution.cool * tierPricing.cool.storagePerGB +
    tierDistribution.cold * tierPricing.cold.storagePerGB +
    tierDistribution.archive * tierPricing.archive.storagePerGB;
  
  // Direct storage cost
  const monthlyDirectStorageCost = inputs.initialDataVolumeGB * storagePerGB;
  
  // Operations cost
  const putCost = (inputs.monthlyPutOperations / 1000) * 0.005;
  const getCost = (inputs.monthlyGetOperations / 1000) * 0.0004;
  const listCost = (inputs.monthlyListOperations / 1000) * 0.005;
  const monthlyOperationsCost = putCost + getCost + listCost;
  
  // Transfer cost (assuming $0.09/GB for internet egress)
  const monthlyTransferCost = inputs.monthlyEgressGB * 0.09 + 
    inputs.crossRegionReplicationGB * 0.02;
  
  // Weighted retrieval cost
  const retrievalPerGB = 
    tierDistribution.hot * tierPricing.hot.retrievalPerGB +
    tierDistribution.warm * tierPricing.warm.retrievalPerGB +
    tierDistribution.cool * tierPricing.cool.retrievalPerGB +
    tierDistribution.cold * tierPricing.cold.retrievalPerGB +
    tierDistribution.archive * tierPricing.archive.retrievalPerGB;
  const monthlyRetrievalCost = inputs.monthlyReadGB * retrievalPerGB;
  
  // Management overhead
  const monthlyManagementCost = inputs.engineeringHoursPerMonth * 
    inputs.hourlyEngineeringCost;
  
  // Risk cost (annualized)
  const annualizedRiskCost = inputs.estimatedComplianceIncidentCost * 
    inputs.incidentProbabilityPerYear / 12;
  
  const totalMonthlyTCO = monthlyDirectStorageCost + monthlyOperationsCost +
    monthlyTransferCost + monthlyRetrievalCost + monthlyManagementCost +
    annualizedRiskCost;
  
  // Calculate lifetime cost with growth
  let totalLifetimeCost = 0;
  let currentVolume = inputs.initialDataVolumeGB;
  for (let month = 0; month < inputs.dataRetentionMonths; month++) {
    totalLifetimeCost += currentVolume * storagePerGB;
    currentVolume *= (1 + inputs.monthlyDataGrowthRate);
  }
  totalLifetimeCost += (monthlyOperationsCost + monthlyTransferCost + 
    monthlyRetrievalCost + monthlyManagementCost + annualizedRiskCost) * 
    inputs.dataRetentionMonths;
  
  return {
    monthlyDirectStorageCost,
    monthlyOperationsCost,
    monthlyTransferCost,
    monthlyRetrievalCost,
    monthlyManagementCost,
    annualizedRiskCost,
    totalMonthlyTCO,
    costPerGBMonth: totalMonthlyTCO / inputs.initialDataVolumeGB,
    totalLifetimeCost
  };
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112

interface CostAttributionReport {
  reportPeriod: { start: Date; end: Date };
  totalCost: number;
  byTeam: Record<string, TeamCostBreakdown>;
  unattributed: UnattributedCost;
  recommendations: CostRecommendation[];
}
 
interface TeamCostBreakdown {
  teamName: string;
  totalCost: number;
  storageCost: number;
  requestCost: number;
  retrievalCost: number;
  transferCost: number;
  volumeGB: number;
  costPerGB: number;
  monthOverMonthChange: number;  // percentage
  tierDistribution: Record<string, number>;
  topBuckets: Array<{ bucket: string; cost: number }>;
}
 
interface UnattributedCost {
  totalCost: number;
  percentageOfTotal: number;
  topUntaggedBuckets: string[];
}
 
interface CostRecommendation {
  team: string;
  recommendationType: 'tier_optimization' | 'deletion_opportunity' | 'tagging_gap';
  potentialMonthlySavings: number;
  description: string;
  affectedResources: string[];
}
 
async function generateCostAttributionReport(
  month: Date,
  costData: AWSCostData[],
  inventoryData: S3InventoryData[]
): Promise<CostAttributionReport> {
  
  const byTeam: Record<string, TeamCostBreakdown> = {};
  let unattributedCost = 0;
  const untaggedBuckets = new Set<string>();
  
  for (const cost of costData) {
    const team = cost.resourceTags?.team || 'unattributed';
    
    if (team === 'unattributed') {
      unattributedCost += cost.amount;
      untaggedBuckets.add(cost.resourceId);
      continue;
    }
    
    if (!byTeam[team]) {
      byTeam[team] = initializeTeamBreakdown(team);
    }
    
    // Categorize cost by type
    switch (cost.usageType) {
      case 'StorageUsage':
        byTeam[team].storageCost += cost.amount;
        break;
      case 'Requests':
        byTeam[team].requestCost += cost.amount;
        break;
      case 'DataTransfer':
        byTeam[team].transferCost += cost.amount;
        break;
      case 'Retrieval':
        byTeam[team].retrievalCost += cost.amount;
        break;
    }
    
    byTeam[team].totalCost += cost.amount;
  }
  
  // Enrich with volume and tier data from inventory
  for (const teamName in byTeam) {
    const teamInventory = inventoryData.filter(i => 
      i.tags?.team === teamName
    );
    
    byTeam[teamName].volumeGB = teamInventory.reduce(
      (sum, i) => sum + i.sizeBytes / (1024 * 1024 * 1024), 0
    );
    
    byTeam[teamName].costPerGB = byTeam[teamName].totalCost / 
      byTeam[teamName].volumeGB;
    
    byTeam[teamName].tierDistribution = calculateTierDistribution(teamInventory);
  }
  
  // Generate recommendations
  const recommendations = generateCostRecommendations(byTeam, inventoryData);
  
  const totalCost = Object.values(byTeam).reduce((sum, t) => sum + t.totalCost, 0) + 
    unattributedCost;
  
  return {
    reportPeriod: { start: startOfMonth(month), end: endOfMonth(month) },
    totalCost,
    byTeam,
    unattributed: {
      totalCost: unattributedCost,
      percentageOfTotal: (unattributedCost / totalCost) * 100,
      topUntaggedBuckets: Array.from(untaggedBuckets).slice(0, 10)
    },
    recommendations
  };
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79

import { createHash } from 'crypto';
 
interface DeduplicatedStore {
  // Store content by its hash; return reference
  storeContent(content: Buffer): Promise<string>;
  
  // Retrieve content by reference
  retrieveContent(reference: string): Promise<Buffer>;
  
  // Create logical object pointing to content references
  createObject(key: string, contentRefs: string[]): Promise<void>;
  
  // Get deduplication statistics
  getStats(): Promise<DeduplicationStats>;
}
 
interface DeduplicationStats {
  uniqueBlocksCount: number;
  totalReferencesCount: number;
  physicalStorageBytes: number;
  logicalStorageBytes: number;
  deduplicationRatio: number;  // logical / physical
  spaceSavedBytes: number;
}
 
class ContentAddressableStorage implements DeduplicatedStore {
  private blockSize = 1024 * 1024;  // 1MB blocks
  
  async storeContent(content: Buffer): Promise<string> {
    // Split into blocks
    const blocks: string[] = [];
    
    for (let offset = 0; offset < content.length; offset += this.blockSize) {
      const block = content.slice(offset, offset + this.blockSize);
      const hash = this.computeHash(block);
      
      // Check if block already exists
      if (!await this.blockExists(hash)) {
        await this.writeBlock(hash, block);
      }
      
      blocks.push(hash);
    }
    
    // Return manifest of block references
    const manifestHash = this.computeHash(Buffer.from(blocks.join(',')));
    await this.writeManifest(manifestHash, blocks);
    
    return manifestHash;
  }
  
  async retrieveContent(reference: string): Promise<Buffer> {
    const blocks = await this.readManifest(reference);
    const buffers = await Promise.all(
      blocks.map(hash => this.readBlock(hash))
    );
    return Buffer.concat(buffers);
  }
  
  private computeHash(data: Buffer): string {
    return createHash('sha256').update(data).digest('hex');
  }
  
  async getStats(): Promise<DeduplicationStats> {
    const uniqueBlocks = await this.countUniqueBlocks();
    const totalRefs = await this.countTotalReferences();
    const physicalBytes = await this.countPhysicalBytes();
    const logicalBytes = await this.countLogicalBytes();
    
    return {
      uniqueBlocksCount: uniqueBlocks,
      totalReferencesCount: totalRefs,
      physicalStorageBytes: physicalBytes,
      logicalStorageBytes: logicalBytes,
      deduplicationRatio: logicalBytes / physicalBytes,
      spaceSavedBytes: logicalBytes - physicalBytes
    };
  }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

import { S3Client, SelectObjectContentCommand } from '@aws-sdk/client-s3';
 
// Instead of this (downloads entire 10GB file):
async function analyzeLogFileExpensive(bucket: string, key: string) {
  const response = await s3.getObject({ Bucket: bucket, Key: key });
  const content = await streamToString(response.Body);  // 10GB download
  const logs = content.split('\n').map(line => JSON.parse(line));
  return logs.filter(log => log.level === 'ERROR' && log.status >= 500);
}
 
// Do this (processes in-place, downloads only matching rows):
async function analyzeLogFileOptimized(bucket: string, key: string) {
  const command = new SelectObjectContentCommand({
    Bucket: bucket,
    Key: key,
    ExpressionType: 'SQL',
    Expression: `
      SELECT s.timestamp, s.message, s.status 
      FROM s3object s 
      WHERE s.level = 'ERROR' AND s.status >= 500
    `,
    InputSerialization: {
      JSON: { Type: 'LINES' }
    },
    OutputSerialization: {
      JSON: {}
    }
  });
  
  const response = await s3.send(command);
  const results: LogEntry[] = [];
  
  // S3 Select streams only matching records - could be 1% of file size
  for await (const event of response.Payload!) {
    if (event.Records?.Payload) {
      const chunk = Buffer.from(event.Records.Payload).toString();
      results.push(...parseNDJSON(chunk));
    }
  }
  
  return results;
}
 
// Cost comparison for 10GB log file with 1% ERROR logs:
// - Full download: 10GB transfer + processing time
// - S3 Select:     100MB transfer + S3 Select fee (~$0.008/GB scanned)
// Net savings: ~$0.80 in transfer + significant processing time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99

interface OptimizationOpportunity {
  type: 'tier_upgrade' | 'deletion' | 'compression' | 'deduplication' | 'consolidation';
  description: string;
  affectedBuckets: string[];
  affectedVolumeGB: number;
  currentMonthlyCost: number;
  projectedMonthlyCost: number;
  annualSavings: number;
  implementationEffort: 'low' | 'medium' | 'high';
  risk: 'low' | 'medium' | 'high';
  recommendation: string;
}
 
async function scanForOptimizationOpportunities(
  buckets: string[]
): Promise<OptimizationOpportunity[]> {
  const opportunities: OptimizationOpportunity[] = [];
  
  for (const bucket of buckets) {
    const inventory = await getS3Inventory(bucket);
    const accessLogs = await getAccessLogs(bucket, 90);
    const currentCosts = await getBucketCosts(bucket);
    
    // Opportunity 1: Hot data that's not accessed
    const coldInHot = inventory.filter(obj =>
      obj.storageClass === 'STANDARD' &&
      !accessLogs.some(log => log.key === obj.key) &&
      daysSince(obj.lastModified) > 30
    );
    
    if (coldInHot.length > 0) {
      const volumeGB = sumBytes(coldInHot) / (1024**3);
      opportunities.push({
        type: 'tier_upgrade',
        description: `${coldInHot.length} objects in STANDARD not accessed in 30+ days`,
        affectedBuckets: [bucket],
        affectedVolumeGB: volumeGB,
        currentMonthlyCost: volumeGB * 0.023,
        projectedMonthlyCost: volumeGB * 0.0125,  // IA pricing
        annualSavings: volumeGB * (0.023 - 0.0125) * 12,
        implementationEffort: 'low',
        risk: 'low',
        recommendation: 'Add lifecycle rule to transition to STANDARD_IA after 30 days'
      });
    }
    
    // Opportunity 2: Data past retention period
    const expired = inventory.filter(obj => {
      const retentionTag = obj.tags?.retention;
      if (!retentionTag) return false;
      const retentionDays = parseInt(retentionTag);
      return daysSince(obj.lastModified) > retentionDays;
    });
    
    if (expired.length > 0) {
      const volumeGB = sumBytes(expired) / (1024**3);
      const currentCost = estimateMonthlyCost(expired);
      opportunities.push({
        type: 'deletion',
        description: `${expired.length} objects past retention period`,
        affectedBuckets: [bucket],
        affectedVolumeGB: volumeGB,
        currentMonthlyCost: currentCost,
        projectedMonthlyCost: 0,
        annualSavings: currentCost * 12,
        implementationEffort: 'low',
        risk: 'medium',
        recommendation: 'Review and delete expired data per retention policy'
      });
    }
    
    // Opportunity 3: Compressible data
    const compressible = inventory.filter(obj =>
      isCompressibleType(obj.contentType) &&
      !isCompressed(obj.key) &&
      obj.sizeBytes > 1024 * 1024  // 1MB+
    );
    
    if (compressible.length > 0) {
      const volumeGB = sumBytes(compressible) / (1024**3);
      const estimatedCompression = 0.3;  // Assume 70% reduction
      opportunities.push({
        type: 'compression',
        description: `${compressible.length} large compressible objects`,
        affectedBuckets: [bucket],
        affectedVolumeGB: volumeGB,
        currentMonthlyCost: volumeGB * 0.023,
        projectedMonthlyCost: volumeGB * estimatedCompression * 0.023,
        annualSavings: volumeGB * (1 - estimatedCompression) * 0.023 * 12,
        implementationEffort: 'medium',
        risk: 'low',
        recommendation: 'Implement compression at ingestion or batch compress existing'
      });
    }
  }
  
  // Sort by annual savings descending
  return opportunities.sort((a, b) => b.annualSavings - a.annualSavings);
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

Storage Decision Matrix
═══════════════════════════════════════════════════════════════════════════
 
For each data type, answer these questions:
 
1. PERFORMANCE REQUIREMENTS
   □ What is the maximum acceptable read latency? _________ ms
   □ What is the expected read throughput? _________ GB/day
   □ Is predictable latency required, or is occasional slowness acceptable?
   □ Are there burst access patterns that require rapid scaling?
 
2. COMPLIANCE REQUIREMENTS
   □ What is the minimum retention period? _________ days/years
   □ Is the data subject to regulatory requirements (HIPAA, GDPR, SOX)?
   □ Are there geographic restrictions on data location?
   □ Is immutability (WORM) required?
   □ What is the required durability? (e.g., 99.999999999%)
 
3. ACCESS PATTERNS
   □ How often is this data accessed after creation? _________
   □ Does access frequency decay over time?
   □ Is the access pattern predictable or variable?
   □ What percentage of objects are ever accessed after creation? _________%
 
4. COST SENSITIVITY
   □ What is the current cost for this data type? $_________/month
   □ What is the target cost reduction? _________% 
   □ Is the cost center/budget for this data under pressure?
   □ Are there hard budget caps that must not be exceeded?
 
DECISION TREE:
═══════════════════════════════════════════════════════════════════════════
 
                    Latency < 100ms required?
                           │
              ┌────────YES─┴─NO────────┐
              ▼                        ▼
        HOT TIER                  Access > 1x/week?
     (S3 Standard)                     │
                          ┌────────YES─┴─NO────────┐
                          ▼                        ▼
                     WARM TIER              Access > 1x/month?
                  (S3 Standard-IA)               │
                                     ┌───────YES─┴─NO────────┐
                                     ▼                       ▼
                                COOL TIER             Compliance retention?
                             (Glacier IR)                   │
                                              ┌─────────YES─┴─NO─────────┐
                                              ▼                          ▼
                                         ARCHIVE                    COLD TIER
                                    (Glacier Deep)              (Glacier Flexible)
 
OVERRIDE CONDITIONS:
- If latency-sensitive AND variable access → Use S3 Intelligent-Tiering
- If regulatory WORM required → Add Object Lock regardless of tier
- If multi-region durability required → Enable cross-region replication
- If objects < 128KB → Keep in Standard (tiering overhead not worthwhile)