System Design (HLD)Kubernetes Storage

Kubernetes Storage

LevelAdvanced

Duration90 mins

TopicKubernetes Storage

2 / 5

Storage Classes

Automating Storage at Scale

In the previous page, we explored Persistent Volumes and their manual provisioning model. While static provisioning provides control, it creates a fundamental bottleneck: every storage request requires administrator intervention. In a cluster with hundreds of applications deploying daily, this model doesn't scale.

Storage Classes transform Kubernetes storage from a manually provisioned resource into a self-service, on-demand capability. They define storage profiles that describe the type of storage available (fast SSD, cheap archival, replicated, etc.) and the provisioner that can create volumes matching those characteristics automatically.

With Storage Classes, developers request storage by specifying a class name, and Kubernetes handles the rest—provisioning volumes from the appropriate backend, applying specified parameters, and binding claims without administrator involvement.

What You Will Learn

By the end of this page, you will understand Storage Class architecture, provisioner configuration, parameter customization, binding modes for topology-aware scheduling, default storage classes, and enterprise patterns for multi-tier storage strategies in production Kubernetes clusters.

Storage Class Fundamentals

A Storage Class is a cluster-scoped Kubernetes resource that describes a class of storage available for provisioning. It serves three primary purposes:

Storage tiering: Define different quality-of-service levels (premium SSD, standard HDD, archival)
Dynamic provisioning: Enable automatic PV creation when PVCs request this class
Parameter abstraction: Hide backend-specific configuration from end users

Core Storage Class fields:

storage-class-anatomy.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
# StorageClass defines a class of storage and its provisioner
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
  labels:
    tier: premium
    environment: production
  annotations:
    # Documentation for users
    description: "High-performance SSD storage for databases"
    storageclass.kubernetes.io/is-default-class: "false"
 
# Provisioner: what creates volumes for this class
# Can be in-tree (kubernetes.io/*) or CSI (*.csi.*)
provisioner: ebs.csi.aws.com
 
# Parameters passed to the provisioner
# These are provisioner-specific
parameters:
  type: gp3           # AWS EBS volume type
  iops: "16000"       # Provisioned IOPS
  throughput: "1000"  # MB/s throughput
  encrypted: "true"   # Encryption at rest
  fsType: ext4        # Filesystem type
 
# Reclaim policy for dynamically provisioned PVs
# Default: Delete (for dynamic), can be: Retain, Delete
reclaimPolicy: Retain
 
# Allow volume expansion after creation
allowVolumeExpansion: true
 
# Mount options applied to all volumes of this class
mountOptions:
  - discard
  - noatime
 
# Volume binding mode: when to provision and bind
# Immediate: provision on PVC creation
# WaitForFirstConsumer: wait for pod scheduling
volumeBindingMode: WaitForFirstConsumer
 
# Topology restrictions for provisioned volumes
allowedTopologies:
  - matchLabelExpressions:
      - key: topology.kubernetes.io/zone
        values:
          - us-east-1a
          - us-east-1b

Key Storage Class components explained:

Storage Class Field Reference
Field	Purpose	Impact
provisioner	Specifies which provisioner creates volumes	Determines what storage backend is used
parameters	Provisioner-specific configuration	Controls volume characteristics (IOPS, encryption, etc.)
reclaimPolicy	What happens when PVC is deleted	Retain preserves data, Delete destroys it
allowVolumeExpansion	Whether volumes can be resized	Enables PVC resource request increases
mountOptions	Mount flags for volumes	Applied when volumes are mounted to pods
volumeBindingMode	When PV is provisioned	Immediate or wait for pod scheduling
allowedTopologies	Where volumes can be created	Restricts provisioning to specific zones/nodes

Provisioners Deep Dive

The provisioner is the engine that creates Persistent Volumes in response to PVCs. Understanding provisioner types and their capabilities is essential for properly configuring Storage Classes.

Provisioner categories:

Types of Provisioners

•In-tree provisioners (deprecated) — Built into Kubernetes core. Named kubernetes.io/<type>. Being migrated to CSI. Examples: kubernetes.io/aws-ebs, kubernetes.io/gce-pd, kubernetes.io/azure-disk.
•CSI provisioners — Container Storage Interface drivers deployed as pods in the cluster. Standard mechanism for all new storage integrations. Examples: ebs.csi.aws.com, pd.csi.storage.gke.io, disk.csi.azure.com.
•External provisioners — Third-party provisioners for specific storage systems. Examples: ceph.rook.io/block, nfs.csi.k8s.io, longhorn-driver.

CSI driver architecture:

CSI drivers consist of multiple components working together:

Controller component: Runs as a Deployment, handles volume provisioning, deletion, attach/detach, and snapshot operations. Communicates with the storage backend API.
Node component: Runs as a DaemonSet on each node, handles mounting/unmounting volumes on the node, filesystem formatting, and volume statistics.
CSI sidecars: Container images from the Kubernetes CSI community that handle communication between Kubernetes and the CSI driver:
- csi-provisioner: Watches PVCs and triggers CreateVolume
- csi-attacher: Watches VolumeAttachments and triggers ControllerPublish
- csi-resizer: Handles volume expansion
- csi-snapshotter: Manages volume snapshots
- csi-node-driver-registrar: Registers the CSI driver with kubelet

csi-driver-example.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
# Example: AWS EBS CSI Driver Controller Deployment (simplified)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ebs-csi-controller
  namespace: kube-system
spec:
  replicas: 2
  selector:
    matchLabels:
      app: ebs-csi-controller
  template:
    spec:
      serviceAccountName: ebs-csi-controller-sa
      containers:
        # Main CSI driver - communicates with AWS EBS API
        - name: ebs-plugin
          image: public.ecr.aws/ebs-csi-driver/aws-ebs-csi-driver:v1.25.0
          args:
            - controller
            - --endpoint=$(CSI_ENDPOINT)
            - --logging-format=json
          # AWS credentials via IAM Roles for Service Accounts
          
        # CSI Provisioner sidecar - watches PVCs
        - name: csi-provisioner
          image: registry.k8s.io/sig-storage/csi-provisioner:v3.6.0
          args:
            - --csi-address=$(ADDRESS)
            - --feature-gates=Topology=true
            - --leader-election=true
            
        # CSI Attacher sidecar - handles volume attachments
        - name: csi-attacher
          image: registry.k8s.io/sig-storage/csi-attacher:v4.4.0
          args:
            - --csi-address=$(ADDRESS)
            - --leader-election=true
            
        # CSI Resizer sidecar - handles volume expansion
        - name: csi-resizer
          image: registry.k8s.io/sig-storage/csi-resizer:v1.9.0
          args:
            - --csi-address=$(ADDRESS)
            - --leader-election=true
            
        # CSI Snapshotter sidecar - handles snapshots
        - name: csi-snapshotter
          image: registry.k8s.io/sig-storage/csi-snapshotter:v6.3.0
          args:
            - --csi-address=$(ADDRESS)
            - --leader-election=true

Checking CSI Driver Capabilities

Use kubectl get csidrivers to list installed CSI drivers and their capabilities. The CSIDriver object declares whether the driver supports volume expansion, snapshots, topology, etc. Always verify driver capabilities match Storage Class requirements.

Parameters and Customization

Storage Class parameters are passed directly to the provisioner and control the characteristics of provisioned volumes. These parameters are highly provisioner-specific—understanding them requires knowledge of the storage backend.

Common parameter patterns across cloud providers:

aws-ebs-storage-classes.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
# High-performance SSD (gp3)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: aws-fast
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "16000"          # Up to 16,000 IOPS
  throughput: "1000"     # Up to 1,000 MB/s
  encrypted: "true"
  kmsKeyId: alias/ebs-key  # Customer-managed encryption key
  fsType: ext4
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
 
---
# Provisioned IOPS SSD (io2) for critical databases
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: aws-ultra
provisioner: ebs.csi.aws.com
parameters:
  type: io2
  iops: "64000"          # Up to 64,000 IOPS
  encrypted: "true"
  blockExpress: "true"   # io2 Block Express for > 64,000 IOPS
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
 
---
# Cold storage (sc1) for archives
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: aws-archive
provisioner: ebs.csi.aws.com
parameters:
  type: sc1
  encrypted: "true"
reclaimPolicy: Delete
volumeBindingMode: Immediate

Parameter Validation

Kubernetes does not validate Storage Class parameters at creation time. Invalid parameters only cause errors when a PVC triggers dynamic provisioning. Always test new Storage Classes with a sample PVC before production use.

Volume Binding Modes

Volume binding mode controls when a PersistentVolume is provisioned and bound to a PVC. This seemingly simple configuration has profound implications for pod scheduling and storage topology.

The two binding modes:

Immediate Binding

•PV provisioned immediately on PVC creation
•Binding happens before pod scheduling
•PV zone chosen by provisioner (may be suboptimal)
•Simple, fast for zone-agnostic storage
•Can cause scheduling failures if pod needs different zone

WaitForFirstConsumer

•PV provisioning delayed until pod uses PVC
•Scheduler considers storage topology
•PV created in same zone as scheduled pod
•Optimal for zone-local storage (EBS, Azure Disk)
•Adds latency to first pod startup

Why WaitForFirstConsumer matters:

Consider a cluster spanning three availability zones. With Immediate binding:

PVC created → EBS volume provisioned in zone us-east-1a
Pod scheduled to us-east-1c (best node for resources)
Pod cannot start—EBS volumes are zone-local, cannot attach to nodes in different zones
Pod remains Pending indefinitely

With WaitForFirstConsumer:

PVC created → binding delayed (no volume yet)
Pod scheduled to us-east-1c (best fit for all constraints including storage topology)
PV provisioned in us-east-1c to match scheduled node
PV bound, pod starts successfully

binding-mode-example.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
# WaitForFirstConsumer is essential for zone-local storage
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-topology-aware
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
volumeBindingMode: WaitForFirstConsumer  # Critical for EBS
 
---
# Immediate is acceptable for network storage (available in all zones)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: efs-shared
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
volumeBindingMode: Immediate  # EFS is available cluster-wide
 
---
# PVC using topology-aware class
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-data
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: ebs-topology-aware
  resources:
    requests:
      storage: 100Gi
# Note: PV won't be created until a pod references this PVC

Best Practice

Use WaitForFirstConsumer for all zone-local storage (EBS, GCE PD, Azure Disk, Local PVs). Use Immediate only for storage accessible from any zone (NFS, CephFS, cloud file services). When in doubt, WaitForFirstConsumer is safer.

Default Storage Class

Kubernetes supports designating one Storage Class as the cluster default. When a PVC doesn't specify a storageClassName, it uses the default class for dynamic provisioning.

Setting the default:

The default class is indicated by an annotation:

default-storage-class.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
  annotations:
    # This annotation marks the class as default
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete  # Common for default to auto-cleanup
 
---
# PVC without storageClassName uses the default
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-data
spec:
  accessModes: [ReadWriteOnce]
  # storageClassName: omitted - will use default class
  resources:
    requests:
      storage: 10Gi
 
---
# Explicitly requesting no class (static binding only)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: static-only
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: ""  # Empty string = no dynamic provisioning
  resources:
    requests:
      storage: 10Gi

Important default class behaviors:

Multiple defaults: If multiple classes have the default annotation, PVCs without a class will fail with an error. Ensure exactly one default.
No default: If no class is marked default and a PVC omits storageClassName, behavior depends on Kubernetes version and DefaultStorageClass admission controller configuration.
Explicit empty string: Setting storageClassName: "" explicitly opts out of dynamic provisioning. The PVC will only bind to pre-existing PVs without a class.
Changing the default: To change the default class, remove the annotation from the old default before adding it to the new one:

# Remove default from old class
kubectl patch storageclass old-default -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

# Set new default
kubectl patch storageclass new-default -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Default Class Gotcha

Existing PVCs are not affected by changing the default class. They remain bound to their original PVs. The default class only affects new PVCs that don't specify a class.

Volume Expansion

Storage needs grow over time. Kubernetes supports online volume expansion—increasing the size of a Persistent Volume while workloads continue running. This capability must be explicitly enabled in the Storage Class.

Enabling volume expansion:

volume-expansion.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
# Storage Class with expansion enabled
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: expandable-ssd
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
allowVolumeExpansion: true  # Enables PVC size increases
volumeBindingMode: WaitForFirstConsumer
 
---
# Original PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-store
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: expandable-ssd
  resources:
    requests:
      storage: 100Gi
 
---
# To expand, increase the storage request
# Use kubectl patch or edit the PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-store
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: expandable-ssd
  resources:
    requests:
      storage: 200Gi  # Increased from 100Gi

The expansion process:

Volume expansion happens in two phases:

Controller expansion: The storage backend extends the volume. For cloud disks, this involves an API call to resize the underlying disk. This happens automatically when you increase the PVC request.
Filesystem expansion: The filesystem on the volume must be grown to use the additional space. This requires:
- For online expansion (most CSI drivers): Filesystem extended while volume is mounted
- For offline expansion (some legacy drivers): Pod must be deleted/restarted for resize

Monitoring expansion status:

Check PVC conditions to monitor expansion progress:

expansion-status.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# Check expansion status
kubectl get pvc database-store -o yaml
 
# Look for conditions:
# status:
#   conditions:
#   - type: Resizing
#     status: "True"
#   - type: FileSystemResizePending
#     status: "True"
 
# Once complete:
# status:
#   capacity:
#     storage: 200Gi  # Matches requested
 
# Wait for filesystem expansion
kubectl describe pvc database-store
# Events:
#   Normal  Resizing   2m   external-resizer   External resizer is resizing volume
#   Normal  FileSystemResizeSuccessful  1m  kubelet  MountVolume.NodeExpandVolume succeeded

Expansion Limitations

Volume expansion is one-directional—you cannot shrink a PVC. Always consider future growth; shrinking requires backing up data, creating a new smaller volume, and restoring. Some CSI drivers have minimum resize increments or cooldowns between operations.

Allowed Topologies

Storage Classes can restrict where volumes are provisioned using allowedTopologies. This is essential for multi-zone clusters where you need to control storage placement for performance, compliance, or cost reasons.

Topology constraint use cases:

When to Use Allowed Topologies

•Data residency: Regulatory requirements mandating storage in specific geographic locations
•Cost optimization: Directing storage to zones with available quotas or cheaper pricing
•Hardware isolation: Restricting to zones with specific storage hardware (NVMe nodes, etc.)
•Failure domain control: Ensuring replicas are spread across zones for HA
•Legacy compatibility: Restricting to zones where older storage types are available

allowed-topologies.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
# Restrict provisioning to specific zones
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: zone-restricted-ssd
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
  - matchLabelExpressions:
      # Only provision in us-east-1a and us-east-1b
      # Excludes us-east-1c (perhaps for cost or capacity reasons)
      - key: topology.kubernetes.io/zone
        values:
          - us-east-1a
          - us-east-1b
 
---
# Data residency: EU zones only
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: eu-compliant
provisioner: pd.csi.storage.gke.io
parameters:
  type: pd-ssd
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
  - matchLabelExpressions:
      - key: topology.kubernetes.io/zone
        values:
          - europe-west1-b
          - europe-west1-c
          - europe-west1-d
 
---
# Multiple topology keys (zone and node type)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nvme-zones
provisioner: local.csi.kubernetes.io
parameters:
  nodetype: nvme
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
  - matchLabelExpressions:
      - key: topology.kubernetes.io/zone
        values: [us-west-2a, us-west-2b]
      - key: node.kubernetes.io/instance-type
        values: [i3.large, i3.xlarge, i3.2xlarge]

Topology with WaitForFirstConsumer

When using allowedTopologies with WaitForFirstConsumer, the scheduler restricts pod placement to nodes in allowed zones before triggering provisioning. This prevents scheduling pods to zones where storage cannot be created.

Multi-Tier Storage Strategy

Enterprise Kubernetes environments typically require multiple Storage Classes to address varying workload needs. A well-designed storage tier strategy balances performance, cost, and durability across different use cases.

Common storage tier model:

Enterprise Storage Tier Strategy
Tier	Class Name	Use Case	Performance	Cost	Durability
Premium	fast-ssd	Production databases, critical workloads	High IOPS, low latency	$$$	High (replicated)
Standard	standard	General-purpose workloads	Balanced	$$	High (replicated)
Economy	cheap-hdd	Dev/test, non-critical	Lower performance	$	Standard
Archive	cold-storage	Backups, logs, rarely-accessed	High latency	$	High
Ephemeral	ephemeral-local	Temp data, caches	Very high (local)	$	None
Shared	nfs-shared	Shared across pods	Moderate	$$	High

multi-tier-classes.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
# Premium tier: Production databases
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
  labels:
    tier: premium
    cost: high
provisioner: ebs.csi.aws.com
parameters:
  type: io2
  iops: "64000"
  encrypted: "true"
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
 
---
# Standard tier: Default for most workloads
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  labels:
    tier: standard
    cost: medium
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
  encrypted: "true"
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
 
---
# Economy tier: Development environments
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cheap-hdd
  labels:
    tier: economy
    cost: low
provisioner: ebs.csi.aws.com
parameters:
  type: st1  # Throughput-optimized HDD
  encrypted: "true"
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
 
---
# Shared tier: RWX volumes via EFS
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-shared
  labels:
    tier: shared
    access: rwx
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
  fileSystemId: fs-abc123
  directoryPerms: "700"
reclaimPolicy: Delete
volumeBindingMode: Immediate  # EFS is zone-agnostic

Storage Class Quotas

Use ResourceQuotas to limit storage consumption per tier: kubectl create quota premium-limit --hard=fast-ssd.storageclass.storage.k8s.io/requests.storage=500Gi. This prevents runaway costs from premium storage overuse.

Summary: Storage Classes Mastery

Storage Classes transform Kubernetes storage from a manually provisioned resource into a self-service capability that scales with your organization. They are the control plane for storage policy, enabling automated provisioning, tiering, and lifecycle management.

Key Takeaways

•Storage Classes enable dynamic provisioning — Volumes created automatically when PVCs reference a class with a capable provisioner.
•Provisioners do the work — CSI drivers communicate with storage backends. Understanding your provisioner's capabilities is essential.
•Parameters control volume characteristics — IOPS, encryption, disk type, and other settings are passed to the provisioner. They're backend-specific.
•WaitForFirstConsumer prevents scheduling issues — For zone-local storage, delaying binding ensures volumes are created in the right zone.
•Default class captures unspecified PVCs — Exactly one class should be marked default for a predictable user experience.
•Volume expansion requires planning — Enable it in Storage Classes for future flexibility. Remember: you can grow but not shrink.
•Allowed topologies enforce placement — Use them for compliance, cost control, or hardware targeting.
•Multi-tier strategy optimizes cost/performance — Different workloads need different storage. Design tiers that match organizational needs.

What's next:

With Persistent Volumes and Storage Classes covered, we'll explore StatefulSet storage—how Kubernetes manages volumes for stateful applications that need stable identities and ordered operations. StatefulSets combine PVCs with stable pod identities to enable reliable database and stateful service deployments.

Page Complete

You now understand Storage Classes comprehensively—from provisioners and parameters through binding modes, expansion, topology constraints, and multi-tier strategies. This knowledge enables you to design and operate storage infrastructure that serves diverse workload needs at scale.

2 / 5

Loading learning content...

System Design (HLD)Kubernetes Storage

Kubernetes Storage

LevelAdvanced

Duration90 mins

TopicKubernetes Storage

2 / 5

Storage Classes

Automating Storage at Scale

What You Will Learn

Storage Class Fundamentals

A Storage Class is a cluster-scoped Kubernetes resource that describes a class of storage available for provisioning. It serves three primary purposes:

Storage tiering: Define different quality-of-service levels (premium SSD, standard HDD, archival)
Dynamic provisioning: Enable automatic PV creation when PVCs request this class
Parameter abstraction: Hide backend-specific configuration from end users

Core Storage Class fields:

storage-class-anatomy.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
# StorageClass defines a class of storage and its provisioner
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
  labels:
    tier: premium
    environment: production
  annotations:
    # Documentation for users
    description: "High-performance SSD storage for databases"
    storageclass.kubernetes.io/is-default-class: "false"
 
# Provisioner: what creates volumes for this class
# Can be in-tree (kubernetes.io/*) or CSI (*.csi.*)
provisioner: ebs.csi.aws.com
 
# Parameters passed to the provisioner
# These are provisioner-specific
parameters:
  type: gp3           # AWS EBS volume type
  iops: "16000"       # Provisioned IOPS
  throughput: "1000"  # MB/s throughput
  encrypted: "true"   # Encryption at rest
  fsType: ext4        # Filesystem type
 
# Reclaim policy for dynamically provisioned PVs
# Default: Delete (for dynamic), can be: Retain, Delete
reclaimPolicy: Retain
 
# Allow volume expansion after creation
allowVolumeExpansion: true
 
# Mount options applied to all volumes of this class
mountOptions:
  - discard
  - noatime
 
# Volume binding mode: when to provision and bind
# Immediate: provision on PVC creation
# WaitForFirstConsumer: wait for pod scheduling
volumeBindingMode: WaitForFirstConsumer
 
# Topology restrictions for provisioned volumes
allowedTopologies:
  - matchLabelExpressions:
      - key: topology.kubernetes.io/zone
        values:
          - us-east-1a
          - us-east-1b

Key Storage Class components explained:

Storage Class Field Reference
Field	Purpose	Impact
provisioner	Specifies which provisioner creates volumes	Determines what storage backend is used
parameters	Provisioner-specific configuration	Controls volume characteristics (IOPS, encryption, etc.)
reclaimPolicy	What happens when PVC is deleted	Retain preserves data, Delete destroys it
allowVolumeExpansion	Whether volumes can be resized	Enables PVC resource request increases
mountOptions	Mount flags for volumes	Applied when volumes are mounted to pods
volumeBindingMode	When PV is provisioned	Immediate or wait for pod scheduling
allowedTopologies	Where volumes can be created	Restricts provisioning to specific zones/nodes

Provisioners Deep Dive

The provisioner is the engine that creates Persistent Volumes in response to PVCs. Understanding provisioner types and their capabilities is essential for properly configuring Storage Classes.

Provisioner categories:

Types of Provisioners

•In-tree provisioners (deprecated) — Built into Kubernetes core. Named kubernetes.io/<type>. Being migrated to CSI. Examples: kubernetes.io/aws-ebs, kubernetes.io/gce-pd, kubernetes.io/azure-disk.
•CSI provisioners — Container Storage Interface drivers deployed as pods in the cluster. Standard mechanism for all new storage integrations. Examples: ebs.csi.aws.com, pd.csi.storage.gke.io, disk.csi.azure.com.
•External provisioners — Third-party provisioners for specific storage systems. Examples: ceph.rook.io/block, nfs.csi.k8s.io, longhorn-driver.

CSI driver architecture:

CSI drivers consist of multiple components working together:

Controller component: Runs as a Deployment, handles volume provisioning, deletion, attach/detach, and snapshot operations. Communicates with the storage backend API.
Node component: Runs as a DaemonSet on each node, handles mounting/unmounting volumes on the node, filesystem formatting, and volume statistics.
CSI sidecars: Container images from the Kubernetes CSI community that handle communication between Kubernetes and the CSI driver:
- csi-provisioner: Watches PVCs and triggers CreateVolume
- csi-attacher: Watches VolumeAttachments and triggers ControllerPublish
- csi-resizer: Handles volume expansion
- csi-snapshotter: Manages volume snapshots
- csi-node-driver-registrar: Registers the CSI driver with kubelet

csi-driver-example.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
# Example: AWS EBS CSI Driver Controller Deployment (simplified)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ebs-csi-controller
  namespace: kube-system
spec:
  replicas: 2
  selector:
    matchLabels:
      app: ebs-csi-controller
  template:
    spec:
      serviceAccountName: ebs-csi-controller-sa
      containers:
        # Main CSI driver - communicates with AWS EBS API
        - name: ebs-plugin
          image: public.ecr.aws/ebs-csi-driver/aws-ebs-csi-driver:v1.25.0
          args:
            - controller
            - --endpoint=$(CSI_ENDPOINT)
            - --logging-format=json
          # AWS credentials via IAM Roles for Service Accounts
          
        # CSI Provisioner sidecar - watches PVCs
        - name: csi-provisioner
          image: registry.k8s.io/sig-storage/csi-provisioner:v3.6.0
          args:
            - --csi-address=$(ADDRESS)
            - --feature-gates=Topology=true
            - --leader-election=true
            
        # CSI Attacher sidecar - handles volume attachments
        - name: csi-attacher
          image: registry.k8s.io/sig-storage/csi-attacher:v4.4.0
          args:
            - --csi-address=$(ADDRESS)
            - --leader-election=true
            
        # CSI Resizer sidecar - handles volume expansion
        - name: csi-resizer
          image: registry.k8s.io/sig-storage/csi-resizer:v1.9.0
          args:
            - --csi-address=$(ADDRESS)
            - --leader-election=true
            
        # CSI Snapshotter sidecar - handles snapshots
        - name: csi-snapshotter
          image: registry.k8s.io/sig-storage/csi-snapshotter:v6.3.0
          args:
            - --csi-address=$(ADDRESS)
            - --leader-election=true

Checking CSI Driver Capabilities

Parameters and Customization

Common parameter patterns across cloud providers:

aws-ebs-storage-classes.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
# High-performance SSD (gp3)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: aws-fast
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "16000"          # Up to 16,000 IOPS
  throughput: "1000"     # Up to 1,000 MB/s
  encrypted: "true"
  kmsKeyId: alias/ebs-key  # Customer-managed encryption key
  fsType: ext4
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
 
---
# Provisioned IOPS SSD (io2) for critical databases
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: aws-ultra
provisioner: ebs.csi.aws.com
parameters:
  type: io2
  iops: "64000"          # Up to 64,000 IOPS
  encrypted: "true"
  blockExpress: "true"   # io2 Block Express for > 64,000 IOPS
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
 
---
# Cold storage (sc1) for archives
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: aws-archive
provisioner: ebs.csi.aws.com
parameters:
  type: sc1
  encrypted: "true"
reclaimPolicy: Delete
volumeBindingMode: Immediate

Parameter Validation

Volume Binding Modes

Volume binding mode controls when a PersistentVolume is provisioned and bound to a PVC. This seemingly simple configuration has profound implications for pod scheduling and storage topology.

The two binding modes:

Immediate Binding

•PV provisioned immediately on PVC creation
•Binding happens before pod scheduling
•PV zone chosen by provisioner (may be suboptimal)
•Simple, fast for zone-agnostic storage
•Can cause scheduling failures if pod needs different zone

WaitForFirstConsumer

•PV provisioning delayed until pod uses PVC
•Scheduler considers storage topology
•PV created in same zone as scheduled pod
•Optimal for zone-local storage (EBS, Azure Disk)
•Adds latency to first pod startup

Why WaitForFirstConsumer matters:

Consider a cluster spanning three availability zones. With Immediate binding:

PVC created → EBS volume provisioned in zone us-east-1a
Pod scheduled to us-east-1c (best node for resources)
Pod cannot start—EBS volumes are zone-local, cannot attach to nodes in different zones
Pod remains Pending indefinitely

With WaitForFirstConsumer:

PVC created → binding delayed (no volume yet)
Pod scheduled to us-east-1c (best fit for all constraints including storage topology)
PV provisioned in us-east-1c to match scheduled node
PV bound, pod starts successfully

binding-mode-example.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
# WaitForFirstConsumer is essential for zone-local storage
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-topology-aware
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
volumeBindingMode: WaitForFirstConsumer  # Critical for EBS
 
---
# Immediate is acceptable for network storage (available in all zones)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: efs-shared
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
volumeBindingMode: Immediate  # EFS is available cluster-wide
 
---
# PVC using topology-aware class
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-data
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: ebs-topology-aware
  resources:
    requests:
      storage: 100Gi
# Note: PV won't be created until a pod references this PVC

Best Practice

Default Storage Class

Kubernetes supports designating one Storage Class as the cluster default. When a PVC doesn't specify a storageClassName, it uses the default class for dynamic provisioning.

Setting the default:

The default class is indicated by an annotation:

default-storage-class.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
  annotations:
    # This annotation marks the class as default
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete  # Common for default to auto-cleanup
 
---
# PVC without storageClassName uses the default
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-data
spec:
  accessModes: [ReadWriteOnce]
  # storageClassName: omitted - will use default class
  resources:
    requests:
      storage: 10Gi
 
---
# Explicitly requesting no class (static binding only)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: static-only
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: ""  # Empty string = no dynamic provisioning
  resources:
    requests:
      storage: 10Gi

Important default class behaviors:

Multiple defaults: If multiple classes have the default annotation, PVCs without a class will fail with an error. Ensure exactly one default.
No default: If no class is marked default and a PVC omits storageClassName, behavior depends on Kubernetes version and DefaultStorageClass admission controller configuration.
Explicit empty string: Setting storageClassName: "" explicitly opts out of dynamic provisioning. The PVC will only bind to pre-existing PVs without a class.
Changing the default: To change the default class, remove the annotation from the old default before adding it to the new one:

# Remove default from old class
kubectl patch storageclass old-default -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

# Set new default
kubectl patch storageclass new-default -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Default Class Gotcha

Existing PVCs are not affected by changing the default class. They remain bound to their original PVs. The default class only affects new PVCs that don't specify a class.

Volume Expansion

Enabling volume expansion:

volume-expansion.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
# Storage Class with expansion enabled
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: expandable-ssd
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
allowVolumeExpansion: true  # Enables PVC size increases
volumeBindingMode: WaitForFirstConsumer
 
---
# Original PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-store
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: expandable-ssd
  resources:
    requests:
      storage: 100Gi
 
---
# To expand, increase the storage request
# Use kubectl patch or edit the PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-store
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: expandable-ssd
  resources:
    requests:
      storage: 200Gi  # Increased from 100Gi

The expansion process:

Volume expansion happens in two phases:

Controller expansion: The storage backend extends the volume. For cloud disks, this involves an API call to resize the underlying disk. This happens automatically when you increase the PVC request.
Filesystem expansion: The filesystem on the volume must be grown to use the additional space. This requires:
- For online expansion (most CSI drivers): Filesystem extended while volume is mounted
- For offline expansion (some legacy drivers): Pod must be deleted/restarted for resize

Monitoring expansion status:

Check PVC conditions to monitor expansion progress:

expansion-status.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# Check expansion status
kubectl get pvc database-store -o yaml
 
# Look for conditions:
# status:
#   conditions:
#   - type: Resizing
#     status: "True"
#   - type: FileSystemResizePending
#     status: "True"
 
# Once complete:
# status:
#   capacity:
#     storage: 200Gi  # Matches requested
 
# Wait for filesystem expansion
kubectl describe pvc database-store
# Events:
#   Normal  Resizing   2m   external-resizer   External resizer is resizing volume
#   Normal  FileSystemResizeSuccessful  1m  kubelet  MountVolume.NodeExpandVolume succeeded

Expansion Limitations

Allowed Topologies

Topology constraint use cases:

When to Use Allowed Topologies

•Data residency: Regulatory requirements mandating storage in specific geographic locations
•Cost optimization: Directing storage to zones with available quotas or cheaper pricing
•Hardware isolation: Restricting to zones with specific storage hardware (NVMe nodes, etc.)
•Failure domain control: Ensuring replicas are spread across zones for HA
•Legacy compatibility: Restricting to zones where older storage types are available

allowed-topologies.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
# Restrict provisioning to specific zones
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: zone-restricted-ssd
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
  - matchLabelExpressions:
      # Only provision in us-east-1a and us-east-1b
      # Excludes us-east-1c (perhaps for cost or capacity reasons)
      - key: topology.kubernetes.io/zone
        values:
          - us-east-1a
          - us-east-1b
 
---
# Data residency: EU zones only
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: eu-compliant
provisioner: pd.csi.storage.gke.io
parameters:
  type: pd-ssd
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
  - matchLabelExpressions:
      - key: topology.kubernetes.io/zone
        values:
          - europe-west1-b
          - europe-west1-c
          - europe-west1-d
 
---
# Multiple topology keys (zone and node type)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nvme-zones
provisioner: local.csi.kubernetes.io
parameters:
  nodetype: nvme
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
  - matchLabelExpressions:
      - key: topology.kubernetes.io/zone
        values: [us-west-2a, us-west-2b]
      - key: node.kubernetes.io/instance-type
        values: [i3.large, i3.xlarge, i3.2xlarge]

Topology with WaitForFirstConsumer

Multi-Tier Storage Strategy

Common storage tier model:

Enterprise Storage Tier Strategy
Tier	Class Name	Use Case	Performance	Cost	Durability
Premium	fast-ssd	Production databases, critical workloads	High IOPS, low latency	$$$	High (replicated)
Standard	standard	General-purpose workloads	Balanced	$$	High (replicated)
Economy	cheap-hdd	Dev/test, non-critical	Lower performance	$	Standard
Archive	cold-storage	Backups, logs, rarely-accessed	High latency	$	High
Ephemeral	ephemeral-local	Temp data, caches	Very high (local)	$	None
Shared	nfs-shared	Shared across pods	Moderate	$$	High

multi-tier-classes.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
# Premium tier: Production databases
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
  labels:
    tier: premium
    cost: high
provisioner: ebs.csi.aws.com
parameters:
  type: io2
  iops: "64000"
  encrypted: "true"
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
 
---
# Standard tier: Default for most workloads
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  labels:
    tier: standard
    cost: medium
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
  encrypted: "true"
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
 
---
# Economy tier: Development environments
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cheap-hdd
  labels:
    tier: economy
    cost: low
provisioner: ebs.csi.aws.com
parameters:
  type: st1  # Throughput-optimized HDD
  encrypted: "true"
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
 
---
# Shared tier: RWX volumes via EFS
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-shared
  labels:
    tier: shared
    access: rwx
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
  fileSystemId: fs-abc123
  directoryPerms: "700"
reclaimPolicy: Delete
volumeBindingMode: Immediate  # EFS is zone-agnostic

Storage Class Quotas

Summary: Storage Classes Mastery

Key Takeaways

•Storage Classes enable dynamic provisioning — Volumes created automatically when PVCs reference a class with a capable provisioner.
•Provisioners do the work — CSI drivers communicate with storage backends. Understanding your provisioner's capabilities is essential.
•Parameters control volume characteristics — IOPS, encryption, disk type, and other settings are passed to the provisioner. They're backend-specific.
•WaitForFirstConsumer prevents scheduling issues — For zone-local storage, delaying binding ensures volumes are created in the right zone.
•Default class captures unspecified PVCs — Exactly one class should be marked default for a predictable user experience.
•Volume expansion requires planning — Enable it in Storage Classes for future flexibility. Remember: you can grow but not shrink.
•Allowed topologies enforce placement — Use them for compliance, cost control, or hardware targeting.
•Multi-tier strategy optimizes cost/performance — Different workloads need different storage. Design tiers that match organizational needs.

What's next:

Page Complete

2 / 5