Storage Virtualization
Volume Groups
Read aloud is unavailable in this browser.
The Storage Pool Abstraction
In the previous page, we explored LVM's three-layer architecture and established that Volume Groups (VGs) form the critical middle layer—the bridge between physical storage devices and the logical volumes that applications actually use. Now we dive deep into volume groups: their internal structure, design considerations, operational mechanics, and how they enable the storage flexibility that makes LVM so powerful.
A volume group is fundamentally a storage pool. It aggregates the capacity of one or more physical volumes into a unified resource from which logical volumes can draw space on demand. This pooling concept transforms rigid, device-bound storage into fluid, allocatable capacity.
Consider the profound shift this enables: without volume groups, if you have three 100GB disks, you have three separate 100GB storage spaces. With a volume group encompassing all three, you have a single 300GB pool from which you can carve logical volumes of any size up to 300GB—or create many smaller volumes that collectively use the pooled capacity.
By the end of this page, you will understand volume group creation strategies, extent size selection trade-offs, VG metadata structure and redundancy, extending and reducing volume groups, VG naming conventions, clustering considerations for shared storage, and production best practices for volume group design.
Volume Group Fundamentals
A Volume Group serves as the administrative container for LVM storage. It defines the extent size used for allocations, maintains metadata describing all physical and logical volumes, and manages the free space pool from which new logical volumes are created or existing ones are extended.
The Extent Size Decision:
When you create a volume group, you specify an extent size—the fundamental unit of allocation for all storage within that VG. This decision, seemingly minor, has architectural implications that persist for the lifetime of the volume group.
The extent size determines:
- Allocation Granularity: You can only allocate storage in whole-extent increments
- Maximum Volume Size: Limited by (max LEs per LV) × (extent size)
- Metadata Size: More extents mean more mapping entries in metadata
- Alignment with Hardware: Optimal performance when aligned with RAID stripes or SSD erase blocks
| Extent Size | Max LV Size (64K LEs) | Waste Per LV (max) | Metadata Entries per TB | Best For |
|---|---|---|---|---|
| 1 MB | 64 GB | ~1 MB | 1,048,576 | Embedded systems, small deployments |
| 4 MB (default) | 256 TB | ~4 MB | 262,144 | General purpose, most servers |
| 8 MB | 512 TB | ~8 MB | 131,072 | Large storage systems |
| 16 MB | 1 PB | ~16 MB | 65,536 | Enterprise SAN, petabyte scale |
| 64 MB | 4 PB | ~64 MB | 16,384 | Massive data lakes, HPC |
Why Can't Extent Size Be Changed?
Once a volume group is created, its extent size is immutable. This is because:
- Physical extents are pre-allocated: The entire PV is divided into PEs at initialization
- Logical extent mappings assume fixed size: LE-to-PE mappings would become invalid
- Segment calculations depend on extent size: All segment start/length values are in extents
- Device-mapper tables are derived from extent mappings: The kernel-level mapping would break
If you need a different extent size, you must create a new volume group and migrate data to it—a process that, while possible with pvmove, requires careful planning and potentially significant time.
For modern systems, the 4MB default is appropriate for most use cases. Only increase extent size if you're building storage systems at petabyte scale or have specific alignment requirements. Decreasing below 4MB is rarely beneficial and increases metadata overhead substantially.
Volume Group Metadata Structure
Volume group metadata is the authoritative record of the entire LVM configuration. It contains definitions for all physical volumes in the VG, all logical volumes, their segment mappings, and historical information. Understanding metadata structure is essential for troubleshooting and disaster recovery.
Metadata Storage Locations:
By default, LVM stores a complete copy of VG metadata on every physical volume in the volume group. This redundancy ensures that:
- Loss of any single PV doesn't result in metadata loss
- Any surviving PV can be used to reconstruct the VG configuration
- The VG can operate even with degraded PV availability (for unmapped LVs)
The metadata area (MDA) is typically located at the beginning of the PV, after the label sector. The default MDA size is approximately 1MB, but this can be adjusted during pvcreate.
12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061
#!/bin/bash# Examining Volume Group Metadata # Display raw metadata from a VG (extremely useful for debugging)vgcfgbackup -f /tmp/vg_storage_backup.txt vg_storage # View the metadata (it's in a human-readable ASCII format)cat /tmp/vg_storage_backup.txt # Example output structure:# vg_storage {# id = "aB1CdE-2fGh-3iJk-4LmN-5oPq-6RsT-7UvWx"# seqno = 42# format = "lvm2"# status = ["RESIZEABLE", "READ", "WRITE"]# flags = []# extent_size = 8192 # In sectors (4MB = 8192 * 512)# max_lv = 0# max_pv = 0# metadata_copies = 0## physical_volumes { ... }# logical_volumes { ... }# } # Check metadata area size on a PVpvs -o+pv_mda_size,pv_mda_free /dev/sdb# Output:# PV VG PSize PFree PMdaSize PMdaFree# /dev/sdb vg_storage 100.00g 20.00g 1020.00k 508.00k # Display where metadata is storedpvck --dump metadata /dev/sdb# Shows detailed metadata area information # === METADATA AREA CONFIGURATION === # Create PV with larger metadata area (2 copies, 2MB each)pvcreate --metadatasize 2m --pvmetadatacopies 2 /dev/sdd # Create PV with NO metadata area (uses another PV's metadata)# Useful for data-only PVs in large configurationspvcreate --metadatacopies 0 /dev/sdd # === METADATA OPERATIONS === # Check metadata consistencyvgck vg_storage # Display metadata sequence number (increments with each change)vgs -o vg_name,vg_seqno vg_storage # Restore metadata from backup (disaster recovery)vgcfgrestore -f /tmp/vg_storage_backup.txt vg_storage # Force metadata refresh from diskpvscan --cache # View metadata history (if archived)ls /etc/lvm/archive/# Files named: vg_storage_00042-xxxxxxxx.vgMetadata Sequence Numbers:
Every time VG metadata is modified (creating an LV, extending a volume, adding a PV), the sequence number increments. This versioning enables:
- Consistency Checking: LVM can detect if PVs have divergent metadata versions
- Recovery: The most recent sequence number identifies the authoritative metadata
- Archiving:
/etc/lvm/archive/stores sequential metadata backups - Debugging: You can trace the history of configuration changes
Metadata Formats:
LVM2 uses a text-based metadata format that is:
- Human-readable: You can view and understand the configuration
- Portable: The same format works across all Linux versions
- Versionless: Forward-compatible by ignoring unknown fields
- Compact: Uses segment representation to minimize size
Always run 'vgcfgbackup' before major storage operations and store the backup off-system. LVM automatically archives metadata to /etc/lvm/archive/, but this won't help if the root filesystem is on the affected VG. Having an external metadata backup can save hours of recovery effort.
Creating Volume Groups
Volume group creation involves several strategic decisions that affect long-term manageability. While the basic vgcreate command is straightforward, understanding all options enables optimal configuration for specific use cases.
Basic VG Creation:
The simplest form creates a VG with default settings:
1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374
#!/bin/bash# Volume Group Creation Examples # === BASIC CREATION === # Create VG with single PV (defaults to 4MB extent size)vgcreate vg_data /dev/sdb # Create VG with multiple PVsvgcreate vg_data /dev/sdb /dev/sdc /dev/sdd # === EXTENT SIZE CONFIGURATION === # Create VG with 8MB extents (larger systems)vgcreate -s 8M vg_large /dev/sdb /dev/sdc # Create VG with 16MB extents (enterprise scale)vgcreate -s 16M vg_enterprise /dev/sd[b-h] # Create VG with 1MB extents (fine-grained control, small systems)vgcreate -s 1M vg_small /dev/sdb # === ALLOCATION POLICY === # Create VG with contiguous allocation policy (for performance)vgcreate --alloc contiguous vg_perf /dev/nvme0n1p2 # Allocation policies:# - inherit: Use default policy# - contiguous: Require physically contiguous extents# - cling: Prefer same PV as existing allocations# - normal: Default, balance between policies# - anywhere: Allow any available extent # === METADATA CONFIGURATION === # Create VG with maximum metadata copies (highest redundancy)vgcreate --vgmetadatacopies all vg_critical /dev/sd[b-e] # Create VG with single metadata copy (minimum redundancy)vgcreate --vgmetadatacopies 1 vg_temp /dev/sdb # Set specific number of metadata copiesvgcreate --vgmetadatacopies 3 vg_balanced /dev/sd[b-f] # === CLUSTERING (for shared storage) === # Create clustered VG (requires clvmd or lvmlockd)vgcreate --shared vg_cluster /dev/mapper/shared_lun # Create VG with system ID for exclusive accessvgcreate --systemid "server1" vg_server1 /dev/sdb # === ADVANCED OPTIONS === # Auto-activate VG at bootvgcreate --setautoactivation y vg_boot /dev/sdb # Set maximum number of LVs (0 = unlimited, default)vgcreate --maxlogicalvolumes 100 vg_limited /dev/sdb # Set maximum number of PVs (0 = unlimited, default)vgcreate --maxphysicalvolumes 10 vg_bounded /dev/sdb # === VERIFY CREATION === # Display detailed information about new VGvgdisplay vg_data # Show VG with all available columnsvgs -o+vg_extent_size,vg_extent_count,vg_free_count vg_data # Verify metadata consistencyvgck vg_dataVG Naming Conventions:
Choosing clear, consistent VG names improves system administration. Common conventions include:
| Pattern | Example | Use Case |
|---|---|---|
| Purpose-based | vg_database, vg_logs | Dedicated VGs for specific workloads |
| Host-based | vg_<hostname> | When VGs may move between systems |
| Storage-tier | vg_ssd, vg_hdd, vg_nvme | Differentiating performance tiers |
| Datacenter | vg_dc1_rack3 | Large-scale deployments |
| Combined | vg_prod_db_ssd | Complex environments |
Avoid names that might conflict with device nodes or contain special characters. Use lowercase with underscores for maximum compatibility.
Before creating a volume group, ensure all target devices are initialized as physical volumes with 'pvcreate'. If a device contains existing partitions or file systems, pvcreate will warn and may require force (-f) flag. Always verify you're operating on the intended devices before proceeding.
Extending and Reducing Volume Groups
One of LVM's most powerful capabilities is dynamic volume group modification. You can add new physical volumes to increase capacity or remove physical volumes during hardware decommissioning—often without any service interruption.
Extending Volume Groups:
Adding capacity to a volume group is a straightforward, non-disruptive operation:
1234567891011121314151617181920212223242526272829303132333435363738394041424344454647
#!/bin/bash# Extending Volume Groups # === ADDING NEW PHYSICAL VOLUMES === # Step 1: Initialize the new device as a PVpvcreate /dev/sdd # Step 2: Add PV to existing VGvgextend vg_data /dev/sdd # Verify the extensionvgdisplay vg_data | grep -E 'VG Size|Free PE'# Before: VG Size 200.00 GiB, Free PE / Size 0 / 0# After: VG Size 300.00 GiB, Free PE / Size 25600 / 100.00 GiB # One-liner: Combine pvcreate and vgextendpvcreate /dev/sde && vgextend vg_data /dev/sde # Add multiple PVs at oncevgextend vg_data /dev/sdf /dev/sdg /dev/sdh # === ADDING PV WITH SPECIFIC MDA OPTIONS === # Add PV without metadata (data-only, relies on other PVs for metadata)pvcreate --metadatacopies 0 /dev/sdivgextend vg_data /dev/sdi # Add PV with extra metadata copiespvcreate --metadatasize 2M --pvmetadatacopies 2 /dev/sdjvgextend vg_data /dev/sdj # === POST-EXTENSION OPERATIONS === # Now you can extend logical volumes using the new spacelvextend -L +50G -r /dev/vg_data/lv_database # Or create new logical volumeslvcreate -L 100G -n lv_archive vg_data /dev/sdd # === CAPACITY PLANNING === # Check how much space was added from new PVpvs /dev/sdd -o pv_name,pv_size,pv_used,pv_free # Show overall VG utilizationvgs vg_data -o vg_name,vg_size,vg_free,pv_countReducing Volume Groups:
Removing physical volumes from a volume group requires more care. You must first ensure no data resides on the PV being removed, either by migrating data or because the PV was never used.
1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859
#!/bin/bash# Reducing Volume Groups # === SAFE REDUCTION (PV has no allocated extents) === # Check if PV has allocated extentspvs -o pv_name,pv_size,pv_used,pv_pe_count,pv_pe_alloc_count /dev/sdd # If Alloc PE is 0, safe to removevgreduce vg_data /dev/sdd # Wipe LVM label from removed PVpvremove /dev/sdd # === REDUCTION WITH DATA MIGRATION === # Step 1: Move all extents from sdd to other PVs in VG# This is an online operation but may take significant timepvmove /dev/sdd # The command will show progress:# /dev/sdd: Moved: 12.5%# /dev/sdd: Moved: 25.0%# ...# /dev/sdd: Moved: 100.0% # Step 2: Verify PV is now emptypvs /dev/sdd -o pv_used# Should show 0 # Step 3: Remove from VGvgreduce vg_data /dev/sdd # Step 4: Remove PV labelpvremove /dev/sdd # === TARGETED DATA MIGRATION === # Move only specific LV's extents off the PVpvmove -n lv_database /dev/sdd /dev/sde # Move data to specific destination PVs onlypvmove /dev/sdd /dev/sde /dev/sdf # === HANDLING ERRORS === # If VG has missing PV (disk failed), reduce with forcevgreduce --removemissing vg_data # If PV is inaccessible, force reductionvgreduce --force vg_data /dev/sdd # === PRE-REDUCTION VALIDATION === # List all LV segments showing which PV they uselvs --segments -o lv_name,seg_start,seg_size,devices # Find all LVs with extents on specific PVpvs -o pv_name,lv_name,seg_all /dev/sddThe pvmove operation physically copies data between devices. For a fully-utilized 1TB disk, this can take hours depending on I/O bandwidth. Plan pvmove operations during maintenance windows, even though they're online-safe. Use 'pvmove -b' for background operation and monitor with 'lvs -a -o +devices %vg_name'.
Volume Group Allocation Strategies
When logical volumes request space, LVM must decide which physical extents to allocate. The allocation policy controls this selection, affecting both performance and flexibility. Understanding allocation behavior is crucial for optimal storage design.
Allocation Policies in Depth:
- •contiguous: All physical extents must be adjacent on a single PV. Maximizes sequential I/O performance but severely limits flexibility. Fails if contiguous space isn't available, even if total free space suffices.
- •cling: Tries to place new extents on the same PV as existing extents of the LV. Falls back to other PVs if necessary. Useful for reducing inter-PV seeks.
- •cling_by_tags: Like cling, but uses PV tags to group 'similar' PVs. Enables cling behavior across multiple PVs with the same tag (e.g., same storage tier).
- •normal: Default policy. Prefers to use fewer PVs and avoids splitting across many devices. Balances between contiguity and flexibility.
- •anywhere: Use any available extent without preference. Enables maximum space utilization but may cause fragmentation. Required for striped volumes spanning multiple PVs.
Performance Implications:
| Policy | Sequential I/O | Random I/O | Space Efficiency | Use Case |
|---|---|---|---|---|
| contiguous | Excellent | Excellent | Poor | High-performance databases, sequential workloads |
| cling | Very Good | Good | Good | General database, consistent performance |
| normal | Good | Good | Very Good | Default, most workloads |
| anywhere | Variable | Fair | Excellent | Space-constrained environments |
Tagging for Advanced Allocation:
PV tags enable sophisticated allocation control by grouping physical volumes with similar characteristics:
123456789101112131415161718192021222324252627282930313233
#!/bin/bash# Using PV Tags for Allocation Control # Tag PVs by storage tierpvchange --addtag @tier_ssd /dev/nvme0n1p2 /dev/nvme1n1p2pvchange --addtag @tier_hdd /dev/sdb /dev/sdc /dev/sdd # Tag PVs by physical location (for fault domains)pvchange --addtag @rack_a /dev/nvme0n1p2 /dev/sdbpvchange --addtag @rack_b /dev/nvme1n1p2 /dev/sdc # View tags on PVspvs -o pv_name,pv_tags # Create LV only on SSD-tier PVslvcreate -L 50G -n lv_fast vg_mixed @tier_ssd # Create LV only on HDD-tier PVslvcreate -L 500G -n lv_archive vg_mixed @tier_hdd # Extend LV using only same-tier storage (cling_by_tags)lvextend --alloc cling_by_tags -L +25G /dev/vg_mixed/lv_fast # Create mirrored LV with legs on different racks (fault isolation)lvcreate --type raid1 -L 100G -n lv_critical --alloc anywhere \ vg_mixed @rack_a @rack_b # Remove a tagpvchange --deltag @tier_ssd /dev/nvme0n1p2 # List all available tags in systemvgs -o tagspvs -o tagsUse tags to implement storage tiering without separate volume groups. A single VG containing both SSDs (tagged @ssd) and HDDs (tagged @hdd) allows you to place high-performance LVs on SSDs and archival LVs on HDDs, while still enabling pvmove between tiers as requirements change.
Volume Group Operations
Beyond creation and extension, volume groups support various administrative operations for maintenance, migration, and troubleshooting.
Activation and Deactivation:
Volume groups and their logical volumes must be activated before use. Activation loads the device-mapper mappings into the kernel, making the LVs accessible as block devices.
1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283848586878889909192939495
#!/bin/bash# Volume Group Administrative Operations # === ACTIVATION CONTROL === # Activate all LVs in a VGvgchange -ay vg_data # Deactivate all LVs in a VG (must unmount first!)vgchange -an vg_data # Activate specific LV onlylvchange -ay /dev/vg_data/lv_database # Check activation statuslvs -o lv_name,lv_attr vg_data# Attribute field 5: 'a' = active, '-' = inactive # Set auto-activation on bootvgchange --setautoactivation y vg_data # Disable auto-activation (for manual control)vgchange --setautoactivation n vg_data # === VG RENAME === # Rename volume group (all LVs automatically get new paths)vgrename vg_data vg_storage # Verify new namevgs vg_storage # Update fstab entries for new device paths! # === VG EXPORT/IMPORT (for moving between systems) === # Step 1: Deactivate VGvgchange -an vg_portable # Step 2: Export VG (marks it as foreign)vgexport vg_portable # === On new system === # Step 3: Scan for all PVspvscan # Step 4: Import the VGvgimport vg_portable # Step 5: Activatevgchange -ay vg_portable # === VG SPLIT (separate PVs into new VG) === # Move LV and its PVs to new VG (must specify all necessary PVs)vgsplit vg_data vg_archive /dev/sdd /dev/sde -n lv_old_data # Note: Source VG must have no LVs with extents on moved PVs# except for the ones being moved # === VG MERGE (combine VGs) === # Merge vg_archive into vg_data# Deactivate source VG firstvgchange -an vg_archivevgmerge vg_data vg_archive # === VG ATTRIBUTE CHANGES === # Make VG read-only (all LVs become read-only)vgchange -p r vg_archive # Restore read-writevgchange -p rw vg_archive # Set VG maximum LV limitvgchange -l 50 vg_data # Remove the limit (set to 0 = unlimited)vgchange -l 0 vg_data # === RECOVERY OPERATIONS === # Repair VG with missing PVsvgreduce --removemissing --force vg_data # Restore VG from metadata backupvgcfgrestore -f /backup/vg_data.txt vg_data # Rebuild metadata from disk scanvgscan --mknodes # Force reread of all LVM metadatapvscan --cacheVG Export/Import Workflow:
When moving storage between systems (e.g., moving a disk array to a new server), the export/import workflow ensures clean handoff:
- Deactivate: Ensures no I/O is in progress
- Export: Marks metadata indicating VG is in transit
- Physical move: Relocate the disks/LUNs
- Scan: New system discovers the PVs
- Import: Clears the export flag, registers VG on new system
- Activate: Makes LVs accessible
This workflow prevents accidentally accessing the same storage from multiple systems, which could cause data corruption.
Two volume groups can only be merged if they have identical extent sizes. If extent sizes differ, you must create a new VG with the desired size and migrate data. Also ensure no LV name conflicts exist between the VGs before merging.
Production Best Practices
Designing volume group configurations for production environments requires balancing flexibility, performance, reliability, and manageability. The following practices derive from years of enterprise deployment experience.
- •Separate VGs by failure domain: Don't mix local disks and SAN storage in one VG; failures affect differently
- •Separate VGs by performance tier: Keep SSDs and HDDs in separate VGs for predictable performance
- •Use descriptive VG names: Names like 'vg_prod_db_ssd' are self-documenting
- •Reserve 15-20% free space: Enables snapshots, emergencies, and pvmove operations
- •Prefer fewer, larger VGs: Reduces administrative overhead; use tags for allocation control
- •Document extent size rationale: Future administrators need to understand why 8MB vs 4MB was chosen
- •Allocating 100% of VG space: Leaves no room for snapshots or growth
- •Mixing boot and data VGs: Boot issues shouldn't affect data availability
- •Using random extent sizes: Stick to defaults unless specific requirements exist
- •Ignoring metadata redundancy: At least 2 PVs should have metadata copies
- •No external metadata backup: Losing all PVs means losing everything
- •Complex VG topologies for small deployments: Adds complexity without benefit
Monitoring and Capacity Planning:
Proactive monitoring prevents emergency situations:
1234567891011121314151617181920212223242526272829303132333435
#!/bin/bash# LVM Monitoring Script - Add to cron for regular checks LOG="/var/log/lvm_monitor.log"THRESHOLD=85 # Alert when VG is 85% full echo "=== LVM Health Check: $(date) ===" >> $LOG # Check VG capacityfor vg in $(vgs --noheadings -o vg_name); do total=$(vgs --noheadings --units g -o vg_size $vg | tr -d ' g') free=$(vgs --noheadings --units g -o vg_free $vg | tr -d ' g') used_pct=$(echo "scale=0; (($total - $free) / $total) * 100" | bc) if [ $used_pct -ge $THRESHOLD ]; then echo "WARNING: VG $vg is ${used_pct}% full" >> $LOG # Send alert (email, Slack, PagerDuty, etc.) # mail -s "LVM Alert: $vg at ${used_pct}%" admin@example.com fidone # Check for missing PVspvscan 2>&1 | grep -i "missing" >> $LOG # Check metadata sequence consistencyfor vg in $(vgs --noheadings -o vg_name); do vgck $vg 2>> $LOGdone # List PVs with low free space (individual PV capacity)pvs --noheadings -o pv_name,pv_size,pv_free | while read pv size free; do # Alert if individual PV is nearly full (affects allocation locality) # This is informational for cling policy effectiveness echo "PV: $pv Size: $size Free: $free" >> $LOGdoneMaintain a storage layout document that includes: VG names and purposes, extent sizes with rationale, which LVs reside in each VG, expected growth patterns, and metadata backup locations. This document is invaluable during incidents and when onboarding new team members.
KEEP LEARNING
There’s more to understand.
You’re reading the introduction. Create a free account to read the complete lesson, save notes and keep track of your progress.