79:T5d7,**Decision Tree for Rebalancing:** ``` Start | Is there an active incident? / Yes No | | Is rebalancing the fix? Any HIGH priority factors? / \ / Yes No Yes No | | | | Emergency Address root Plan Any 2+ MEDIUM rebalance cause first proactive priority factors? rebalance / Yes No | | Schedule Continue rebalance monitoring ``` **Questions to Ask Before Rebalancing:** 1. **What problem are we solving?** Be specific about the symptom or projected issue 2. **What is the blast radius?** How much data will move? How long will it take? 3. **What is the risk if we DON'T rebalance?** Quantify in terms of outage probability or degradation 4. **What is the risk if we DO rebalance?** What could go wrong during the operation? 5. **Do we have a rollback plan?** Can we undo the rebalancing if issues arise? 6. **What is the maintenance window?** When can we execute with minimal user impact? 7. **Who needs to be informed?** Stakeholders, on-call teams, downstream service owners71:{"json":{"id":"seed_a368e6ca4e3311cb79d7d85ce8598bba","slug":"rebalancing-and-resharding","category":"System Design (HLD)","topic":"Rebalancing and Resharding","title":"Rebalancing and Resharding","description":"Master the critical techniques for redistributing data across shards and partitions in distributed databases. Learn when rebalancing is necessary, strategies for minimal disruption, consistent hashing algorithms, online resharding challenges, and practical considerations for production systems.","difficulty":"ADVANCED","estimatedMinutes":90,"pages":[{"id":"seed_d52d5732ab9828e827941db9a0dbdb13","title":"When Rebalancing Is Needed","order":0,"content":[{"id":"opening-hook","title":"The Rebalancing Imperative","content":[{"data":{"content":"In the lifecycle of any distributed database system, there comes an inevitable moment when the initial data distribution no longer serves the workload. Data that was once evenly spread across nodes becomes skewed. Hot partitions emerge where none existed. Nodes that were sufficient yesterday become bottlenecks today. **Rebalancing**—the process of redistributing data across the cluster—becomes not just desirable, but essential for system survival.\n\nUnderstanding when rebalancing is needed is arguably more important than knowing how to execute it. Rebalancing too early wastes resources and introduces unnecessary risk. Rebalancing too late leads to cascading failures, degraded performance, and potential data loss. The decision of *when* to rebalance is one of the most consequential choices a database administrator or systems architect can make."},"type":"text"},{"data":{"icon":"Target","title":"What You Will Learn","content":"By the end of this page, you will understand the fundamental triggers for rebalancing, how to detect imbalanced states through monitoring and metrics, the difference between proactive and reactive rebalancing, and how to develop a systematic framework for making rebalancing decisions in production environments.","variant":"info"},"type":"callout"}]},{"id":"data-distribution-drift","title":"Understanding Data Distribution Drift","content":[{"data":{"content":"When a distributed database is first deployed, data is typically distributed according to a carefully chosen partitioning strategy. Whether using range-based, hash-based, or directory-based partitioning, the initial distribution is designed to spread load evenly across nodes. However, real-world systems exhibit **distribution drift**—a gradual deviation from the intended data distribution over time.\n\n**Why Distribution Drift Occurs:**\n\nDistribution drift is not a bug but an inevitable consequence of how real applications generate and access data. Understanding its causes is the first step toward recognizing when rebalancing is needed."},"type":"text"},{"data":{"items":["**Non-Uniform Data Growth** — In most applications, data growth is not uniform. An e-commerce platform might see certain product categories explode in popularity. A social media platform might experience viral content that concentrates activity on specific user accounts. A financial system might see transaction volumes spike for particular accounts or regions.","**Temporal Access Patterns** — Data access often follows temporal patterns that weren't anticipated during initial design. Recent data is accessed more frequently than historical data. Certain time windows (business hours, holiday seasons) create concentrated load on specific partitions.","**Changing Business Requirements** — As applications evolve, query patterns change. New features might require accessing data in ways the original partitioning scheme wasn't optimized for. What was once a write-heavy workload might become read-heavy, or vice versa.","**Key Space Evolution** — The distribution of partition keys changes over time. Sequential ID generation can cause range-partitioned systems to accumulate data on the 'latest' partition. User-based partitioning can become skewed as certain users become power users.","**Infrastructure Changes** — Hardware failures, node additions, and capacity changes all affect the expected data distribution. A cluster that was balanced at 5 nodes may become imbalanced when expanded to 7 nodes."],"title":{"icon":"AlertCircle","text":"Primary Causes of Distribution Drift","type":"icon","color":"primary","variant":"outline-circle"},"variant":"accent"},"type":"list"},{"data":{"icon":"Info","title":"The Inevitability of Drift","content":"Distribution drift is not a sign of poor design—it's a fundamental property of dynamic systems. Even the best partitioning scheme will eventually drift from its optimal state. The goal is not to prevent drift but to detect it early and respond appropriately.","variant":"note"},"type":"callout"}],"titleType":"number","titleNumber":1},{"id":"hot-spots-and-skewed-partitions","title":"Hot Spots and Skewed Partitions","content":[{"data":{"content":"The most visible symptom of distribution drift is the emergence of **hot spots**—partitions that receive disproportionately high load compared to others. Hot spots are particularly dangerous because they create single points of congestion that can limit the entire system's throughput, regardless of how much capacity exists elsewhere.\n\n**Anatomy of a Hot Spot:**\n\nA hot spot forms when one or more partitions receive significantly more operations than the cluster average. This can manifest as:"},"type":"text"},{"data":{"rows":[["Write Hot Spot","Concentrated write operations on specific keys","High write latency, WAL contention, replication lag","Auto-incrementing IDs on range partitions, counters, event logs"],["Read Hot Spot","Popular data accessed far more than average","Cache misses, high read latency, connection exhaustion","Viral content, celebrity users, trending items"],["Size Skew","Data volume imbalance across partitions","Disk space exhaustion, compaction storms, backup delays","Large text fields, media storage, audit logs"],["Mixed Hot Spot","Combination of read/write concentration","All of the above, plus lock contention","Popular user profiles, frequently updated leaderboards"]],"caption":"Hot Spot Types and Their Characteristics","headers":["Hot Spot Type","Cause","Symptoms","Examples"]},"type":"table"},{"data":{"content":"**The Cascade Effect:**\n\nHot spots rarely remain isolated problems. When one partition becomes overloaded:\n\n1. **Request Queuing**: Operations targeting the hot partition begin to queue, increasing latency\n2. **Connection Pool Exhaustion**: Application connections waiting on the hot partition block connections to healthy partitions\n3. **Timeout Cascade**: Timeouts on the hot partition cause retries, which increase load further\n4. **Replication Lag**: If using replication, the hot partition's followers fall behind\n5. **Failover Instability**: The hot partition may appear unhealthy, triggering unnecessary failovers\n\nThis cascade effect means that a hot spot affecting 10% of data can degrade the entire system by 50% or more. The non-linear relationship between localized overload and global performance is why hot spot detection is critical."},"type":"text"},{"data":{"icon":"AlertCircle","title":"Hot Spot Detection Is Non-Trivial","content":"Average metrics hide hot spots. A cluster showing 60% average CPU utilization might have one node at 95% and others at 50%. Always examine distributions (p50, p90, p99, max) rather than averages when assessing partition health.","variant":"warning"},"type":"callout"}],"titleType":"number","titleNumber":2},{"id":"capacity-exhaustion-signals","title":"Capacity Exhaustion Signals","content":[{"data":{"content":"Beyond hot spots, rebalancing becomes necessary when individual partitions or nodes approach their capacity limits. **Capacity exhaustion** can occur along multiple dimensions, each requiring careful monitoring and distinct response strategies.\n\n**The Five Dimensions of Capacity:**\n\nModern distributed databases must manage capacity across multiple resource dimensions. Exhaustion in any single dimension can necessitate rebalancing, even if other dimensions have ample headroom."},"type":"text"},{"data":{"items":["**Storage Capacity** — The most fundamental constraint. When a partition approaches disk capacity, writes begin to fail. Storage exhaustion is typically the most visible but also the most predictable, as data growth rates are usually observable. Warning threshold: 70-80% utilization.","**Memory Capacity** — Critical for databases that maintain indexes, caches, or hot data in memory. Memory pressure leads to increased disk I/O, cache evictions, and severely degraded query performance. Warning threshold: 80-85% utilization.","**CPU Capacity** — Compute-bound operations (complex queries, cryptographic operations, compression) can exhaust CPU capacity. Unlike storage, CPU exhaustion is often bursty and harder to predict. Warning threshold: 70% sustained utilization.","**Network Capacity** — Often overlooked, network bandwidth can become the limiting factor for data-intensive operations, replication, and inter-node communication. Warning threshold: 60-70% sustained utilization.","**Connection Capacity** — Every database has limits on concurrent connections. Connection exhaustion blocks new operations entirely, regardless of other available capacity. Warning threshold: 70-80% of maximum connections."],"title":{"icon":"Target","text":"Critical Capacity Dimensions","type":"icon","color":"destructive","variant":"outline-circle"},"variant":"glass"},"type":"list"},{"data":{"content":"**Predictive Capacity Planning:**\n\nEffective capacity management requires not just monitoring current utilization but projecting future needs. Key metrics to track:\n\n- **Growth Rate**: How fast is data/traffic growing? (Daily, weekly, monthly growth percentages)\n- **Time to Exhaustion**: At current growth rates, when will each capacity dimension hit critical thresholds?\n- **Seasonality**: Are there predictable spikes (end of month, holiday seasons, promotional events)?\n- **Headroom Requirements**: How much spare capacity is needed for burst handling and operational safety?"},"type":"text"},{"data":{"icon":"Lightbulb","title":"The 80% Rule","content":"A common operational guideline is to begin rebalancing planning when any capacity dimension reaches 80% utilization. This provides sufficient headroom for unexpected spikes while allowing time to plan and execute a careful rebalancing operation.","variant":"tip"},"type":"callout"}],"titleType":"number","titleNumber":3},{"id":"infrastructure-changes-as-triggers","title":"Infrastructure Changes as Triggers","content":[{"data":{"content":"Some of the most common triggers for rebalancing are changes to the infrastructure itself. Unlike gradual drift, infrastructure changes create sudden imbalances that may require immediate attention.\n\n**Node Failures and Replacements:**\n\nWhen a node fails and is replaced, the new node starts empty. Depending on the database architecture, this might mean:\n\n- **Immediate Data Migration**: The system automatically begins moving data to the new node\n- **Gradual Rebalancing**: New writes are directed to the new node while old data remains skewed\n- **Manual Intervention Required**: An operator must initiate data movement to the new node\n\nRegardless of the approach, the period during and immediately after node replacement is one of heightened imbalance."},"type":"text"},{"data":{"left":[{"data":{"items":["New nodes have no data initially","Hash-based systems require key remapping","Range-based systems need partition splits","Replication topology changes","Network reconfiguration needed","Client routing updates required"],"title":{"icon":"XCircle","text":"Cluster Expansion Challenges","type":"icon","color":"destructive","variant":"outline-circle"},"variant":"accent"},"type":"list"}],"right":[{"data":{"items":["Data from removed nodes must migrate","Remaining nodes must absorb extra load","Risk of overloading remaining nodes","Potential consistency issues during drain","Requires careful coordination with clients","May trigger compaction storms"],"title":{"icon":"CheckCircle","text":"Cluster Contraction Challenges","type":"icon","color":"primary","variant":"outline-circle"},"variant":"accent"},"type":"list"}]},"type":"two-column"},{"data":{"content":"**Hardware Heterogeneity:**\n\nModern clusters often have heterogeneous hardware due to incremental upgrades. A cluster might have:\n\n- Older nodes with slower disks and less memory\n- Newer nodes with NVMe SSDs and larger RAM\n- Different CPU generations with varying performance characteristics\n\nEven if data is evenly distributed by count, performance imbalances arise because newer hardware can handle more load. Rebalancing might be needed to shift more data to more capable nodes—a process called **capacity-aware balancing**.\n\n**Cloud Considerations:**\n\nIn cloud environments, additional triggers include:\n\n- **Instance Type Changes**: Migrating to different instance types (e.g., upgrading from m5.large to m5.xlarge)\n- **Availability Zone Changes**: Spreading data across AZs for resilience\n- **Regional Expansion**: Extending to new geographic regions for lower latency\n- **Spot Instance Volatility**: Handling sudden termination of spot/preemptible instances"},"type":"text"}],"titleType":"number","titleNumber":4},{"id":"performance-degradation-patterns","title":"Performance Degradation Patterns","content":[{"data":{"content":"Often, the need for rebalancing reveals itself through observable performance degradation. Recognizing these patterns early allows for proactive intervention before degradation becomes critical.\n\n**Key Performance Indicators (KPIs) That Signal Rebalancing Need:**"},"type":"text"},{"data":{"rows":[["Read Latency (p99)","< 10ms","> 50ms","> 200ms","Hot read partitions or cache inefficiency"],["Write Latency (p99)","< 20ms","> 100ms","> 500ms","Write contention or replication lag"],["Replication Lag","< 100ms","> 1s","> 10s","Overloaded primary or network issues"],["Query Queue Depth","< 10","> 100","> 1000","Compute or I/O saturation"],["Connection Utilization","< 50%","> 70%","> 90%","Client concentration on specific nodes"],["Disk I/O Wait","< 5%","> 15%","> 30%","Storage saturation"]],"caption":"Performance Degradation Indicators for Rebalancing","headers":["Indicator","Normal Range","Warning Threshold","Critical Threshold","What It Signals"]},"type":"table"},{"data":{"content":"**Interpreting Performance Variance:**\n\nOne of the most telling signs of imbalance is high variance in performance metrics across partitions. When examining cluster health:\n\n- **Healthy Cluster**: All nodes show similar p99 latencies (within 20% of each other)\n- **Moderate Imbalance**: Some nodes show 2-3x higher latency than others\n- **Severe Imbalance**: Worst-performing nodes show 5-10x higher latency than best-performing nodes\n\n**The Long Tail Problem:**\n\nIn a sharded system, overall system latency is determined by the slowest shard involved in any operation. If a query touches 10 shards and 9 respond in 5ms but one responds in 500ms, the user experiences 500ms latency. This 'long tail' effect means that even a small number of overloaded partitions can dramatically impact user experience."},"type":"text"},{"data":{"icon":"AlertCircle","title":"The Degradation Spiral","content":"Performance degradation often accelerates. Initial slowdowns cause retry storms, which increase load, which causes more slowdowns. By the time degradation is noticeable to users, the system may already be in a dangerous state. Early detection through monitoring is essential.","variant":"error"},"type":"callout"}],"titleType":"number","titleNumber":5},{"id":"proactive-vs-reactive-rebalancing","title":"Proactive vs. Reactive Rebalancing","content":[{"data":{"content":"The timing of rebalancing operations falls into two broad categories: **proactive** (before problems occur) and **reactive** (in response to problems). Each approach has its place, but mature operations teams strongly favor proactive rebalancing.\n\n**Reactive Rebalancing:**\n\nReactive rebalancing occurs when system health has already degraded:\n\n- Triggered by alerts, user complaints, or outages\n- Often executed under time pressure\n- Higher risk of errors due to urgency\n- May conflict with peak usage periods\n- Limited time for testing and validation\n\n**Proactive Rebalancing:**\n\nProactive rebalancing anticipates future needs:\n\n- Scheduled during low-traffic windows\n- Based on capacity projections and trend analysis\n- Allows thorough planning and testing\n- Can be rolled back if issues arise\n- Minimizes user impact"},"type":"text"},{"data":{"left":[{"data":{"items":["Any capacity dimension approaching 70%","Variance in partition sizes exceeds 30%","Planned infrastructure changes upcoming","Seasonal traffic increase anticipated","New features will change access patterns","Geometric data growth detected"],"title":{"icon":"Target","text":"When to Proactively Rebalance","type":"icon","color":"primary","variant":"outline-circle"},"variant":"accent"},"type":"list"}],"right":[{"data":{"items":["Unexpected viral content/traffic spike","Sudden node failure without redundancy","Security incident requiring isolation","Unanticipated business event impact","Discovery of critical data corruption","Vendor or cloud provider incidents"],"title":{"icon":"AlertCircle","text":"When Reactive Is Unavoidable","type":"icon","color":"destructive","variant":"outline-circle"},"variant":"accent"},"type":"list"}]},"type":"two-column"},{"data":{"content":"**Building a Proactive Rebalancing Culture:**\n\nOrganizations that successfully practice proactive rebalancing share common characteristics:\n\n1. **Comprehensive Monitoring**: Detailed metrics on all capacity dimensions, with historical trend analysis\n2. **Defined Thresholds**: Clear, documented thresholds that trigger rebalancing consideration\n3. **Regular Reviews**: Weekly or monthly capacity planning reviews\n4. **Runbook-Driven Operations**: Pre-written, tested procedures for common rebalancing scenarios\n5. **Post-Incident Learning**: Every reactive rebalancing leads to process improvements to prevent recurrence"},"type":"text"},{"data":{"icon":"Lightbulb","title":"The 3:1 Rule","content":"Industry experience suggests that proactive rebalancing is approximately three times less disruptive than reactive rebalancing. The time invested in monitoring, planning, and scheduled maintenance pays dividends in reduced incidents and better user experience.","variant":"tip"},"type":"callout"}],"titleType":"number","titleNumber":6},{"id":"decision-framework-for-rebalancing","title":"Decision Framework for Rebalancing","content":[{"data":{"content":"Given the complexity and risk of rebalancing operations, a structured decision framework helps ensure that rebalancing is undertaken only when truly necessary and with appropriate planning.\n\n**The Rebalancing Decision Matrix:**\n\nWhen evaluating whether to rebalance, consider these factors:"},"type":"text"},{"data":{"rows":[["Capacity Utilization","< 60% peak","60-80% peak","> 80% peak"],["Partition Size Variance","< 20% deviation","20-40% deviation","> 40% deviation"],["Performance Variance","< 20% latency diff","20-50% latency diff","> 50% latency diff"],["Time to Exhaustion","> 6 months","3-6 months","< 3 months"],["Upcoming Events","None known","Moderate growth expected","Major traffic spike expected"],["System Stability","Stable, no recent issues","Minor issues observed","Active incidents related to balance"]],"caption":"Rebalancing Decision Matrix","headers":["Factor","Low Priority","Medium Priority","High Priority"]},"type":"table"},{"data":{"content":"$79"},"type":"text"},{"data":{"icon":"Info","title":"Document the Decision","content":"Every rebalancing decision—whether to proceed or to defer—should be documented. This creates an audit trail, facilitates post-mortems, and helps refine decision criteria over time.","variant":"note"},"type":"callout"}],"titleType":"number","titleNumber":7},{"id":"summary-and-next","title":"Summary: When Rebalancing Is Needed","content":[{"data":{"content":"Recognizing when rebalancing is needed is a critical skill for database administrators and system architects. The key insights from this page:"},"type":"text"},{"data":{"type":"ordered","items":["**Distribution drift is inevitable** — Real-world data and access patterns never match initial assumptions perfectly. Drift detection, not prevention, is the goal.","**Hot spots are system-wide threats** — A single overloaded partition can degrade the entire cluster through cascade effects. Early hot spot detection is critical.","**Monitor multiple capacity dimensions** — Storage, memory, CPU, network, and connections can each become the limiting factor. Balance all five.","**Infrastructure changes create sudden imbalances** — Node failures, additions, and hardware changes require immediate attention to data distribution.","**Performance variance signals imbalance** — When partition latencies diverge significantly, rebalancing should be considered.","**Proactive beats reactive** — Scheduled, planned rebalancing is far less disruptive than emergency operations under outage conditions.","**Use a structured decision framework** — Quantitative thresholds and decision trees reduce subjectivity and improve consistency."],"title":{"icon":"CheckCircle","text":"Key Takeaways","type":"icon","color":"primary","variant":"outline-circle"},"variant":"accent"},"type":"list"},{"data":{"content":"**What's Next:**\n\nNow that we understand *when* rebalancing is needed, the next page explores *how* to execute rebalancing with minimal disruption. We'll examine specific strategies for moving data between partitions, techniques for maintaining availability during rebalancing, and approaches to coordinate rebalancing with ongoing operations."},"type":"text"},{"data":{"icon":"CheckCircle","title":"Page Complete","content":"You now have a comprehensive understanding of the triggers, signals, and decision criteria for database rebalancing. This foundation prepares you for the practical strategies covered in the following pages.","variant":"success"},"type":"callout"}],"titleType":"number","titleNumber":8}],"isLocked":false},{"id":"seed_1caae81cc269dd97dc27738daea748ed","title":"Strategies for Minimal Disruption","order":1,"content":null,"isLocked":true},{"id":"seed_d8ee540dfd5680980d672d2fa9ba7377","title":"Consistent Hashing for Distribution","order":2,"content":null,"isLocked":true},{"id":"seed_8eade76e219c73646bd72d5de3616045","title":"Online Resharding Challenges","order":3,"content":null,"isLocked":true},{"id":"seed_7e1bc2204c59a4dcfaf4b251fc7cc376","title":"Practical Considerations","order":4,"content":null,"isLocked":true}]}}