GameDays - Learning Module

Loading content...

0/273

Planning a GameDay

The Plan That Makes the Practice

The difference between a chaotic mess and a valuable GameDay lies almost entirely in the planning. A poorly planned GameDay wastes everyone's time, potentially damages production systems, and—worst of all—teaches the wrong lesson: that chaos engineering is dangerous and unproductive.

Effective GameDay planning is an art form that balances ambition with safety, realism with control, and learning objectives with operational constraints. The hours invested in planning multiply the value extracted from the exercise itself.

This page provides a comprehensive framework for planning GameDays that deliver maximum learning while maintaining organizational trust and system stability.

What You Will Learn

By the end of this page, you will understand how to define clear GameDay objectives, select appropriate failure scenarios, identify the right participants, establish safety controls, communicate with stakeholders, and prepare your environment. You'll have a planning checklist that ensures no critical element is overlooked.

Defining Clear Objectives

Every effective GameDay begins with a clear answer to the question: What are we trying to learn? Without defined objectives, GameDays become unfocused chaos—entertainment rather than education.

Objectives should be specific, measurable, and tied to genuine organizational concerns. Vague goals like 'test our resilience' provide no guidance on what scenarios to run or how to evaluate success.

GameDay Objective Examples: Weak vs. Strong
Weak Objective	Why It's Weak	Strong Alternative
Test our failover	Which failover? What constitutes success? How will we measure it?	Validate that database failover from primary to replica completes in under 2 minutes with zero data loss
See how the team responds to incidents	No specific focus; difficult to extract actionable learning	Evaluate whether our on-call engineers can diagnose a cascading cache failure using only documented runbooks
Break things and see what happens	No hypothesis; pure exploration without direction	Test our hypothesis that the payment service remains functional when the recommendation service is unavailable
Practice incident response	Too generic; no specific skills or processes targeted	Practice the escalation workflow from P2 to P1 severity, including executive notification procedures
Validate our DR capability	DR is broad—which aspect? Which region? What RTO/RPO?	Validate that complete regional failover to DR site achieves our 4-hour RTO with less than 15 minutes of data loss

The hypothesis-driven approach:

Borrowing from the scientific method, effective GameDays are structured around hypotheses that the exercise will test:

Form a hypothesis — 'We believe that when the primary database becomes unavailable, the application will automatically fail over to the replica within 60 seconds with no manual intervention.'
Design the experiment — Define how the failure will be introduced, what measurements will be taken, and what outcomes would confirm or refute the hypothesis.
Execute and observe — Run the GameDay, carefully observing system behavior and human response.
Draw conclusions — Did the system behave as hypothesized? What did we learn?

This structure ensures that every GameDay produces actionable knowledge, whether the hypothesis is confirmed or refuted.

Common GameDay Objective Categories

•Failover Validation — Testing automated failover mechanisms for databases, load balancers, DNS, or entire regions
•Degradation Patterns — Validating that systems degrade gracefully rather than failing catastrophically when dependencies are unavailable
•Alert Effectiveness — Confirming that monitoring detects failures and routes alerts to appropriate responders
•Runbook Accuracy — Verifying that documented procedures can be followed successfully by current team members
•Recovery Time — Measuring actual recovery time against stated RTO commitments
•Communication Flows — Testing that incident communication reaches all stakeholders appropriately
•Cross-Team Coordination — Evaluating how multiple teams collaborate during complex incidents
•Capacity Under Stress — Validating that autoscaling or resource allocation handles sudden load changes

Start with Your Biggest Fear

When selecting objectives, ask the team: 'What failure scenario keeps you up at night?' Those fears often point to known unknowns—areas where the team suspects weakness but hasn't validated. These make excellent GameDay targets because they address real organizational anxiety.

Selecting Failure Scenarios

Once objectives are clear, the next step is designing failure scenarios that will test those objectives effectively. Scenario selection requires balancing realism with controllability, and ambition with safety.

Effective scenarios share several characteristics:

Realistic — They mirror failures that could actually happen in production
Controllable — They can be initiated and terminated reliably
Observable — Teams can see the effects and measure the response
Recoverable — Systems can be restored to normal operation
Scalable — Impact can be expanded or contracted as needed

Failure Scenario Categories

•Infrastructure Failures — Server crashes, disk failures, memory exhaustion, CPU saturation, out-of-memory kills
•Network Failures — Latency injection, packet loss, network partition, DNS failures, certificate expiration
•Dependency Failures — Database unavailability, cache failures, external API outages, message queue backup
•Application Failures — Process crashes, deadlocks, memory leaks, thread pool exhaustion, configuration errors
•Data Failures — Corrupt data, stale cache, replication lag, inconsistent state, backup restoration
•Security Incidents — Credential compromise (simulated), unusual access patterns, DDoS (controlled)
•Human Errors — Misconfiguration, accidental deletion, deployment rollback necessity
•Cascade Scenarios — Multi-component failures, regional outages, simultaneous dependent service failures

Progressive scenario complexity:

Organizations new to GameDays should start with simple, single-failure scenarios and progressively increase complexity:

Level 1: Single Component

Terminate a single non-critical service instance
Observe: Does the load balancer route around it? Do alerts fire?

Level 2: Single System

Make an entire service unavailable (all instances)
Observe: Do dependent services degrade gracefully? Does the circuit breaker open?

Level 3: Multi-System

Fail multiple related systems simultaneously
Observe: Can the on-call team diagnose the root cause amid cascading effects?

Level 4: Regional/Cross-cutting

Simulate regional datacenter failure affecting many systems
Observe: Does regional failover work? Is data replicated correctly?

Level 5: Business Process

Inject failures that disrupt an end-to-end business flow
Observe: Does the organization know how to communicate with affected customers?

Converting Mermaid diagram...

Avoid the 'Gotcha' Trap

GameDay scenarios should not be designed to trick or embarrass participants. The goal is learning, not proving that you can stump the on-call engineer. Excessively obscure scenarios produce anxiety rather than growth. Keep scenarios realistic enough that encountering them in production would be plausible.

Identifying Participants and Roles

The right participants make the difference between a GameDay that reveals organizational insights and one that merely demonstrates that experts can solve problems. Careful role assignment ensures productive dynamics during the exercise.

Essential GameDay Roles
Role	Responsibilities	Who Should Fill This Role
Game Master / Exercise Lead	Controls scenario execution, maintains timeline, manages safety controls, can pause or abort exercise	Senior engineer familiar with chaos tooling, not involved in response
Incident Commander (simulated)	Leads the response effort, coordinates responders, makes escalation decisions	Whoever would actually lead real incidents—test their capability
Technical Responders	Diagnose issues, execute remediation, follow runbooks	Engineers who would actually be on-call—rotate in less experienced engineers for training
Observers	Watch without intervening, take notes on process gaps, capture learning points	Senior engineers, SREs, or managers—anyone who can provide constructive debrief input
Safety Officer	Monitors for actual production impact, authorized to call abort, watches blast radius	Senior SRE or platform engineer with deep system knowledge and authority to halt
Scribe	Captures timeline, key decisions, observed behaviors for post-exercise analysis	Anyone organized and able to document rapidly without participating
Customer Representative (optional)	Provides customer perspective, evaluates communication effectiveness	Customer success or support team member, product manager

Participant selection principles:

•Test the actual responders — Don't stack the GameDay with experts who would never be first-responders. Put the on-call engineers who would actually face these scenarios in the responder seat.
•Cross-functional representation — Include representatives from all teams that would be involved in a real incident. GameDays often reveal coordination failures that only surface with multiple teams present.
•Rotate participation — Each GameDay should include some participants from previous exercises (for continuity) and some newcomers (for fresh perspectives and training).
•Include skeptics — Engineers who doubt the value of GameDays often become the strongest advocates after experiencing well-run exercises. Include them.
•Consider management presence — Having an engineering manager or director observe can provide valuable executive perspective and demonstrates organizational commitment.

The separation of knowledge:

A critical principle in GameDay design is asymmetric information. The Game Master and Safety Officer know the full scope of planned failures. The responders should not.

This separation is essential because:

Real incidents come without advance warning—simulating this builds realistic response skills
Responders who know the 'answer' will short-circuit diagnostic processes
Genuine surprise reveals actual team capabilities rather than rehearsed responses
Learning about systematic diagnosis is more valuable than demonstrating pre-known solutions

That said, responders should know that a GameDay is occurring—true 'surprise drills' require extremely mature organizational trust and are not appropriate for most organizations.

The Observer Effect

Having observers present changes responder behavior—usually for the better during early GameDays, as people try harder. This is acceptable. The goal isn't to trick people into revealing incompetence; it's to practice and improve. As GameDays become routine, the observer effect diminishes and behavior becomes more natural.

Establishing Safety Controls

Safety is paramount in GameDay planning. The exercise must be designed such that value is extracted without causing unacceptable damage to production systems, customer experience, or team morale. Comprehensive safety controls are non-negotiable.

Essential Safety Controls

•Blast Radius Limits — Define precisely which systems can be affected. In production, this might mean only specific regions, only certain tenants, or only traffic meeting specific criteria. Never allow unbounded impact.
•Abort Criteria — Specify exact conditions that trigger immediate exercise termination: 'If customer error rate exceeds 5%...', 'If any P1 alert fires for non-exercised systems...', 'If the Safety Officer calls abort for any reason...'
•Rollback Procedures — Every failure injection must have a tested reversal mechanism. Before starting, verify that the Game Master can undo each planned failure reliably and quickly.
•Time Boundaries — Set hard limits on exercise duration. If recovery isn't achieved within the time boundary, abort and restore based on rollback procedures rather than continuing indefinitely.
•Communication Channels — Establish separate channels for exercise coordination (where participants can ask 'is this real?') and for observers, keeping them distinct from actual incident channels.
•Customer Protection — For production exercises, implement safeguards such as feature flags routing affected traffic away from critical customers, synthetic traffic for testing, or canary deployments limiting exposure.
•Authorization Chain — Document who authorized the GameDay, ensuring appropriate management approval especially for production exercises or those with potential business impact.

The safety control checklist:

Before any GameDay execution, verify each of these items:

Pre-Exercise Verification

•☐ Rollback tested within last 24 hours
•☐ Safety Officer confirmed available
•☐ Abort criteria documented and shared
•☐ Blast radius limits configured in tooling
•☐ Communication channels established
•☐ All participants briefed on safety procedures
•☐ No conflicting maintenance windows
•☐ Appropriate authorization obtained

Abort Readiness

•☐ Game Master has tested abort mechanism
•☐ Safety Officer has direct abort capability
•☐ Fallback to manual rollback documented
•☐ Emergency contact list prepared
•☐ Real incident response on standby
•☐ Abort does not depend on affected systems
•☐ Time boundary alarm set
•☐ Metric dashboards showing blast radius

Never Compromise on Safety

If any safety control cannot be verified, do not proceed with the GameDay. Postponing is always preferable to an uncontrolled incident caused by your own exercise. A GameDay that damages customer trust or destabilizes production systems is worse than no GameDay at all.

Stakeholder Communication

GameDays don't happen in isolation. They affect—or could affect—various stakeholders across the organization. Proactive communication builds trust, prevents confusion, and ensures appropriate support is available.

Stakeholder Communication Matrix
Stakeholder Group	What They Need to Know	When to Communicate	Communication Channel
Engineering Leadership	Purpose, scope, timeline, potential risks, expected outcomes	1-2 weeks before (for approval), day-of reminder	Planning document, calendar invite
Participating Teams	Their role, time commitment, what to prepare, meeting details	1 week before (detailed briefing)	Team meetings, detailed email with agenda
On-Call Engineers	That a GameDay is occurring (prevents confusion with real incidents)	Day of exercise, at start time	Slack/chat announcement, on-call handover notes
Customer Support	Potential customer impact, timeline, what to tell customers if asked	Day before (briefing), day-of (start/end confirmation)	Email to support leadership, status page update if appropriate
Product Teams	That systems they depend on may behave unusually during the window	1 week before (awareness), day-of (confirmation)	Cross-functional sync, calendar blocking
Executive Sponsors	That exercise is occurring, expected learning value, any notable findings	Pre-exercise summary, post-exercise brief report	Executive summary document, follow-up meeting if findings warrant

The 'real vs. exercise' communication challenge:

During GameDays, especially those that generate real alerts, distinguishing between exercise-caused events and genuine incidents is critical. Establish clear protocols:

Prefix all GameDay-related communications — 'GAMEDAY: Service X is degraded per plan'
Create a dedicated GameDay channel — Separate from real incident channels
For particularly confusing scenarios, consider a codeword — Something observers can use to confirm 'this is still the exercise'
Maintain the real incident channel — If an actual incident occurs during the GameDay, it takes precedence; the exercise pauses

The goal is to practice realistic response while never creating confusion that could delay response to actual customer-impacting incidents.

Build Organizational Buy-In

Early GameDays are an opportunity to build organizational support. Communicate results widely, celebrate learnings (not just successes), and acknowledge participants. When people see GameDays as valuable learning opportunities rather than disruptive time-wasters, scheduling future exercises becomes easier.

Environment Preparation

The technical environment where the GameDay will run requires careful preparation. This includes ensuring chaos tooling is functional, monitoring is in place, and the target systems are in a known-good state before introducing failures.

Environment Preparation Checklist

•Chaos Tooling Verification — Test that Chaos Monkey, Gremlin, LitmusChaos, or whatever tooling you use can successfully inject the planned failures and can abort them reliably. Run a dry-run on a test system.
•Monitoring Baseline — Capture a baseline of normal system behavior before the exercise. You'll compare this against exercise behavior to identify anomalies.
•Dashboard Preparation — Create or configure dashboards that display the specific metrics relevant to your GameDay objectives. Participants should be able to observe the system's state clearly.
•Logging Verification — Ensure application and infrastructure logs are flowing to your aggregation system. Responders will need these for diagnosis.
•Runbook Accessibility — Confirm that all relevant runbooks are accessible, up-to-date, and that responders know where to find them.
•System Health Check — Verify that target systems are healthy before starting. Running a GameDay on a system already experiencing issues creates confusion.
•Artifact Preparation — If using pre-prepared scripts, configuration files, or test data, ensure they are ready and tested.
•Recording Setup — If you plan to record the session (for training or review), ensure screen recording tools are ready and participants consent.

Environment selection considerations:

Choosing the right environment for your GameDay depends on your maturity level and objectives:

Development/Local:

✅ Safe, low risk
✅ Easy to reset
❌ Doesn't test production infrastructure
❌ Responders know it's not 'real'

Staging/Pre-Production:

✅ Production-like infrastructure
✅ Minimal customer risk
✅ Can test most failure modes
❌ May have data or load differences from production

Production (with controls):

✅ Tests actual production resilience
✅ Most realistic learning
❌ Customer risk requires strict controls
❌ Stakes are higher for all participants

Canary/Shadow Production:

✅ Production traffic without production risk
✅ Realistic load characteristics
⚠️ More complex to set up
⚠️ Not all failure modes can be tested

The 'No Conflicting Changes' Rule

Schedule GameDays during periods without other significant changes. Don't run GameDays during deployment windows, database migrations, or major feature launches. If something goes wrong, you want to be confident it's related to the exercise, not an unrelated change.

The GameDay Planning Document

Every well-planned GameDay produces a planning document that serves as the authoritative reference for the exercise. This document captures all the decisions made during planning and provides the structure for execution.

A comprehensive planning document should include:

gameday-planning-template.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
# GameDay Planning Document
 
## Exercise Overview
- **Exercise Name:** [Descriptive name, e.g., "Q1 Database Failover GameDay"]
- **Date & Time:** [Start and end times, including timezone]
- **Environment:** [Production/Staging/etc. + specific systems]
- **Authorization:** [Who approved this exercise]
 
## Objectives
1. [Primary objective with measurable success criteria]
2. [Secondary objective with measurable success criteria]
3. [Learning questions to answer]
 
## Hypothesis
We believe that [system/process/team] will [expected behavior] when [failure scenario is introduced].
 
## Failure Scenarios
| Scenario | Implementation | Expected Behavior | Abort Trigger |
|----------|----------------|-------------------|---------------|
| [Name]   | [How injected] | [What should happen] | [When to abort] |
 
## Participants
| Role | Person | Contact | Backup |
|------|--------|---------|--------|
| Game Master | | | |
| Safety Officer | | | |
| Incident Commander | | | |
| Responders | | | |
| Observers | | | |
 
## Safety Controls
- **Blast Radius:** [Specific limits]
- **Abort Criteria:** [List of conditions]
- **Rollback Procedure:** [How to undo each failure]
- **Time Boundary:** [Maximum duration before forced abort]
- **Emergency Contacts:** [For escalation beyond GameDay scope]
 
## Communication Plan
- **Pre-Exercise Notification:** [Who, what, when]
- **During-Exercise Channel:** [Slack channel, etc.]
- **Post-Exercise Communication:** [Results summary distribution]
 
## Timeline
| Time | Activity | Owner |
|------|----------|-------|
| -30 min | Final preparation, system health check | Game Master |
| -15 min | Participant briefing | Game Master |
| 0:00 | Inject Failure #1 | Game Master |
| +X min | Inject Failure #2 (optional) | Game Master |
| +Y min | Time boundary / nominal end | Game Master |
 
## Post-Exercise
- **Debrief Time:** [Immediately after]
- **Debrief Location:** [Room/video call]
- **Action Item Tracking:** [Where will actions be logged]
- **Report Distribution:** [Who gets the summary]

Circulating the planning document:

The planning document should be shared with:

All participants (before the briefing, for their role-relevant sections)
Observers (full document)
Leadership/sponsors (summary sections)
Support teams (communication plan sections)

Store the document where it can be referenced during and after the exercise. It becomes part of your organizational memory of resilience practices.

Summary: The Art of GameDay Planning

Effective GameDay planning sets the stage for valuable learning. Let's consolidate the essential elements:

Key Planning Elements

•Clear objectives drive everything — Define specific, measurable goals and frame them as testable hypotheses.
•Scenario selection balances realism and safety — Start simple and progressively increase complexity with organizational maturity.
•Participants should reflect actual incident response — Test your on-call engineers, not just your experts, and clearly separate roles.
•Safety controls are non-negotiable — Blast radius limits, abort criteria, and rollback procedures must be tested before execution.
•Stakeholder communication builds trust — Proactive notification prevents confusion and builds organizational support for future exercises.
•Environment preparation ensures valid results — Verify tooling, monitoring, and system health before introducing failures.
•Document everything in a planning document — Capture all decisions in a reference that guides execution and preserves institutional knowledge.

What's next:

With a solid plan in place, the next challenge is execution. The following page covers how to run a GameDay effectively—managing the flow, observing behaviors, handling unexpected developments, and maintaining safety throughout the exercise.

Page Complete

You now understand how to plan GameDays that deliver maximum learning value while maintaining safety. Thorough planning ensures your chaos engineering exercises are productive rather than chaotic. Next, we'll learn how to execute these carefully planned exercises effectively.