GameDays - Learning Module

Loading content...

0/273

Running a GameDay

Showtime: Where Planning Meets Practice

The planning is complete. Stakeholders are informed. Safety controls are verified. Now comes the moment of truth: actually running the GameDay.

Execution is where chaos engineering theory transforms into organizational capability. The best-laid plans are useless if the exercise devolves into confusion, if responders feel unsupported, or if the Game Master loses control of pacing. Conversely, skillful execution extracts maximum learning from even modest scenarios.

This page provides a comprehensive guide to running GameDays that are educational, safe, and—importantly—create appetite for future exercises rather than organizational trauma.

What You Will Learn

By the end of this page, you will understand how to conduct effective pre-exercise briefings, manage failure injection and timing, observe responder behaviors, handle unexpected situations, maintain safety throughout execution, and close the exercise cleanly. You'll learn the art of Game Mastering that makes GameDays valuable.

The Pre-Exercise Briefing

Every GameDay begins with a briefing that sets expectations, assigns roles, and ensures all participants are ready. This briefing typically occurs 15-30 minutes before failure injection begins.

The briefing agenda:

Briefing Checklist

•Welcome and Purpose — Remind everyone why we're here: to learn, not to judge. Emphasize that discovering weaknesses is success, not failure.
•Role Confirmation — Verify each person understands their role. Game Master? Safety Officer? Responders? Observers? Scribe? Confirm everyone is present.
•Scope Review — Clearly state which systems are in scope, what types of failures may occur, and the time boundary. Responders should know the scope but not the specific scenarios.
•Safety Procedures — Review abort criteria, the abort mechanism, and who has authority to call abort. Emphasize: anyone can call for a pause if something seems wrong.
•Communication Channels — Point everyone to the GameDay channel, confirm they can access dashboards and runbooks, and establish how to signal 'this is real, not the exercise.'
•Observer Guidelines — Remind observers not to help responders during the exercise—their job is to watch and take notes, not to rescue.
•Questions — Answer any final questions. If participants have concerns, address them before starting.
•Starting Signal — Announce the exact time failure injection will begin, or confirm 'starting now' and begin immediately after briefing.

Setting the psychological tone:

The briefing sets the emotional tone for the entire exercise. Key psychological principles:

Create psychological safety:

'This is a learning exercise—there are no wrong answers'
'If something seems confusing, that's information, not personal failure'
'We're testing our systems and processes, not testing you'

Establish realistic urgency without panic:

'Treat this as you would a real incident'
'Work through your normal diagnosis process'
'If you need to escalate, escalate'

Frame discovery as success:

'If we find a gap, that's a successful GameDay'
'We want to discover problems here, not at 3 AM with angry customers'
'Every finding makes us stronger'

Respectors should feel appropriately focused but not terrified. If participants are visibly anxious, that's a signal that either the GameDay scope is too ambitious or organizational culture needs work before production exercises.

The 'Two-Minute Rule'

After the briefing, give participants two minutes to gather their thoughts, pull up dashboards, and settle in. This transition time helps responders mentally shift from briefing mode to response mode, mimicking the moment when an on-call engineer gets paged and needs to orient themselves.

Failure Injection: The Game Master's Role

The Game Master controls the introduction of failures. This role requires technical competence, situational awareness, and judgment about pacing. The goal is to create conditions that simulate real incidents while maintaining safety and learning opportunities.

Injection execution principles:

•Verify readiness before injection — Confirm monitoring dashboards are visible, responders are ready, and communication channels are open.
•Start single-threaded — Inject one failure at a time. Wait to see the system and responder response before introducing additional complexity.
•Log injection timing precisely — Record exactly when each failure was injected. 'Database primary terminated at 14:32:15 UTC.'
•Observe immediate effects — Watch metrics and logs for expected behavior. If the failure doesn't manifest as expected, that itself is information.
•Hold position when responders are actively working — Don't inject new failures while responders are in the middle of diagnosis or remediation. Let scenarios play out.
•Maintain silence unless safety requires intervention — The Game Master observes, doesn't advise. If responders are struggling, that's what observers should note, not a cue to help.

Converting Mermaid diagram...

Multi-scenario GameDays:

For GameDays with multiple planned failure scenarios, pacing is critical:

Allow recovery between scenarios — Don't layer new failures onto unrecovered previous failures unless specifically testing cascade resilience
Build in natural break points — After recovery from scenario one, a 5-minute pause lets responders reset before the next challenge
Increase complexity progressively — Start with the simpler scenario, then escalate. This builds responder confidence and matches how real incidents often evolve
Be prepared to skip scenarios — If time runs short or responders are fatigued, it's better to skip a planned scenario than to rush through it without learning

The Art of Not Helping

The hardest skill for experienced engineers serving as Game Masters is watching responders struggle without intervening. Resist the urge to drop hints. If responders take an unusual diagnostic path, let them—that's exactly what would happen at 3 AM. The struggle is where learning happens. Observers should note the struggle for the debrief, not interrupt to 'save' responders.

Observation and Documentation

A GameDay without effective observation produces stories but not actionable insights. Observers and the scribe capture the raw material that will be refined into improvements during the debrief.

What observers should watch for:

Things That Went Well

•Quick detection: 'Alert fired 45 seconds after failure'
•Effective use of runbooks: 'Responder found relevant section immediately'
•Good communication: 'Incident commander kept everyone informed'
•Correct diagnosis: 'Team identified root cause in 8 minutes'
•Smooth escalation: 'Appropriate expert was pulled in efficiently'
•Automation worked: 'Failover completed without manual intervention'
•Calm demeanor: 'No panic even under pressure'

Areas for Improvement

•Delayed detection: 'No alert for 10 minutes despite error spike'
•Runbook gaps: 'Documentation referenced non-existent server'
•Communication breakdown: 'Support team wasn't notified'
•Diagnostic dead-ends: 'Spent 15 minutes on unrelated subsystem'
•Escalation hesitation: 'Needed expert help but didn't ask'
•Manual toil: 'Failover required 12 manual steps'
•Stress indicators: 'Visible frustration impacting decisions'

The Scribe's Timeline:

The scribe maintains a chronological record of the exercise. This timeline is invaluable for post-exercise analysis and for building organizational memory.

gameday-timeline-example.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# GameDay Timeline - Database Failover Exercise
## Date: 2025-03-15, 14:00-16:00 UTC
 
| Time (UTC) | Event | Actor | Notes |
|------------|-------|-------|-------|
| 14:00 | Briefing begins | Game Master | All participants present |
| 14:15 | Exercise starts, failure injection | Game Master | Primary DB network isolated |
| 14:16 | First alert fires | PagerDuty | "Database connection pool exhausted" |
| 14:17 | Incident Commander acknowledges | Alice | Opens incident channel |
| 14:19 | Initial diagnosis begins | Bob | Checking database metrics |
| 14:23 | Root cause identified | Bob | "Primary DB unreachable" |
| 14:24 | Decision to initiate failover | Alice | |
| 14:25 | Failover command executed | Bob | Following runbook Step 5.2 |
| 14:26 | Runbook issue discovered | Bob | "Step 5.3 references old configuration path" |
| 14:28 | Workaround applied | Bob | Found correct path in sidebar notes |
| 14:31 | Failover complete | System | Replica promoted to primary |
| 14:33 | Customer traffic restored | System | Error rates returning to baseline |
| 14:35 | Verification checks pass | Carol | All endpoints responding |
| 14:40 | Exercise concludes | Game Master | Begin wrap-up |
 
## Key Observations
- Detection was fast (< 2 minutes)
- Runbook had outdated configuration reference (action item)
- Failover took longer than expected (16 minutes vs. target 5 minutes)
- Team communication was clear throughout

Observation tips for effectiveness:

Capture quotes — Exact phrases like 'I don't know where to find that' or 'This runbook is outdated' are powerful debrief material
Note timestamps — Approximate is fine; 'around 14:25' helps reconstruct the timeline
Observe body language — Confusion, frustration, or hesitation indicate potential process issues even when technical steps succeed
Don't filter — Write down everything; you can filter for relevance later. Details that seem minor often prove significant in retrospect
Stay separate — Observers should be visible but not in the responders' working area. Becoming part of the team defeats the purpose

Record the Session

With participant consent, recording the screen-share or video call during GameDays provides invaluable debrief material. Reviewing recordings often reveals dynamics that live observers missed. The recordings also serve as training material for engineers who couldn't participate.

Handling the Unexpected

Despite careful planning, GameDays often produce surprises. Part of the Game Master's skill is handling these unexpected developments while maintaining exercise value and safety.

Common Unexpected Situations and Responses
Situation	Risk Level	Recommended Response
Failure has bigger impact than expected	High	Immediately assess blast radius. If exceeding controls, abort and rollback. If within acceptable bounds but larger, continue with heightened monitoring.
Failure has no visible impact	Low	Verify injection succeeded. If yes, this might be a resilience success. If no, troubleshoot injection mechanism.
Responders are completely stuck	Medium	Wait 15-20 minutes. If no progress, Game Master may offer a general hint (not the answer). Consider if scenario is too difficult.
Real incident occurs during GameDay	High	Immediately pause GameDay. Rollback injected failures. Switch to real incident response. Resume GameDay later if feasible.
Responder becomes visibly distressed	Medium	Pause exercise briefly. Check in privately. Offer to swap in a different responder or simplify remaining scenarios.
Unexpected cascade failure occurs	High	Assess severity. If within controls, this is valuable discovery—observe carefully. If approaching abort criteria, execute rollback.
Responders solve too quickly	Low	Good problem to have! Either introduce planned secondary scenario or conclude early with lessons about effective preparation.
Key participant becomes unavailable	Medium	If backup is available, bring them in. If not, simplify remaining exercise or postpone complex scenarios.

The 'Real vs. Exercise' dilemma:

One of the trickiest situations: an alert fires during a GameDay, and it's unclear whether it's related to the exercise or a genuine production issue.

Protocol for ambiguous alerts:

Pause failure injection — Don't add complexity while investigating
Game Master checks scope — Is the alert for a system within GameDay scope? If yes, likely related. If not, potentially real.
Quick triage — Responders briefly investigate. If obviously GameDay-related, continue. If uncertain, escalate.
Default to caution — If there's genuine doubt, treat it as real. A false positive (treating a GameDay artifact as real) is far less costly than a false negative (treating a real incident as an exercise)
Clear communication — If deemed real, announce clearly: 'This appears to be a real incident. GameDay is paused. Switching to actual response.'

The Abort Decision

When abort criteria are approached, don't hesitate. The decision to abort should be swift and unequivocal. A GameDay that causes real customer impact undermines years of trust-building. The Safety Officer has unconditional authority to abort, and that authority should be exercised whenever there's genuine concern, not only when criteria are clearly exceeded.

Maintaining Psychological Safety

The most valuable GameDay learning often involves humans, not systems. For participants to be honest about their confusion, uncertainty, and mistakes, psychological safety must be maintained throughout the exercise.

Principles for psychological safety during execution:

Creating Safe Learning Conditions

•No public judgment — Neither Game Master nor observers should express disappointment or criticism during the exercise. 'Interesting' is neutral; 'That's wrong' is harmful.
•Normalize uncertainty — When responders say 'I don't know,' that's information, not failure. Acknowledge it neutrally: 'Note that we're uncertain about X—good to know.'
•Frame struggle as learning — 'This is exactly the kind of thing we want to practice' reminds everyone that difficulty is the point, not a problem.
•Protect responders from observers — If observers are reacting visibly to responder decisions, the Game Master should address it. Sighing, eye-rolling, or side conversations undermine safety.
•Allow pauses — If a responder needs a moment to think or calm down, that's fine. Real incidents don't come with pause buttons, but exercises can accommodate human needs.
•Celebrate small wins — 'Good catch on that log entry' or 'Nice use of the runbook' provides positive reinforcement even when overall progress is slow.
•Decouple performance from evaluation — GameDays are not performance reviews. If any participant feels their job is at stake, they'll hide weaknesses instead of exposing them.

When tensions rise:

Even with the best preparation, stress can escalate during GameDays. Signs that psychologically safety is eroding:

Responders becoming defensive about their actions
Blaming other teams or individuals
Refusing to try things because they might 'look stupid'
Shutting down or becoming silent
Aggressive questioning of the exercise value

De-escalation techniques:

Pause the exercise — A five-minute break can reset emotional states
Acknowledge difficulty — 'This is a hard scenario—harder than we expected'
Reinforce learning frame — 'Remember, we're here to learn. Everything we're seeing is valuable information'
Offer support — 'Would it help to have [expert] join as backup?'
Consider concluding early — A shortened exercise that ends well is better than a full exercise that traumatizes participants

The Game Master as Protector

The Game Master's responsibility extends beyond controlling the exercise—they're also protecting participants from negative experiences that would make future GameDays harder to staff. If observers are being unhelpful, if leadership is making responders nervous, or if the technology is creating undue frustration, the Game Master should intervene.

The Recovery and Wrap-Up Phase

Every GameDay must end with systems returned to known-good states. The wrap-up phase ensures the exercise concludes cleanly and sets the stage for the debrief.

Wrap-Up Checklist

•Announce exercise conclusion — 'The GameDay exercise is now complete. We're entering wrap-up.' Clear signal that the 'game' is over.
•Execute rollback procedures — Reverse all injected failures. Even if responders already recovered the system, ensure no GameDay artifacts remain.
•Verify system health — Check that all affected systems are healthy. Run your standard health checks, verify key metrics are nominal.
•Confirm no ongoing alerts — All GameDay-triggered alerts should be cleared. Any persisting alerts need investigation—either they're real issues or incomplete rollback.
•Update status channels — Post to the GameDay channel: 'Exercise complete, all systems verified healthy.' Remove any status page notices if used.
•Gather observation notes — Collect scribe timeline and observer notes while memories are fresh. These will feed the debrief.
•Thank participants — Express genuine appreciation for participation. Whether the GameDay went smoothly or was challenging, thank everyone for their effort.
•Transition to debrief — Either proceed immediately to debrief (preferred if energy allows) or confirm the debrief time if it's scheduled separately.

Time between exercise and debrief:

The ideal time for a debrief is immediately after the exercise, when details are fresh and emotional states are still elevated (in a good way). However, if the exercise was particularly long or intense, a short break is appropriate.

Immediately after (recommended):

Details are fresh
Observations haven't been filtered through retrospection
Participants are already assembled

After a break (15-30 minutes):

Allows biological needs
Lets initial emotions settle
Gives observers time to organize notes

Next day (avoid if possible):

Details are forgotten or distorted
Rationalization has occurred
Reassembling participants is difficult
Must be used if exercise ends late

If more than an hour passes between exercise and debrief, have participants write down their top 3 observations immediately, preserving them for later discussion.

The 'Clean Room' Check

Before declaring the exercise complete, imagine an engineer who wasn't involved in the GameDay looking at your systems tomorrow. Would they notice anything unusual? Would there be any artifacts, configurations, or states that would confuse them? The goal is a 'clean room'—no evidence of the exercise beyond documentation.

Game Master Decision Framework

Game Mastering is part art, part science. The following framework helps Game Masters navigate common decision points during execution.

Game Master Decision Guide
Situation	Question to Ask	Decision Guidance
Responders are stuck	How long have they been stuck? Are they learning from the struggle?	<10 min: Wait. 10-20 min: Consider a general hint. >20 min: Offer targeted help or pivot scenario.
Want to inject next failure	Have responders stabilized from previous failure? Are they ready for more?	If actively working on current issue, wait. If recovered and stable, proceed.
Observer wants to help	Is this a safety issue or just discomfort watching struggle?	If safety, allow. If discomfort, remind observer of role and politely redirect.
Approaching time boundary	Is remaining scenario worth rushing, or better to end gracefully?	Quality over completion. If rushing would reduce learning, end early.
Scenario going differently than expected	Is this interesting deviation or unproductive tangent?	If learning is happening, let it continue. If truly unproductive, gently redirect.
Responder asks if this is the exercise	Is clarification appropriate at this moment?	Generally yes—clarify that this is the exercise. Don't let confusion about reality persist.
Metrics approaching abort criteria	Are we trending toward or just near limits?	If improving, monitor closely. If worsening, prepare for abort. If flat near limit, consider preemptive abort.
Responder does something unexpected	Is this creative problem-solving or concerning improvisation?	Generally, let responders lead. Intervene only if safety is at risk.

The 'Lean Back' principle:

New Game Masters tend to over-manage—intervening too quickly, introducing failures too rapidly, or providing hints too readily. The general rule is: when in doubt, lean back.

Most GameDay value comes from watching what naturally happens, even when that involves struggle, confusion, or suboptimal decisions. These observations reveal where your systems and processes actually stand, not where they stand with active expert guidance.

The exception is safety: when safety is in question, lean forward immediately. But for all other concerns, bias toward patience and observation.

Summary: The Art of Running GameDays

Running a GameDay effectively requires balancing control with observation, support with challenge, and safety with realism. Let's consolidate the key principles:

Key Execution Principles

•Set the tone in the briefing — Establish psychological safety, clarify roles, and confirm safety procedures before any failures are injected.
•Game Masters control pacing, not outcomes — Inject failures deliberately, maintain silence to allow natural response, and resist the urge to help.
•Observe comprehensively — Both what goes well and what struggles are valuable. Capture detailed timelines and quotes for the debrief.
•Handle the unexpected calmly — Have protocols for ambiguous situations. When in doubt about safety, abort.
•Protect psychological safety throughout — The exercise's value depends on honest participation. If people feel judged, they'll hide weaknesses.
•Close cleanly — Verify all systems are healthy, all artifacts are removed, and transition to debrief while observations are fresh.
•When uncertain, lean back — The best GameDay learning comes from observation, not intervention. Patience reveals truth.

What's next:

The exercise is complete, but the work isn't finished. The next page covers learning from GameDays—the debrief process that converts observations into improvements, and the follow-up that ensures findings translate into lasting organizational change.

Page Complete

You now understand how to execute GameDays effectively—from briefing through failure injection, observation, and recovery. Skilled execution transforms careful planning into organizational learning. Next, we'll explore how to maximize the value extracted from these exercises through effective debriefs and follow-up.