Resilience testing and chaos engineering

Exam alignment: DOP-C02 Domain 3 — highly available, scalable, and automatically recoverable solutions.

Learning objective

Design, implement, and troubleshoot hypotheses, steady state, bounded experiments, stop conditions, and remediation from measurable availability, scaling, and recovery requirements.

Professional scenario

Architecture diagrams claim redundancy, but no controlled failure has ever been tested.

Core concepts

• This lesson focuses on hypotheses, steady state, bounded experiments, stop conditions, and remediation.
• Start with business impact and measurable availability, RTO, and RPO.
• Map infrastructure, data, identity, quota, network, deployment, and human dependencies.
• Remove accidental single points of failure and limit blast radius.
• Define health signals, failure detection, failover, rollback, and failback.
• Measure actual capacity, replication lag, and recovery duration.
• Treat testing and operational readiness as part of the architecture.

Architecture walkthrough

1. Translate the business requirement into availability, RTO, RPO, capacity, and failure-domain needs.
2. Draw every dependency and identify shared control paths.
3. Select Multi-AZ, cross-Region, decoupling, replication, or backup patterns.
4. Define technical and business health signals.
5. Automate recovery with scoped permissions and known quotas.
6. Run a tabletop, restore test, or controlled fault experiment.
7. Compare actual results with objectives and improve the system.

Decision guide

Requirement	Preferred direction	Why
Continue during one AZ failure	Multi-AZ targets and surviving capacity	Removes the AZ as a single point of failure
Regional disaster recovery	A defined cross-Region strategy	Matches RTO, RPO, and cost
Burst or dependency failure	Asynchronous decoupling and backpressure	Prevents cascading synchronous failure
Logical data corruption	Protected backup and point-in-time recovery	Replication can copy bad changes
Production confidence	Game days and controlled fault injection	Architecture diagrams are not evidence

Failure modes and troubleshooting

• A standby exists but lacks current data, secrets, certificates, quotas, or capacity.
• Traffic moves before the recovery environment is ready.
• Retries amplify overload or duplicate non-idempotent work.
• A health check is shallow and does not represent the business function.
• Replication is enabled but lag and failed replication are not monitored.

Security and operational considerations

• Keep recovery access independent from the failed or compromised environment.
• Protect backup vaults, KMS keys, DNS changes, replication roles, and experiment controls.
• Use least-privilege automation roles and audit failover actions.
• Do not expose customer data in experiments, metrics, logs, or game-day evidence.

Hands-on lab

Goal: Design and validate one controlled resilience scenario.

Tasks

1. Write measurable availability, RTO, and RPO requirements.
2. Draw dependencies and failure domains.
3. Choose the architecture and rejected alternatives.
4. Define health, failover, rollback, and failback.
5. Introduce one controlled fault or tabletop inject.
6. Measure recovery and record improvements.

Validation

• The architecture meets the stated objective.
• Recovery does not rely on unavailable access paths.
• Actual results are compared with targets.

Cost control: Use tabletop simulations for expensive multi-Region designs and delete every live test resource.

Cleanup

1. Stop experiments and synthetic traffic.
2. Delete temporary compute, replicas, alarms, backups, and DNS test records.
3. Confirm that production recovery resources remain intact.

Exam traps

• Equating backup, replication, durability, and availability.
• Choosing active/active without considering cost, conflict, and operational complexity.
• Assuming a feature automatically meets RTO or RPO.
• Failing over traffic before data and capacity are ready.
• Treating an untested runbook as a reliable recovery mechanism.

Key takeaways

• Master hypotheses, steady state, bounded experiments, stop conditions, and remediation.
• Resilience objectives must be measurable.
• Recovery requires ready data, capacity, routing, permissions, and operations.
• Testing produces the evidence that the design works.

Review questions

1. Which business requirement drives this design?
2. What is the main failure domain?
3. Which hidden dependency can still fail?
4. Which health signal initiates recovery?
5. What are the RTO and RPO?
6. What must be ready before traffic moves?
7. How is failback performed?
8. How is the design tested?

Answers

1. The explicit availability and recovery objective.
2. The smallest boundary whose failure affects service.
3. A shared resource, credential, quota, control path, or human process.
4. A tested technical or business indicator.
5. They come from business impact analysis.
6. Data, capacity, version, configuration, credentials, routing, and observability.
7. Through a separate controlled and tested workflow.
8. With restore tests, failover exercises, game days, and fault injection.