CodePipeline and CodeBuild failure analysis

Exam alignment: DOP-C02 Domain 5 — event processing, configuration response, and troubleshooting.

Learning objective

Design, implement, and troubleshoot stage state, execution history, artifacts, IAM, environment variables, logs, caches, timeouts, and retries with precise event contracts, least-privilege execution, bounded retries, reliable evidence, and safe recovery.

Professional scenario

A production environment relies on stage state, execution history, artifacts, IAM, environment variables, logs, caches, timeouts, and retries. During an operational event, the workflow either drops context, repeats an unsafe action, or leaves responders without a reliable path from detection to mitigation and verified recovery.

Core concepts

• The lesson focuses on stage state, execution history, artifacts, IAM, environment variables, logs, caches, timeouts, and retries.
• An event describes something that happened; a command requests that an action be performed.
• Event consumers must be idempotent because retries and duplicate delivery can occur.
• Every automated action needs scope, preconditions, permissions, rate limits, failure handling, and evidence.
• Detection, notification, mitigation, recovery, and prevention are different stages.
• The first failed component or earliest unexpected state usually provides the highest-value troubleshooting evidence.

Architecture and response walkthrough

1. Define the triggering condition and business impact.
2. Capture a representative event, alarm, log, or failure state.
3. Identify source, routing, target, execution role, and affected resources.
4. Validate event contracts, permissions, quotas, and dependency readiness.
5. Execute a small controlled response or diagnostic step.
6. Verify the resulting state and preserve evidence.
7. Escalate, roll back, or continue with the next hypothesis.
8. Document prevention and retest.

Decision guide

Requirement	Preferred direction	Why
Known event requiring routing	EventBridge rule with precise pattern	Native event-driven response
Durable decoupling	SQS queue and idempotent consumer	Buffers and preserves work
Multi-step response	Step Functions or Systems Manager Automation	Explicit retries, branching, and history
Resource-state violation	AWS Config plus safe remediation	Evaluates and corrects non-desired state
Unknown failure	Hypothesis-driven investigation using logs, metrics, traces, events, and health	Avoids random changes

Failure modes and troubleshooting

• The rule matches too broadly and triggers unrelated actions.
• A retry repeats a destructive non-idempotent operation.
• The execution role is missing a required permission or has excessive permissions.
• The remediation target is stale, already recovered, or an approved exception.
• Responders change several variables at once and destroy useful evidence.
• Recovery restores the service but the underlying risk is not corrected or retested.

Security and operational controls

• Separate detection, orchestration, remediation, and audit roles.
• Prefer reversible containment and mitigation before destructive actions.
• Protect event buses, runbooks, pipelines, incident data, and evidence from unauthorized changes.
• Do not include secrets, credentials, or sensitive payloads in events, notifications, or incident channels.
• Record who initiated, approved, executed, and verified each sensitive action.

Hands-on lab

Goal: Build and validate a small incident or event-response workflow for this topic.

Tasks

1. Define one triggering event or failure condition.
2. Draw source, routing, target, permissions, retries, and failure destinations.
3. Build a minimal workflow or detailed configuration.
4. Introduce one controlled failure.
5. Diagnose the first failed component.
6. Perform a safe remediation or rollback.
7. Verify service state and record evidence.
8. Write one prevention action with owner and retest date.

Validation

• The trigger matches only the intended condition.
• Duplicate delivery does not create duplicate effects.
• The execution role is scoped.
• Failure is visible and recoverable.
• The final service and configuration state is verified.

Cost control: Use synthetic events and a narrow test scope. Remove rules, queues, state machines, runbooks, test resources, logs, and notifications.

Cleanup

1. Stop test publishers and scheduled rules.
2. Delete temporary targets, queues, DLQs, functions, state machines, runbooks, and alarms.
3. Revert test configuration changes and confirm the final state.

Exam traps

• Treating an event as guaranteed exactly-once delivery.
• Paging on every event without severity or impact classification.
• Automatically remediating before validating scope and current state.
• Changing infrastructure manually without preserving evidence or reconciling IaC.
• Stopping after mitigation without root-cause and prevention work.

Key takeaways

• stage state, execution history, artifacts, IAM, environment variables, logs, caches, timeouts, and retries.
• Automated response must be precise, idempotent, bounded, observable, and reversible where possible.
• Troubleshooting begins with evidence and hypotheses, not random configuration changes.

Review questions

1. What is the trigger and business impact?
2. Which service produces or captures the event?
3. How is duplicate delivery handled?
4. Which execution role performs the action?
5. What is the failure destination?
6. How is the current target state validated?
7. What is the rollback or escalation path?
8. Which evidence proves recovery?

Answers

1. A precisely defined event, alarm, health state, or failure with measurable impact.
2. An AWS service, CloudTrail, AWS Health, EventBridge, a custom producer, or telemetry system.
3. With an idempotency key, state check, conditional write, or safe repeatable runbook.
4. A least-privilege role dedicated to the workflow step.
5. A DLQ, failed execution state, OpsItem, alert, or preserved failed record.
6. By reading the resource and service state immediately before action.
7. A documented reversible action, previous version, manual approval, or incident escalation.
8. Metrics, logs, traces, event history, execution output, resource state, and user validation.