Chapter project: Delivery baseline

This project turns the chapter's ideas into one reviewable delivery design. You will not deploy a large application. The goal is to show that you can connect requirements, accounts, identities, artifacts, pipeline stages, evidence, and rollback into one coherent system.

The result should be understandable by a developer, a security reviewer, and an operator. If a diagram or table only makes sense after a long verbal explanation, improve the artifact.

Scenario

Example Retail runs a containerized order API. The source code is stored in a central tooling account, while development and production workloads run in separate AWS accounts.

The company currently builds the application separately in every environment and deploys production manually. This creates three problems:

1. The production image may not be the same image that passed testing.
2. Administrators use broad permissions to complete releases.
3. Failed releases depend on a person remembering the recovery procedure.

Design a delivery baseline that corrects those weaknesses.

Fixed requirements

Your design must satisfy all of the following:

• A commit to the main branch starts the delivery process.
• The application is built once and the same immutable image is promoted.
• Development and production are separate AWS accounts.
• The tooling account must not receive permanent administrator credentials in workload accounts.
• Unit, security, integration, and production-readiness checks must produce evidence.
• Production uses a deployment strategy that can shift traffic gradually.
• A failed production health check starts an automatic rollback.
• Every deployment must be attributable to a source revision and pipeline execution.
• Production approval is required during the first phase of adoption.
• Artifacts must be encrypted and available only to the delivery identities that need them.
• The design must explain cost control and cleanup.

Assumptions you may make

Use these assumptions unless you document a justified alternative:

• AWS CodePipeline orchestrates the workflow.
• AWS CodeBuild runs build and test commands.
• Amazon ECR stores container images.
• Amazon ECS runs the service.
• AWS CodeDeploy controls an ECS blue/green deployment.
• Amazon CloudWatch provides alarms and operational metrics.
• AWS CloudTrail records API activity.
• The tooling account owns the pipeline artifact bucket and its customer managed KMS key.

The exact services are less important than the quality of your reasoning. A different service is acceptable when it satisfies the same constraints and you explain the trade-off.

Learning outcomes

After completing the project, you should be able to:

• translate prose requirements into architecture properties;
• distinguish orchestration, build, deployment, and runtime identities;
• trace an immutable artifact from source revision to production;
• specify measurable entry and exit criteria for each stage;
• design cross-account access without long-lived credentials;
• connect health evidence to an automatic rollback decision;
• explain the design under normal, failure, and recovery conditions;
• identify the most likely cause of a failed cross-account deployment.

Before you begin

Review the first four lessons and create a working folder for your artifacts. A diagramming tool is helpful, but a carefully labeled Markdown diagram is enough.

Do not begin by drawing AWS service icons. First extract the requirements. Architecture is a response to constraints, not a collection of services.

Phase 1: Build the requirement ledger

Create a table with one row per fixed requirement. Classify each item as a goal, hard constraint, preference, or operating assumption. Then translate it into a property that the design must expose.

Use this example as a starting point:

Requirement	Classification	Required property	Evidence
Build once and promote	Hard constraint	One image digest crosses all environments	Manifest and ECR digest
No permanent workload credentials	Hard constraint	STS role assumption	CloudTrail AssumeRole event
Gradual production traffic	Goal	Blue/green or canary traffic shifting	Deployment configuration
Automatic rollback	Hard constraint	Alarm is attached to deployment	Failed-health drill
Human approval in phase one	Operating constraint	Approval action before production	Pipeline execution history

For every requirement, answer two questions:

1. Where is it enforced?
2. What evidence proves that it worked?

If you cannot answer both, the requirement is still only an intention.

Phase 2: Draw the architecture

Your diagram must show security and artifact boundaries, not just service names.

At minimum, include:

• tooling, development, and production account boundaries;
• source, pipeline, build, artifact bucket, KMS key, and image repository;
• cross-account deployment roles;
• development and production ECS services;
• test and approval gates;
• production listener or load balancer;
• CloudWatch alarms and the rollback signal;
• CloudTrail as the audit source.

A valid high-level flow could look like this:

Developer commit
      |
      v
Tooling account
  Source -> Build/Test -> ECR image by digest -> Pipeline artifact
      |                                      |
      | STS AssumeRole                       | encrypted with KMS
      v                                      v
Development account                    Evidence and manifest
  Deploy -> integration tests -> promote decision
      |
      | STS AssumeRole + approval
      v
Production account
  Green replacement -> traffic shift -> alarms -> full traffic OR rollback to blue

Annotate every cross-account arrow with the role that is assumed. Annotate every artifact arrow with the artifact identifier and encryption boundary.

Architecture review questions

• Which account owns each resource?
• Which identity performs each API call?
• Which resource policies must trust an external identity?
• Can production pull the exact image digest tested in development?
• Where is deployment health evaluated?
• What happens when the pipeline is interrupted after partial traffic shifting?

Phase 3: Define identities and encryption

Create an account and role matrix. Avoid labels such as "pipeline role has access." State the principal, action, resource, and trust relationship.

Identity	Home account	Assumes or is assumed by	Required actions	Explicitly excluded
Pipeline service role	Tooling	Assumes deployment roles	Read artifacts, invoke stages, call STS	Application data access
Build role	Tooling	CodeBuild service	Read source, run tests, push one ECR image	Production deployment
Development deployment role	Development	Trusted pipeline role	Update development deployment resources	Production resources
Production deployment role	Production	Trusted pipeline role	Start and inspect approved deployment	IAM administration
ECS task role	Workload account	ECS tasks	Application-specific API calls	Pipeline and deployment actions

For each cross-account role, document both halves of authorization:

1. The target role's trust policy allows the tooling pipeline role to call sts:AssumeRole.
2. The tooling pipeline role's permissions policy allows sts:AssumeRole on that target role ARN.

Then document the resource layer:

• S3 bucket policy for pipeline artifacts;
• KMS key policy and the caller's KMS permissions;
• ECR repository policy or replication strategy;
• permissions required to pass deployment and task roles;
• conditions that restrict unexpected principals or contexts.

Encryption checklist

• Artifact bucket blocks public access.
• Artifact objects use a customer managed KMS key where cross-account access is required.
• The KMS key policy names the intended cross-account principals.
• Caller policies allow only the required KMS operations.
• ECR scanning and image tag mutability settings are documented.
• Logs use an explicit retention period.
• Secrets are referenced from a managed store and are not placed in build logs or artifacts.

Phase 4: Specify the delivery flow

Create a stage contract for every stage. A stage contract states what must already be true, what the stage does, what it emits, and what allows the execution to continue.

Stage	Entry criteria	Main work	Output evidence	Exit criteria
Source	Main-branch event received	Resolve commit	Commit ID	Revision recorded
Build	Source available	Compile, unit test, scan, build image	Reports, image digest	Required checks pass
Development deploy	Manifest available	Assume role and deploy digest	Deployment ID	Service stable
Integration test	Development endpoint healthy	Run API tests	Test report	Threshold met
Production approval	All automated evidence present	Reviewer evaluates risk	Approval identity and time	Approved before timeout
Production deploy	Approved digest available	Blue/green traffic shift	Deployment and alarm history	Full traffic and stable alarms

The artifact manifest

The manifest is the chain of custody for the release. Define a machine-readable format containing at least:

{
  "sourceRevision": "<commit-sha>",
  "pipelineExecution": "<execution-id>",
  "imageRepository": "<repository-uri>",
  "imageDigest": "sha256:<digest>",
  "buildProject": "<project-name>",
  "buildNumber": "<build-number>",
  "testReports": ["unit", "security", "integration"],
  "createdAt": "<UTC timestamp>"
}

Do not use a mutable tag such as latest as the release identity. A friendly tag may exist for people, but the deployment must resolve to the tested digest.

Phase 5: Design feedback and rollback

Select a small set of signals that describe both deployment health and user impact. More alarms are not automatically safer; noisy alarms make automatic rollback unreliable.

Recommended signals include:

• target health and task startup failure;
• HTTP 5xx error rate;
• latency at a relevant percentile;
• application error count;
• a business signal such as successful order creation;
• deployment and rollback duration.

For each signal, define the data source, evaluation period, threshold, missing-data behavior, and owner.

Rollback decision tree

Your design must answer this sequence:

Did the new task set become healthy?
  no  -> stop deployment and rollback
  yes -> shift the first traffic increment

Did a deployment alarm enter ALARM?
  yes -> stop traffic shift and rollback
  no  -> continue according to the deployment configuration

Did full traffic remain healthy for the bake period?
  no  -> rollback while the previous task set is available
  yes -> complete deployment and retain evidence

Also document the case that automation cannot resolve. For example, a broken database migration may require forward recovery rather than simply returning application traffic to the previous task set.

Phase 6: Run three tabletop drills

Walk through each failure from first symptom to final evidence. Do not stop at "check IAM" or "inspect logs."

Drill A: Development deployment returns AccessDenied

Trace the request in this order:

1. Confirm the pipeline action attempted the expected target role ARN.
2. Confirm the pipeline role permits sts:AssumeRole on that ARN.
3. Confirm the target role trust policy accepts the pipeline principal.
4. Inspect CloudTrail in both relevant accounts.
5. If role assumption succeeded, inspect the temporary session's deployment permissions.
6. Inspect S3, KMS, ECR, and iam:PassRole only where the failed API requires them.

Record the exact failed API, caller ARN, resource ARN, account, and policy layer.

Drill B: Integration tests fail

The production stage must not start. Preserve the failed reports, mark the execution as failed, and leave the currently running development and production revisions identifiable. Decide whether a new commit starts a new execution or whether the failed execution may be retried.

Drill C: Production error rate rises during traffic shifting

The configured alarm enters ALARM. CodeDeploy stops the deployment and returns traffic to the previous task set. Capture the alarm history, deployment ID, image digest, source revision, rollback result, and incident owner.

Explain how you would detect a rollback that also fails.

Phase 7: Measure the delivery system

Include at least these metrics:

Metric	What it reveals	Useful dimension
Lead time for change	Delivery speed	Repository or service
Deployment frequency	Batch size and flow	Environment
Change failure rate	Release quality	Service and deployment type
Mean time to restore	Recovery capability	Incident severity
Stage duration	Bottlenecks	Pipeline stage
Gate rejection rate	Feedback quality	Gate type

Define where each metric is calculated and who reviews it. Metrics without an owner or decision cadence are decoration.

Phase 8: Cost control and cleanup

Your plan must cover both routine cost control and lab cleanup.

Include:

• S3 lifecycle rules for old pipeline artifacts;
• ECR lifecycle rules for unreferenced images;
• log retention appropriate to audit needs;
• build compute size and timeout limits;
• removal of idle development services where acceptable;
• cleanup of test resources after failed executions;
• retention of manifests and audit evidence for the required period;
• deletion order for roles, policies, buckets, keys, repositories, and workload resources.

Do not delete evidence that the organization is required to retain. "Cleanup" is a lifecycle policy, not unconditional deletion.

Required deliverables

Submit one concise design package containing:

1. Requirement ledger with enforcement and evidence.
2. Architecture diagram with accounts, roles, artifact flow, and rollback signal.
3. Account, identity, and role matrix.
4. IAM and encryption design.
5. Stage contracts and artifact manifest schema.
6. Normal, failure, and rollback flows.
7. Metrics, alarms, ownership, and review cadence.
8. Cost-control and cleanup plan.
9. Results of the three tabletop drills.
10. Answers to the project self-check.

Acceptance rubric

Score the project out of 100 points.

Area	Points	Full-credit evidence
Requirements and reasoning	15	Every hard constraint maps to enforcement and evidence
Architecture and boundaries	15	Accounts, trust, artifacts, and failure signals are visible
IAM and encryption	20	Both sides of role assumption and resource policies are explained
Delivery flow	20	One immutable digest moves through measurable stage contracts
Failure and rollback	15	Automatic and manual recovery paths are testable
Operations and metrics	10	Signals have thresholds, owners, and actions
Cost and cleanup	5	Retention and cleanup are explicit and safe

Interpretation:

• 90-100: review-ready professional baseline;
• 75-89: sound design with a few missing operational details;
• 60-74: plausible architecture but important controls remain implicit;
• below 60: revisit the chapter and make requirements, identities, and evidence explicit.

Project self-check

Answer each question before opening the model answers.

1. Why is an ECR tag alone weak evidence that production runs the tested image?
2. What are the two IAM policy decisions required before a tooling role can assume a production role?
3. Why might the artifact bucket and KMS key require separate cross-account permissions?
4. Where should production approval appear, and what evidence should the reviewer inspect?
5. Which signals are suitable for an automatic rollback, and what makes a signal unsafe?
6. What should happen to production when development integration tests fail?
7. Why is the ECS task role different from the production deployment role?
8. What evidence links a production incident to a source change?
9. When can rollback to the previous application revision be insufficient?
10. Which design decision most directly reduces the blast radius of compromised build credentials?

Show model answers

1. Image identity

Tags can be moved or overwritten. A digest identifies the image content. The manifest should connect that digest to the source revision, build, tests, and pipeline execution, and the deployment should use that verified digest.

2. Cross-account role assumption

The production role's trust policy must trust the tooling pipeline principal, and the tooling role's identity policy must permit sts:AssumeRole on the production role ARN. After assumption, the session still needs permissions for the deployment APIs it calls.

3. S3 and KMS authorization

Reading an encrypted object is two authorization operations: access to the S3 object and permission to use the KMS key for decryption. The bucket policy cannot grant use of the key, and the key policy cannot grant access to the object.

4. Approval evidence

Place the approval after all automated pre-production evidence and before any production mutation. The reviewer should see the revision and digest, test and scan results, planned deployment strategy, recent operational health, and rollback readiness. The decision identity and time should be retained.

5. Rollback signals

Signals should be timely, attributable to the deployment, and strongly correlated with user or service health. Error rate, unhealthy targets, and a critical business transaction can work. A noisy metric, an overly short evaluation period, or ambiguous missing-data behavior can cause unsafe rollbacks.

6. Failed integration tests

Production remains unchanged. The pipeline stops before production, preserves failure evidence, and identifies the last known good revision. A correction should pass the same controls rather than bypassing the failed gate.

7. Runtime and deployment roles

The task role authorizes application code while tasks run. The deployment role changes deployment resources and traffic. Combining them gives the application unnecessary control over its own delivery system and increases blast radius.

8. Incident traceability

Use the manifest and execution history to connect the production deployment ID and image digest to the pipeline execution, build, test reports, source commit, approver, and timestamps. CloudTrail supplies API-level caller evidence.

9. Limits of rollback

An incompatible or destructive database migration, externally visible side effect, corrupted data, or irreversible event may survive an application rollback. Such changes need backward-compatible migration, forward recovery, or a separate restoration procedure.

10. Build credential blast radius

Give the build role only build-time permissions and prevent it from deploying production or administering IAM. Let a separate pipeline identity assume tightly scoped deployment roles only at the required stages.

Completion checklist

• Every requirement has enforcement and evidence.
• The same image digest is used in development and production.
• Every cross-account call names the caller and target role.
• S3, KMS, ECR, and iam:PassRole permissions are addressed.
• Pipeline stages have measurable entry and exit criteria.
• Production alarms drive a tested rollback path.
• Normal, failure, and recovery flows are documented.
• Metrics have owners and review actions.
• Cost and retention rules are explicit.
• All ten self-check answers can be explained without memorized slogans.

Further reading

• AWS DevOps Engineer Professional exam guide: Domain 1
• CodePipeline pipeline structure reference
• Create a pipeline with cross-account actions
• IAM cross-account resource access
• Operational Excellence: frequent, small, reversible changes