Stage 3: An immutable AWS platform for the product
The startup has raised a Series A, the team is doubling, and the CTO wants every environment reproducible from code instead of console clicks. Three environments in Terraform behind Atlantis, CIS-hardened golden AMIs scored before release, app versions pinned in SSM, and fleets that switch off outside office hours.
0 hand-edited servers- Role
- Team of four; I built the network, VPN, Terraform workflow and delivery tooling
- Scale
- 3 environments, 3 AZs, 8 Auto Scaling groups
- Stack
- Terraform, Atlantis, Packer
- Targets
- No hand-edited servers, CIS score gate, off-hours shutdown
Context
By this stage the startup has funding and a growing team, and clicking through the cloud console no longer scales. The product is a backend with background workers, a web client and a mobile app, all needing a home that behaves the same in dev, staging and production. The application itself is not the subject of this page; the platform under it is.
The team was four engineers. I built the network foundation (VPC, NAT and the OpenVPN gateway), the Terraform workflow on Atlantis, and the delivery tooling. In the revision I added a self-hosted Git and Jenkins host when the hosted CI hit its workload limits, the web deploy pipeline, and the office-hours scheduler.
Constraints:
- A restricted IAM role that could not create IAM roles or large instances
- Three environments that must stay identical in shape
- Every change reviewable before it reaches AWS
Architecture and failover
Each environment gets its own subnets across three availability zones, an Application Load Balancer, Auto Scaling groups for the web, API and worker tiers, RDS Postgres and a Valkey cache. Production adds SQS. A shared-services zone, reachable only over VPN, runs the CI runner, Atlantis, the monitoring stack and SonarQube behind an internal certificate authority.
App servers are never configured after boot. Packer bakes a CIS Level 1 golden image per role, the pipeline scores and scans it, and SSM Parameter Store holds the image ID and the app version each fleet should run. A deploy uploads the artifact to S3, moves the SSM pointer, and restarts the instances through SSM. Long-lived tooling hosts, the pets, are the only machines Ansible touches.
Losing an instance is routine: the Auto Scaling group replaces it with an identical one. Losing the load balancer's targets is not, which is what the incident above taught.
- 1Build the CIS Level 1 base image
Packer, then a role image (API, web, tooling) on top.
- 2Scan
OpenSCAP scores CIS compliance; Trivy checks packages; goss checks the running image.
- 3GateScore gate
A CIS score under 80 fails the build.
If it fails: The image is never published. Fix the build and rerun.
- 4Publish to SSM
The image ID goes to a parameter per role; the last three are kept.
- 5GateManual promote
A person decides when fleets move to the new image.
- 6DeployFleets roll to the new image
Auto Scaling groups replace instances one at a time.
- 1GateQuality checks and SonarQube gate
If it fails: Nothing is uploaded.
- 2Upload the artifact to S3
- 3Move the SSM version pointer
CI writes the new version for the role.
- 4DeployRestart instances through SSM
Each instance pulls the pinned version.
- 5GateHealth checks
If it fails: CI writes the previous version back to the pointer and restarts.
Try it: replay the port change, or roll out a release through the SSM pointer.
- Primary / serving
- Recovering
- Degraded
- Down
- Replication / control
Pick a failure to see how the system detects and recovers from it.
Key decisions
Golden images for app fleets, Ansible only for tooling
Configuring app servers after boot lets them drift apart, and a replaced instance may not match the rest.
Trade-off: Every OS or dependency change means a new image build, which is slower than editing a running host.
A scored, gated image pipeline
A hardened image only helps if we measure the hardening and stop a bad build from reaching a fleet.
Trade-off: Images take longer to publish, and we documented some CIS controls as deliberate exceptions.
Infrastructure changes through Atlantis on pull requests
With four people applying Terraform from laptops, state conflicts and nobody reviews the changes.
Trade-off: Every infrastructure change waits for a plan and a review, including small ones.
One repo per tool instead of one monorepo
The first version kept Terraform, Ansible, Packer and the apps in one repository with one pipeline.
Trade-off: More repositories to keep in step, and changes that span tools need two pull requests.
Environments that sleep outside office hours
A startup pays for dev and staging around the clock, but people only use them in working hours.
Trade-off: Anyone working late or early starts with a cold environment.
Lifecycle
- Open a pull request on the Terraform repo.
- Atlantis posts the plan on the pull request and locks the state.
- A reviewer approves, then Atlantis applies.
- Terraform creates anything a listener depends on before it destroys the old copy.
- Packer builds the CIS Level 1 base image, then a role image on top.
- OpenSCAP scores it; Trivy and goss check it. A score under 80 fails the build.
- The pipeline publishes the image ID to SSM and keeps the last three images per role.
- Fleets move to it only on a manual promote. A full rebuild runs monthly for OS patches.
- CI runs quality checks and the SonarQube gate.
- The artifact goes to S3, and CI moves the SSM version pointer.
- Instances restart through SSM and pull the pinned version.
- Health checks failing triggers an automatic rollback to the previous pointer.
Failure modes
| Failure | Blast radius | Detection | Recovery |
|---|---|---|---|
| Instance lost | None | Auto Scaling health check | Automatic replacement from the golden image |
| Target groups replaced in the wrong order | Every route returns 503 | Load balancer health checks | Port in the name, create_before_destroy |
| Image fails the CIS score | None: it is never published | Pipeline gate | Fix the build and rerun |
| Bad release | One fleet | Health checks after deploy | Automatic rollback of the SSM pointer |
| Jenkins crash-looping while Ansible reports success | CI unavailable | Service status, not the playbook result | Fix the unit and verify the service, not the task |
Results
What I'd change
I would review every Terraform plan for the word "replace" before applying it. The port change that took all four target groups offline showed a replacement in its plan, and nobody read it closely. Naming target groups by port with create_before_destroy now makes that change safe.
I would put integration tests in the pipeline from the start. The build hosts had no containers, so the backend's integration tests stayed switched off and the gate checked less than it appeared to.
I would make Ansible verify the service itself instead of trusting its own task results. Jenkins crash-looped twenty times while the playbook reported success.
I would give each tool its own repository from the first week. We split the monorepo in the revision, and every split cost a migration we could have avoided.
Next stage: the product moves into containers, and the images shrink by almost six times.