Stage 2: Three environments and a pipeline that guards production
The product has users now, so changes need a path to production that cannot skip a check. Dev, staging and prod on AWS, a tag-only release pipeline with no static keys, and the monitoring that caught a $5,000-a-month mistake before the invoice did.
$111.52/mo saved- Role
- Team of three; I owned production, alerts and security
- Scale
- 3 environments, 1 VPC
- Stack
- CI/CD pipelines, AWS OIDC, SSM
- Targets
- Tag-only prod deploys, no static AWS keys, CIS Level 1
Context
Stage 1 got the product online. Stage 2 is what a startup faces once people depend on it and the team grows: more than one engineer changing things, a staging environment that has to match production, and an audit question about who can touch prod.
The team was three engineers. I owned production, alerting and security: the release pipeline's production path, the monitoring stack, and the CIS hardening of the hosts.
Constraints:
- Three people, three environments, one VPC
- No long-lived AWS keys in CI
- Hosts hardened to CIS Level 1
Architecture and failover
Dev, staging and prod each have a web tier and a data tier inside one VPC. Production data moved to RDS, with credentials in Secrets Manager. Internal tools are reachable only over VPN.
The CI pipeline runs on every pull request: lint, tests, a Go vulnerability check, gitleaks for secrets, and a build. Merging to main must pass a blocking SonarQube gate. Production only deploys from a version tag. The pipeline assumes an AWS role through OIDC and runs the deploy on the servers through SSM, so nothing in CI holds a key that works on its own. Release, rollback and database restore each have their own pipeline.
Grafana Alloy on each host collects metrics, logs and traces. After the cardinality incident, the team moved from a hosted backend to a self-hosted Mimir, Loki and Tempo stack, with the label leak fixed in Alloy.
- 1GatePull request checks
Lint, tests, Go vulnerability check, gitleaks secret scan, build.
If it fails: The pull request cannot merge.
- 2Merge to main
- 3GateSonarQube quality gate
If it fails: Main is blocked until the code is fixed.
- 4Push a version tag
Only a v*.*.* tag starts the production pipeline.
- 5Assume an AWS role through OIDC
Short-lived credentials for this run only.
- 6DeployDeploy through SSM
Commands run on the servers; no SSH, no open port 22.
- 7GateHealth checks
If it fails: Run the rollback pipeline to the previous tag.
Try it: replay the bot scan, or ship a tagged release. Times are approximate.
- Primary / serving
- Recovering
- Degraded
- Down
- Replication / control
Pick a failure to see how the system detects and recovers from it.
Key decisions
OIDC and SSM instead of SSH keys in CI
The first pipeline deployed over SSH through a jump host, with long-lived keys stored in CI.
Trade-off: More IAM to set up, and debugging a deploy means reading SSM command output instead of a shell session.
Production changes only from a version tag
Merging to main and releasing to production should be two separate decisions.
Trade-off: Shipping takes one more deliberate step, and hotfixes need a tag too.
Self-hosted monitoring with cardinality limits
Hosted monitoring billed per active series, and one bot scan multiplied the series count by ten.
Trade-off: Running the monitoring stack is now the team's job, including its storage and upgrades.
Lifecycle
- Open a pull request. CI lints, tests, scans dependencies and secrets, and builds.
- Merge to main once the SonarQube gate passes.
- Push a version tag. The production pipeline assumes its role through OIDC and deploys through SSM.
- If health checks fail, run the rollback pipeline.
- Alert on active series count, not just on cost.
- Drop high-cardinality labels in Alloy and map unknown routes to a single value.
- Cap samples and labels per scrape, so Alloy cuts a leak off at the source.
- Apply CIS Level 1 with Ubuntu Security Guide.
- Check the result in Security Hub.
- Record any control that is deliberately left open, with the reason.
Failure modes
| Failure | Blast radius | Detection | Recovery |
|---|---|---|---|
| Metric cardinality explosion | Monitoring cost, not availability | Series-count alert | Relabel, scrape limits |
| Bad release | Production | Health checks after deploy | Rollback pipeline |
| Leaked CI variable | None for AWS: no static keys exist | Secret scanning | Rotate the affected secret |
| Data loss in prod | Production data | RDS monitoring | Database restore pipeline |
Results
What I'd change
We saw the warning a day before we acted and dismissed it as noise. I would alert on active series count from day one, before any cost forecast moves, and give that alert an owner.
I would treat every new metric label as a schema change that needs review. The URL path should never have become a label, and a review would have caught it before a bot did.
I would also set scrape sample and label limits before the first deploy, not after the first incident. A limit turns a runaway leak into a capped one.
Next stage: the startup's own product, built on infrastructure that never gets edited by hand.