Stage 5: GitOps on Kubernetes, ready for scale
The company is chasing a billion-dollar valuation, and the servers themselves have become the bottleneck. The product runs on Amazon EKS and every change reaches the cluster through git. Argo CD syncs three environments, Argo Rollouts releases the API as a canary with a manual production gate, and secrets come from AWS through workload identity with no static keys.
Git-only changes- Role
- Team project; I restructured the GitOps layout and fixed the canary
- Scale
- 3 environments, 9 Argo CD applications
- Stack
- Amazon EKS, Argo CD, Argo Rollouts
- Targets
- Git as the source of truth, manual prod gate, no static keys
Context
The product the startup began with on two servers now runs on Amazon EKS, and nobody runs kubectl apply against production. Git is the only way in.
Stages 1 to 4 kept costs down and proved the product could run. This stage is the infrastructure a company needs once several teams ship at once: capacity that appears in seconds, services that restart themselves, releases that cannot take down the whole platform, and an audit trail for every change.
The cluster was a team project. My part was restructuring the GitOps layout to one namespace per environment, adding the web and worker Helm charts, fixing the canary release, and security fixes in the backend: rate limiting, a refresh-token gap and CORS. Before that, I built a separate lab cluster to trace exactly how internet traffic reaches a Service on EKS.
Constraints:
- A small cluster shared by three environments
- No static AWS keys in the cluster
- Production changes need a human approval
Architecture and failover
Traffic enters through an Application Load Balancer that the AWS Load Balancer Controller creates from Kubernetes Ingress objects, with TLS from ACM and DNS in Route 53. All environments share one load balancer through an ingress group, instead of paying for one each.
Argo CD watches git. One app-of-apps creates nine applications: the API, web and workers, once each for dev, staging and production, each in its own namespace. Dev and staging sync automatically. Production syncs only by hand, and the API rolls out through Argo Rollouts as a canary that pauses before taking all traffic.
External Secrets reads credentials from AWS Secrets Manager through IAM roles for service accounts and EKS Pod Identity, so no pod or pipeline holds a static AWS key.
- 1Merge to git
A new image tag or Helm values change.
- 2DeployArgo CD syncs dev and staging
Automatic, each in its own namespace.
- 3GateManual production sync
A person starts the production rollout.
- 4Canary starts and pauses
Argo Rollouts runs the new API version beside the old one.
If it fails: Abort the rollout, or revert the commit and Argo CD restores the old state.
- 5GatePromote
An engineer promotes; the new version takes over.
Try it: ship a release through the canary, or replay the pods stuck in Pending.
- Primary / serving
- Recovering
- Degraded
- Down
- Replication / control
Pick a failure to see how the system detects and recovers from it.
Key decisions
One namespace per environment
Dev, staging and production shared a namespace, so a mistake in one could touch the others.
Trade-off: Three copies of every resource, and the cluster has to hold all three.
Automatic for dev and staging, manual for production
Every merge should reach the test environments quickly, but production needs a person to say yes.
Trade-off: Production releases wait for someone to be available.
External Secrets with workload identity
Pods need database and API credentials, and copying them into Kubernetes secrets by hand drifts and leaks.
Trade-off: Another controller to run, and a secret change in AWS takes a refresh interval to arrive.
Lifecycle
- Merge a change to the image tag or the Helm values in git.
- Argo CD syncs dev and staging automatically.
- Sync production by hand. The API starts as a canary and pauses.
- Promote the canary once it is healthy. Roll back by reverting the commit.
- Measure real CPU and memory use under load.
- Set requests from the measurement, not a guess, and leave room for a rolling update.
- Alert on pods that stay Pending, not only on pods that crash.
- Route 53 resolves the name to the load balancer.
- The load balancer terminates TLS with an ACM certificate and redirects HTTP to HTTPS.
- It sends traffic straight to pod IPs, registered by the AWS Load Balancer Controller.
Failure modes
| Failure | Blast radius | Detection | Recovery |
|---|---|---|---|
| Pods stuck Pending | New pods never start | Pending-pod alert | Right-size requests from measured use |
| Bad release | Canary traffic only | Rollout paused before full traffic | Abort the rollout or revert in git |
| Volume in the wrong zone after scale-down | Stateful pod cannot start | Pod events | Snapshot the volume and restore it in the right zone |
| Change made by hand in the cluster | Drift from git | Argo CD out-of-sync status | Argo CD reverts it to match git |
Results
What I'd change
I would pin production to tagged releases. Every application still tracks the main branch, so production follows main through a manual sync instead of a version someone chose.
I would give the canary real traffic routing and a metrics check. The rollout declares a canary but the load balancer still sends everything to the stable service, so the pause is a manual gate, not an analysis.
I would split secrets per environment. Dev, staging and production read the same secret today, which means dev can read production credentials.
I would alert on pods stuck in Pending from day one. Ours waited 20 hours because nothing crashed, so nothing paged.
Next stage: the same discipline, applied to a live production estate that was never built as code.