Stage 6: Bringing a hand-built production estate under Terraform
The real thing. A regulated company runs production on cloud infrastructure that was clicked together over years, and nobody can rebuild it. I imported it into Terraform without recreating a single resource, behind a pipeline where no change reaches a cloud project without a review, a gated plan and an explicit apply.
Zero-downtime imports- Role
- Solo engineer; Terraform and the delivery pipeline
- Scale
- 2 cloud projects, ~500 resources
- Stack
- Terraform, Google Cloud, Cloud Run
- Targets
- Import, never recreate; no keys; every apply reviewed
Context
Stages 1 to 5 built platforms from nothing. This stage starts from the opposite problem: a regulated company whose production already runs on cloud infrastructure built by hand in the console over several years. Nothing was written down as code. The written inventory listed about half of what actually existed, and some of the rest was attached to nothing at all.
I had about ten weeks as the only engineer on it, with read-only access to start, no access to personal data, and one hard rule: import what exists, never recreate it. Every write to production needed approval from the company's infrastructure lead.
Constraints:
- Import only: recreating any resource was ruled out
- State must stay in the company's own region and buckets
- No service account keys anywhere
- Every production change approved by the company
Architecture and failover
Each cloud project is its own Terraform root with its own state bucket, versioned and locked down, so staging and production can never share a plan. Shared building blocks live in modules: foundation (network, load balancer), stateful services, serverless apps and data. Resources are declared as map entries, IAM grants are additive only, and every adoption is an import block in code.
The pipeline is the part that keeps production safe. A pull request runs static checks, then a plan per project with a read-only identity. A scripted gate reads each plan before any person does. Nothing applies on merge: an engineer dispatches the apply workflow by hand, and it applies only if the review, the plan fingerprint and an explicit apply comment all line up.
- 1GateStatic checks
terraform fmt and validate, tflint, Trivy config scan against a baseline, gitleaks, actionlint, unit tests for the gate.
If it fails: The pull request cannot merge.
- 2Plan per project
Read-only identity through Workload Identity Federation. It cannot read stored files.
- 3GatePlan gate
Every import covered, no create, delete or replace without a label, sensitive values redacted, plan fingerprinted.
If it fails: The pull request is blocked with the exact resource that failed.
- 4GateReview and merge
A reviewer approves the change and the plan it produced.
- 5GateDispatch the apply workflow
Checks the approval and that the plan fingerprint still matches.
If it fails: The apply is refused; nothing touches the cloud.
- 6GateComment /apply with the run ID
A person releases this exact run.
- 7DeployApply and tag
The apply identity runs only from main, then tags the environment and time.
If it fails: A revert branch is created for the failed run.
A nightly drift check plans every project against the live cloud and opens an issue when someone has changed something outside Terraform.
Try it: merge a change without a review, lose an import silently, or watch the gate stop a replacement.
- Primary / serving
- Recovering
- Degraded
- Down
- Replication / control
Pick a failure to see how the system detects and recovers from it.
Security audit
Terraform makes infrastructure reviewable, but it does not make it safe. Alongside the import, I audited both projects and turned the result into a register the company could work through.
The audit had two lenses. Trivy scans the Terraform code on every pull request, against a baseline of known findings, so nothing new can be merged without someone seeing it. But Trivy only sees what has been imported. For everything else, I ran a scripted, read-only pass over each live project: a dump script exported configuration with a viewer identity, never secret values, stored files or database rows, and an analysis script scored that dump against all 69 controls of the CIS Google Cloud Foundations Benchmark v4.0.
- 1Dump configuration, read-only
A viewer identity exports IAM, network, compute, storage, database and logging settings to a private temporary folder. No secrets, objects or rows.
- 2Score against CIS v4.0
A script checks the dump against all 69 controls and records pass, fail, partial, not applicable or unknown for each.
- 3GateScan the Terraform code
Trivy checks every imported resource; known findings sit in a baseline so new ones fail CI.
- 4Rate and map each finding
Critical, high, medium or low, mapped to CIS, ISO 27001:2022 Annex A and the local data protection law.
- 5Build the register
A generated spreadsheet with an owner, status and fix for every finding, plus a separate register for plaintext settings that should be secrets.
- 6DeployOrder the fixes
Each fix becomes its own reviewed pull request through the same Terraform pipeline.
If it fails: A fix that would break a service waits for an owner decision instead of shipping.
- 7Delete the dump
The temporary folder is removed once the register is built.
Every CIS control ended with a recorded result, including the ones I could not check. Those are listed by name with the reason, so the company knows where the audit stopped instead of assuming it covered everything.
The findings themselves stay with the company. What the page can say is the shape of them: the largest group was over-broad identity and access grants, followed by services reachable from the internet that did not need to be, settings that should have been secrets, and logging that was too thin to investigate an incident.
The audit also changed the import plan. Importing a service before moving its plaintext settings into the secret store would have copied those values into Terraform state, so the settings move came first and those services were deliberately left out of the import until then.
Key decisions
Two lenses on the same estate: the code and the live cloud
A scanner on Terraform code only sees what has been imported, and most risk in a hand-built estate lives in what has not.
Trade-off: The live pass needs its own tooling and a read-only identity, and some checks stayed unknown where that identity or the local tools could not reach.
Import what exists, never recreate it
Production was already serving customers. Recreating anything meant downtime, and the brief ruled it out.
Trade-off: The provider reads some values back differently from how they are written, so early plans proposed changes that were not real and every plan needed a careful read.
A gate that reads the plan before any person does
In an import project, the dangerous plan is the one that quietly creates, deletes or replaces something, and long plans are easy to skim.
Trade-off: A legitimate change needs a label on the pull request, which adds a step to every real update.
GitHub Actions instead of a Terraform server
The pipeline needed reviewed applies, state that stays in the company's own region, and nothing new to host.
Trade-off: The review workflow had to be built by hand, because the repository plan had no branch protection.
Two identities, no keys
CI needs to read the cloud to plan and write to it to apply, and a leaked key would open production.
Trade-off: Two identities to maintain, and the plan identity cannot use the convenient built-in viewer role.
The pipeline's own identities stay outside Terraform
If Terraform manages the identities that run Terraform, one bad apply can remove the pipeline's own access.
Trade-off: A small set of resources is managed by hand and documented separately.
Lifecycle
- Write the resource in code to match what is live, plus an
importblock for it. - Open a pull request. The plan must show an import and zero changes.
- The gate confirms every import block appears in the plan.
- After review, apply. Terraform now owns the resource without having touched it.
- Edit the code and open a pull request with the change label.
- Read the gated plan. A replace needs its own label and a reason.
- Merge, dispatch the apply workflow, and release the run with an
/applycomment.
- The nightly drift check plans each project against the live cloud.
- Any difference opens an issue with the affected resources.
- An engineer decides whether to write the change into code or revert it with the next reviewed apply.
Failure modes
| Failure | Blast radius | Detection | Recovery |
|---|---|---|---|
| Change merged without review | None: the apply refuses it | Approval check in the apply workflow | Revert and resubmit for review |
| Provider drops an import silently | One resource left unmanaged | Import coverage check in the gate | Fix the import address and re-plan |
| Plan proposes a replace | Would interrupt every dependent service | Gate blocks unlabelled replaces | Match the code to what the provider reads back |
| Someone edits the cloud by hand | Code and reality diverge | Nightly drift issue | Write it into code, or revert it |
| Apply fails partway | One project | Workflow failure | Revert branch created for that run |
| Tag step fails after a good apply | Apply succeeded but is not recorded | Workflow failure | Fixed the shell step; tag now runs safely |
Results
Production was enabled in the pipeline, with its imports planned and reviewed, at the end of this stage. The security audit above was delivered as a register, with the critical and high findings ordered to be fixed one reviewed pull request at a time through the same pipeline.
What I'd change
I would count imports from the first day instead of trusting plan output. The provider dropped imports without an error, and I only built the coverage check after I found one missing.
I would agree with the company up front on exactly what Terraform owns. Partway through, the company asked to stop managing some application settings in Terraform, and that cost a round of rework.
I would keep resource counts in one generated report. Three documents ended up disagreeing about how many resources each project had, which undermines a project whose whole point is an accurate inventory.
I would run the live audit before writing any Terraform. It found settings that changed the import order, and finding them earlier would have saved re-planning.
I would test the tag and rollback steps before the first real apply. The first successful apply failed at its tagging step, after the infrastructure had already changed.
That closes the journey: from two servers built by hand, to a live production estate where nothing changes without a reviewed plan.