Skip to main contentUkasyah Nata Dewastoro
Cloud

Stage 4: Containers, and images six times smaller

The company is now a household name. Traffic spikes by orders of magnitude when the app goes viral, and the backend has grown into fourteen services, so shipping fourteen AMIs no longer scales. Static Rust binaries on empty base images, cross-compiled without emulation, run by Podman Compose. The backend image went from 137 MB to 23.1 MB.

137 → 23.1 MB
Role
Team project; I built the container images and local stack
Scale
14 service images
Stack
Podman, Compose, Rust
Targets
Small, rootless images that build on a laptop

Context

The product kept growing. The backend split into per-domain services and workers, fourteen binaries in all. Baking a golden image for each, as in stage 3, meant fourteen slow image builds for every release. Containers were the next step, and this stage is about making them small and safe before anything orchestrates them.

Constraints:

  • Images must run without root
  • Builds have to work on a laptop and in CI alike
  • Fourteen services, so per-image size multiplies

Architecture and failover

Each service compiles to a static binary against musl and ships on a scratch base that holds only the binary and CA certificates. It runs as an unprivileged user. cargo-chef caches the dependency layer, so a code-only change rebuilds in a fraction of the time. Images cross-compile for each architecture without emulation.

How one service becomes a 23 MB image
  1. 1
    Cache dependencies

    cargo-chef builds the dependency layer once.

  2. 2
    Cross-compile

    A static musl binary for each target, without emulation.

    If it fails: Builders under 8 GiB run out of memory; move the build.

  3. 3
    Assemble on scratch

    Only the binary and CA certificates, with a non-root user.

  4. 4
    DeployRun with Podman Compose

    Postgres, Redis, NATS, the backend and three workers.

Podman Compose runs the whole stack locally: Postgres, Redis, NATS, the backend and its three workers. The same images later run on Kubernetes, and the node preparation scripts for kubeadm on Ubuntu and Rocky Linux are part of this stage.

Key decisions

Static binaries on an empty base image

  • containers
  • security

The first backend image carried a full OS layer for a program that needs almost none of it.

  • Debian or Alpine baseFamiliar and debuggable, but ships a package manager and a shell nobody uses in production.
  • Static musl binary on scratch, running as a non-root userChosenThe image holds the binary and CA certificates, nothing else.

Trade-off: An empty image has no shell, so debugging needs an ephemeral sidecar, and a container HEALTHCHECK command cannot run inside it.

Lifecycle

  1. cargo-chef builds and caches the dependency layer.
  2. The service compiles statically for the target architecture.
  3. The build copies the binary and CA certificates onto scratch and sets a non-root user.

Failure modes

FailureBlast radiusDetectionRecovery
Builder out of memoryBuild failsrustc killedBuilders with 8 to 10 GiB
Missing CA certificatesEvery outbound TLS call failsErrors at first requestCA bundle copied into every image
Debugging a running containerNo shell insideEngineer needs oneEphemeral debug sidecar

Results

137 → 23.1 MBbackend image, about 5.9 times smaller
3.6–23.1 MBrange across all 14 images
0shells or package managers in production images

What I'd change

I would remove the logging library that pulled in a Kafka client and its C dependencies before writing a single Containerfile. It caused most of the cross-compilation trouble, and nothing in the product used Kafka.

I would commit the container work as I built it. Mine stayed uncommitted while the team moved on, so the images and the measurements lived on one laptop.

I would add a health check that works without a shell, such as a small endpoint the orchestrator probes, before any of these images reach Kubernetes.

Next stage: the containers move onto Kubernetes, managed through GitOps.