The image tag problem, in Bicep and in Terraform

This site runs as one container in Azure Container Apps. Until recently every Azure resource behind it was created by hand, from az commands pasted out of a deployment guide, and the pipeline did exactly one thing on each push: swap the image on an app it assumed already existed.

Replacing that with infrastructure as code turned up a trap that has nothing to do with which tool you pick.

The trap

The platform's natural deployment step is:

az containerapp update --name web --image ghcr.io/me/site:sha-c3d4

Infrastructure as code, meanwhile, wants to declare the whole resource — including the image, because the image is part of what the app is. So the tag ends up written in two places, and only one of them is current.

The failure mode is quiet, which is what makes it worth writing down. Nothing breaks at deploy time. The tag in the template is simply stale, so the next time you touch infrastructure for an unrelated reason — bumping a memory limit, adding a probe — the deployment faithfully reverts production to whatever tag the file happens to name. You changed the memory limit; you also rolled back three weeks of releases.

There are three ways out.

Pin the tag in the template and let CI keep swapping the image. This is the smallest diff and the one that feels most natural when the pipeline already works. It is also guaranteed drift: the file is wrong between every pair of deploys, and its wrongness is load-bearing. Rejected.

Tell the tool to ignore the image — ignore_changes in Terraform, or omit the container and patch it afterwards. This trades a silent lie for a visible hole: the single most important field in the resource becomes undeclared, and "infrastructure as code, except the part that changes" is not a description worth defending.

Make the tag a required parameter. Applying the template is the deploy. CI passes sha-<commit>, the file is always accurate, and re-applying is safe by construction. It removes a pipeline step rather than adding one.

The third option is obvious in hindsight, and it is not where most people start, because the platform's own documentation leads with containerapp update. The tooling nudges you into the trap.

The same fix in two tools

In Bicep the parameter has no default, so a deployment cannot silently pick a tag:

@description('Fully qualified image including tag. CI passes ghcr.io/<owner>/<repo>:sha-<commit>.')
@minLength(3)
param image string

The awkwardness is on the CI side. A .bicepparam file does not compose cleanly with inline --parameters overrides, so the tag arrives through the environment:

param image = readEnvironmentVariable('CONTAINER_IMAGE', 'ghcr.io/OWNER/REPO:latest')

That fallback is deliberately invalid. A manual deployment that forgets to export the variable fails on an image pull, which is a good failure — loud, immediate, and impossible to mistake for success. The alternative was defaulting to :latest, which would deploy something and leave you guessing what.

Terraform takes the value directly, which lets the validation be sharper:

variable "image" {
  type = string

  validation {
    condition     = can(regex(":", var.image))
    error_message = "image must include an explicit tag; an untagged reference would silently mean :latest."
  }
}

Bicep cannot express that without a regex over a string parameter. It is a small win, but characteristic: Terraform's input validation is more expressive, and you notice it in exactly these small places.

Keeping both, and what that costs

I wrote both definitions to decide between them, then kept both — a worked comparison against one real deployment is more useful than either alone.

That is only defensible with a rule: exactly one definition is applied. Two things reconciling the same resources would fight, last write wins. Bicep applies, because the state backend Terraform needs must itself be bootstrapped by hand, and a second role assignment on a storage container outside the resource group is real cost for a site this small.

Which leaves the actual problem. The Terraform definition is never applied, so it has no deployment to catch its mistakes. An unapplied definition rots, and a comparison that has quietly stopped being true is worse than no comparison at all.

So parity is enforced rather than trusted. A shell script compares the values a reader would compare — replica counts, CPU and memory, port, log retention and quota, all three probe thresholds, the probe path, and the output names, normalised across camelCase and snake_case — and fails the build on divergence:

  minReplicas default                0
  startup failure threshold          20
  probe path                         /healthz
  output names                       containerappname customdomainverificationid ...

Bicep and Terraform definitions agree.

Fifteen checks, and it does fail: change a probe threshold in one file, rename an output in the other, and the gate reports both and exits non-zero. A gate nobody has watched fail is decoration.

The honest cost is that every infrastructure change now has to be made twice, in two languages, or CI rejects it. That is paid deliberately here, for the explanatory value. On a team shipping features it would be an obviously bad trade, and the right move would be to delete one directory and the gate along with it.

This is the same approach the repository already takes to its layering rules — the build greps for architecture violations rather than trusting a convention. Enforcement is cheap; documentation that drifts is not.

What declaring it actually bought

Not reproducibility, or not mainly. This app has never needed rebuilding from scratch, and if it had, the guide would have worked.

What it bought was seeing the defaults. Writing the resources out longhand surfaced two things that had been invisible.

az containerapp env create quietly provisions a Log Analytics workspace. Nobody chose its retention, and nobody chose its ingestion quota, so there wasn't one — an unbounded billing surface on a project whose first stated constraint is a zero bill. It is now 30 days with a 1 GB/day cap, and those numbers sit in a file somebody can argue with.

The second was worse. An architecture decision record for this site claims "a health endpoint backs the platform probes." The /healthz endpoint exists. No probe had ever been configured, so the platform default applied: a TCP check that passes the moment Kestrel binds the port — before a single post has been parsed. A container that started but could not serve a page would have taken traffic. The document had been wrong since the day it was written, and declaring the infrastructure is what caught it.

That is the argument for infrastructure as code at small scale, and it is not the usual one. Reproducibility barely matters with one environment. Being forced to write down every value, including the ones a CLI was choosing for you, matters a lot.

What is still not code

Three things stayed manual, on purpose.

The resource group and the deploy credential. The service principal is scoped to a single resource group, so a leaked credential cannot reach the rest of the subscription. Declaring the group would require the pipeline to hold subscription-level rights — trading a real security boundary to eliminate one az group create. A credential also cannot create itself.

Custom hostnames and managed certificates. Azure will not issue a certificate until the hostname's DNS records resolve, so a declarative binding must fail on first apply and succeed on a second. Encoding a known-failing first run into the pipeline is worse than the documented manual step it would replace. The templates do the useful half instead: the environment's static IP and the domain verification ID come out as deployment outputs, so the values the DNS setup needs fall out of the deploy rather than needing three more az queries.

"Everything is code" was available both times. It would have been worse both times.