"latest" is not a version: three lessons from deploying our own website with Docker Swarm

This page you are reading published itself. Nobody uploaded files over FTP: someone pushed a commit, a pipeline validated the code, built an image and updated the service on the cluster. The first time we wired it up this way, the whole pipeline went green, the deploy webhook returned a flawless HTTP 200… and the site kept serving exactly what it had before. The deployment "worked" without deploying anything. Out of that afternoon — and a couple more like it — came the three rules we now apply to every application we ship, ours or a customer's.

A pipeline is not there to deploy fast. It is there so that broken things never reach production, and so you can roll back without thinking. Speed is a pleasant side effect; what you are actually buying is control.

How this gets published (three steps, no magic)

The site runs on our own container platform: Docker Swarm with Traefik in front, a private registry, and GitLab CI/CD as the glue. When something lands on the main branch, three things happen, in this order. First lint: php -l across every .php file in the repository, four at a time. Second build: the image is built with Kaniko — no Docker daemon, because the runner that builds it doesn't start one — with the site baked inside: the code is copied into the image, not mounted from the server's disk. Third deploy: the cluster service, managed through Portainer, is updated to use that image.

That is the whole secret behind what people call GitOps: the repository is the single source of truth, and deployment is a consequence of a commit rather than someone's manual chore. If a change isn't in git, it doesn't exist. Nobody logs into the server to edit a file, not least because it would achieve nothing. And you don't need an internal platform with a codename or a Kubernetes cluster to get there.

Lesson 1: "latest" is a name, not a version

Back to the afternoon of the phantom deploy. The new image was built and pushed to the registry tagged :latest. The cluster service also pointed at :latest. We restarted the service and it started… the same old image. Docker's own documentation says it plainly: when you create a service, it is pinned to a specific image digest until you update it with service update --image; only then does the manager ask the registry which digest that tag currently points to.

Put differently: the orchestrator does not chase your tag. It resolved it once, kept the fingerprint, and stays loyal to that fingerprint. Restarting the service, forcing it, or asking it to refresh changes nothing, because from its point of view you never asked for a different image. It is reasonable behaviour — nobody wants a service's replicas running different versions just because one started five minutes later — but it breaks everyone's intuition.

What we do now: every build tags the image twice, with :latest and with the short commit SHA, so that every version exists in the registry under its own name. And the deployment we consider valid is the one that points the service at the SHA tag with service update --image — an explicit command that does force the manager to resolve the digest again — not a plain restart against :latest. With the truth up front: this site's stack file still declares :latest, and pinning each commit's SHA in there is next on the list. The debt exists and we know where it is.

Lesson 2: the linter said yes, the website said no

A linter checks that code is well written. It does not run it. We were reminded of that by a runtime fatal in the blog listing's pagination — an operation between an integer and a string — that sailed through php -l and took down the index page. Cruel detail: the article pages themselves were fine; what broke was the index, which is exactly where readers arrive.

The answer wasn't a test suite nobody would maintain. It was a twenty-line smoke test running inside the pipeline itself: it starts the site with PHP's built-in server, requests four real routes — the listing in all three languages plus the second pagination page — and fails if the response contains Fatal error, Uncaught, Parse error or Unsupported operand — that last one is precisely what catches the integer against the string — or if the HTML never closes </html>. Nothing else. That minimal question — "does the page with the most traffic render end to end?" — is exactly what would have warned us about the index fatal before it reached production.

This ties back to something we wrote a few days ago about who actually maintains your website: having a documented process is not the same as having a working one. The difference between the two is whether something automated dares to say "this is not shipping" even when everyone is in a hurry.

Lesson 3: the "zero downtime" our own file claimed

We tell this one because it makes us look bad, which is precisely why it is useful. Right now, as you read this, this site's deployment file still carries a comment that calmly reads "rolling update (zero downtime)". With a single replica — which is what Swarm gives you if you declare none — that is not true. The Compose specification is explicit: the default update order is stop-first, the old task is stopped before the new one starts. That means a few seconds of errors while the new container comes up.

Making it true takes four things at once: at least two replicas, order: start-first (the new task starts before the old one goes away), a healthcheck that tells the orchestrator when the new one is actually serving — otherwise it assumes readiness the moment the process exists — and failure_action: rollback, because the default is pause: the update simply stalls halfway, waiting for a human to notice.

For a corporate website, a few seconds of flicker mid-morning ruins nothing, and running two replicas for sport isn't free either. For the ERP, the order portal, or the app your people live in eight hours a day, it does matter. The point isn't to display "zero downtime" on a slide: it is to know what each deployment costs you and to have decided it on purpose. A comment in a YAML file is not an SLA.

The best part isn't deploying: it's undoing

With images tagged per commit, rolling back means deploying the previous tag. No reverting code under pressure, no rebuilding anything, no trying to remember which files were touched on Friday. Yesterday's version exists, whole, with its number, and it is still sitting in the registry. That is what turns a deployment into a reversible decision instead of a leap.

Hence the question we ask in IT meetings, and that almost nobody has an answer for. It isn't "how long does it take you to ship a change?". It is: how long does it take you to go back to yesterday's version, and who can do it at eight on a Friday evening? If the answer contains the word "depends" or the name of one specific person, the deployment isn't the problem.

Four shortcuts we don't take

This isn't about our website

We tell our own story because we look at it every day, but the same pattern is what we build for our customers' business applications: the corporate site, the internal portal, the bespoke application that has been running for fifteen years and that nobody dares touch. The specific technology matters less than it seems — what changes daily life is having a single source of truth, an automated gate that can say no, and a way back that doesn't depend on anyone's memory.

Four questions to know where you stand in a minute: could you rebuild your application today from the repository alone, with no stray files on a server? Do you know, without checking, which exact version is serving right now? Is there something automated that prevents publishing a broken page? How long does it take you to roll back? With those four answered, the conversation about orchestrators, containers and acronyms becomes secondary. Without them, no tool will save you — just as the monitoring we run in containers is only worth something if someone reads what it says.

We have been running this container platform in production for a while now, with the same philosophy we apply to the rest of the infrastructure we manage: we prefer boring and reproducible over brilliant and fragile. This post, incidentally, was published through the very pipeline it describes. Had the smoke test failed, you would not be reading it.

Sources (verified): tag and digest behaviour in Swarm services, from Docker's official documentation — "Deploy services to a swarm" ("when you create a service, it is constrained to create tasks using a specific digest of an image until you update the service using service update with the --image flag"). Defaults for update_config (order: stop-first, failure_action: pause) from the Compose deploy specification. Everything else — the pipeline, the pagination fatal, the smoke test and the optimistic YAML comment — is ours, scars included.