Healthchecks

A healthcheck is a command that runs inside a container and says whether the app works. It passes when the command exits 0. Yard’s deploys lean on it more than on anything else, so every app deserves one. This chapter explains what it does, what it does not do, and how to write one that does not slow your deploys.

What happens during a swap#

First, the honest picture of a deploy, because the healthcheck only makes sense against it.

Yard does not run the new version next to the old one. Compose stops the old notes-app-1, then starts the new notes-app-1 under the same name. Caddy sends traffic to that name. So the new container gets real visitors as soon as it listens, before any check has run. During the gap between stop and listen, Caddy holds each request for up to ten seconds, so a visitor waits instead of seeing an error.

The healthcheck does not keep a bad version away from visitors. Its job is the decision after the swap: within two minutes, is the new version good, or does the old one come back?

With a healthcheck, and without#

With one, the deploy waits until the new container reports healthy. If it never does, the deploy starts the previous image again and ends as reverted. A broken app is live for at most those two minutes, and then the old version is back without anyone touching anything.

Without one, “healthy” means “running”. A container that starts and runs, but answers nothing, counts as a success. It stays live until you notice and push a fix. yard status shows it as up, and the browser shows Caddy’s 502.

A build that fails is a different case. It never starts a container, so the old version keeps running, with or without a healthcheck. The healthcheck catches a broken app, not a broken build.

A first deploy has no previous version to go back to, so a first deploy that never becomes healthy ends as failed. A compose URL app goes back to its previous compose file.

Write one#

The simplest place is the Containerfile:

HEALTHCHECK --interval=10s --timeout=3s --start-period=30s \

  CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/')"

The command runs inside the container, so it must use a tool the image has. The line above uses Python, which every Python image has. An image with curl can use curl -fsS http://127.0.0.1:8000/. Point it at a page that is cheap to serve and that fails when the app is broken.

The Containerfiles that Yard writes have no healthcheck, because Yard cannot know where your app answers. Take the file into your repo, add the line, and pin it:

yard app cat containerfile > Containerfile

# add the HEALTHCHECK line, then:

git add Containerfile && git commit -m "Add a healthcheck"

yard app set --runs Containerfile

git push

In a compose file, the same check goes on the service:

services:

  app:

    # …

    healthcheck:

      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/')"]

      interval: 10s

      timeout: 3s

      start_period: 30s

The interval decides the speed#

The interval sets how fast a good deploy goes live, not only how often Podman checks a running app. Here is why.

Podman runs the first check the moment the container starts, usually before the app answers. That check fails. The next one runs one interval later. Yard waits at most two minutes for healthy. So:

An interval of 10 seconds makes a good deploy go live in about 10 seconds.
An interval of 5 minutes makes every deploy time out after two minutes and revert, even a good one.

Compose has start_interval for exactly this, but Podman 5 ignores it.

There is a second way: a check that waits for the app itself. Yard’s own container does this. Its check runs curl with --retry 60 --retry-delay 1 --retry-connrefused --retry-all-errors, a timeout of 75 seconds, and an interval of one hour. The first check keeps trying once a second, so the deploy ends the moment the app answers. After that, one check an hour is enough. Podman splits each argument of a CMD check at spaces, so give each flag and value as its own list item, with no spaces inside one.

Yard checks your checks#

yard status --full has one healthcheck row per app:

warn when the app has no healthcheck.
warn when the interval is over a minute and the timeout is 30 seconds or less. That check cannot wait for the app, so every deploy would revert.
FAIL when the app is unhealthy right now.

The image format#

Podman can build an image in two formats. The OCI format has no field for HEALTHCHECK and drops the line without a word. Yard builds in the Docker format, which keeps it. If you build the same Containerfile with plain podman build somewhere else, add --format docker there too.