Deploying Is Three Things, Not One

Modern hosting platforms have made deployment feel like a single event. You push, something goes green, the change is live. On most managed platforms that is three separate things wearing one coat, and each can succeed while the next has not happened.

The three stages

• Build: your code compiles into an artefact. Success here means it compiled, and nothing more
• Rollout: a release referencing that artefact is created and applied. This can sit queued for a while
• Traffic: requests actually move onto the new release. Until this happens, visitors get the previous one

We watched a build report success while its rollout was still queued and every visitor was being served the previous version. Nothing was broken. Nothing said anything was wrong. The site was simply running old code while three separate status screens showed green.

The only check that cannot lie

Ask the live site what it is running. We expose a small endpoint that reports the running revision, and our deploy script resolves that back to the commit it was built from. It either matches the commit you just pushed or it does not. No interpretation required.

Before that, we tried checking for a marker in the page HTML, which broke twice: once because the page was cached and once because the element gained an attribute that changed how it was rendered. Assert on something that exists only in the version under test, and that nothing can cache.

Why this is worth ten minutes

The failure mode is expensive out of all proportion to the fix. You deploy, you test, the bug is still there, and you spend an hour investigating code that was never running. Every time we have skipped this check we have paid for it later, usually while under pressure.