Making releasesboring onpurpose

The single most common source of "it worked in staging" is that staging and production were built from the same source at different times, with different dependency resolutions, by different machines.

Build once, on merge, into one immutable artefact with a content-addressed name. Every environment after that receives that exact artefact. Nothing is rebuilt on the way to production, and nothing is built on a developer's machine and uploaded.

Everything that differs between environments is configuration injected at start-up, never baked in. The practical test: you should be able to point at a running production instance and name the commit it came from, without looking anything up.

Four questions, answered before the first merge: how many environments, who owns each, what data lives in each, and what must be true for a change to move from one to the next.

Teams that answer these in week one release weekly for the life of the project. Teams that leave them to whoever owns the infrastructure discover in month six that staging shares a database with another department and can only be refreshed by raising a ticket.

Insist on one environment with production-shaped data — the real volume, the real messiness, anonymised where the law requires. A clean seeded dataset passes every test and tells you nothing. The queries that fall over do so because of a table with nine million rows and an index that made sense when it had nine thousand.

Down migrations are a comforting fiction. In practice, by the time anybody wants one, data has been written in the new shape and reversing the schema either loses it or fails on a constraint.

Write migrations so they are safe to run against the currently deployed code — add the column, backfill it in a separate step, start writing to it, start reading from it, and only then remove the old one. Each of those is its own deploy. It is slower and it means the system is never in a state where the schema and the code disagree.

If something goes wrong, the fix is another forward migration. Design on that basis and the rollback question stops being about the database at all, which is exactly where you want it.

The single highest-value practice on this list: put anything that changes what a user sees behind a flag, so the code can reach production long before the behaviour does.

This decouples two things that are usually fused. Deployment becomes a technical act that happens many times a day and nobody attends. Release becomes a business decision made by whoever is ready to answer the phone about it, at a time that suits them, with no engineer in the loop.

Flags need discipline or they become a second configuration system nobody understands. Ours have an owner, a reason and a removal date written down when they are created, and a flag that outlives its date gets raised in the weekly half hour. The ones that matter are gone within a month of full rollout.

The failure mode of monitoring is alerting on resources. CPU is high, memory is high, disk is filling. None of those tell you whether anybody is having a bad time, and all of them fire during perfectly healthy traffic spikes.

Alert instead on the things users experience: checkout failing above a rate, a queue growing instead of draining, a nightly job that has not finished, a login success rate dropping, a page's server time crossing a threshold. These are fewer, they are harder to set up, and when one fires somebody is actually affected.

The test for an alert is whether a human would do something different in the next ten minutes because of it. If not, it is a dashboard line, not an alert, and treating it as one is how teams learn to ignore the channel.

One request crosses several services, a queue and a scheduled job. Without a single identifier threaded through all of it, debugging is a matter of correlating timestamps across four log streams, which is roughly as pleasant as it sounds.

Generate the identifier at the edge, pass it everywhere including into async work, and log it on every line. Structured logs — fields, not sentences — so they can be filtered rather than grepped.

The payoff is not routine operations; it is the one incident a year where somebody says "a customer says their order vanished at about three", and the answer takes four minutes instead of an afternoon.

Every cutover and every significant release gets a written sequence: the steps in order, who does each, what is verified after each, and an explicit point where somebody says go or no-go against criteria agreed in advance. And then the rollback, written out.

The rollback is almost never used. Writing it is still worth the hour, because it is the thing that lets the team proceed calmly — and because writing it is how you discover the step that cannot be undone, which you would otherwise have discovered live.

All of the above is machinery. The practice that makes it stay true is a standing half hour each week where somebody looks at what actually happened: which alerts fired, which ones were ignored, what the slowest endpoint was, which flags are still open, what broke and why.

It is the least glamorous meeting on the calendar and it is the one that keeps the system honest. Pipelines rot quietly. Alerts drift into noise. Flags accumulate. None of that announces itself; somebody has to go and look.

Do that, and a deploy becomes what it should be — the least interesting thing that happens on a Tuesday.