How to roll back a bad desktop app release

Why is rolling back a desktop app harder than a website?

A website rollback changes one place: the server. A desktop release lives on thousands of PCs, each of which already downloaded the bad version and only changes when its updater runs. You cannot reach in and swap files. You can only change what the update server offers and wait for each copy to ask again.

That has three consequences. Rollback takes as long as your slowest check interval. It depends on the bad version’s updater still working. And it cannot undo what the bad version did to local data, such as a migrated settings file or a new database format.

How do you design releases so they can be rolled back?

Make rollback a pointer change. Every upload becomes an immutable build with its own number, a channel points at one build, and releasing or rolling back only moves that pointer. If you ever edit files of a published build in place, you lose the copy you would roll back to.

• Immutable builds. Never overwrite a release. A fix is a new build.
• Channel pointers. Users follow a channel such as production; the channel names the build. Rolling back is pointing the channel at an older build.
• Keep the last good builds. Retention policies are useful, but make sure the build you would return to is still there.
• The client accepts a lower version. If the updater only installs "newer" versions, the server cannot send anyone back. The server should decide which build belongs on the channel.
• Data stays readable by the previous version. Avoid one-way migrations in a release, or ship them one release after the code that can read both formats.
• Stage releases. Ship to a beta channel first, so fewer people meet a bad build. See beta channels for desktop apps.

How do you know a release is bad?

The signals usually come from several places at once: crash reports, a spike in support messages, failed updates, or users stuck on a start-up error. Decide the rollback threshold before you need it, so nobody spends an hour debating whether a crash rate is bad enough.

• Crash or error reports that start right after the release.
• A rise in failed installs or updates in your analytics.
• Support messages that mention the new version number.
• Your own smoke test on a clean PC failing after the release went out.

When in doubt, roll back first and investigate second. A rollback that turns out to be unnecessary costs one more update for users. A bad build left live costs a lot more.

How to roll back a release in Omnihorn

In Omnihorn every upload is an immutable build, and a channel points at exactly one build. A rollback is choosing an older build for the channel. Nothing is re-uploaded, re-signed or recompiled, and the installer does not change.

1. 1

   Find the last good build

   Open your app’s Builds tab and note the build that was live before the bad one: its number and version, such as 1.4.2+build.41. It must still be active.

2. 2

   Open the channel

   Go to the Channels tab and open the channel that shipped the bad build, usually production.

3. 3

   Assign the older build

   Use Assign build and pick the last good build. Save. From now on, update checks on this channel get that build.

4. 4

   Fix the other channels too

   If the bad build is also live on beta or another channel, decide for each one whether to move it back or leave it for testing the fix.

5. 5

   Stop automatic promotion

   If the channel follows a branch from CI, the next successful build on that branch moves the channel forward again. Hold pushes to that branch until the fix is ready. See CI/CD.

6. 6

   Ship the fix as a new build

   Upload the fixed version as a new build, test it on beta, then assign it to production.

Pin the builds you may want to return to, so automatic retention never removes them. The channel model is described in Channels and rollouts, and a team process around it in Team release process.

What happens on users’ PCs after a rollback?

Nothing happens until each installed copy checks for updates. When it does, the server answers with the older build. The Omnihorn updater treats any version different from the installed one as an update, so the older build is applied like any other release.

• Users still on the good build see no change: the channel offers what they already have.
• Users who installed the bad build get the older build on their next check. Only files whose hashes differ are downloaded, and files the bad build added that the good one lacks are removed after the apply.
• Users mid-download when you switch may see that update stop with a message that the update changed while downloading; their next check picks up the older build.
• New installs download the build the channel points at, so they get the good build straight away. The installer does not need rebuilding.

Timing follows the update loop. Opening the updater window checks immediately. The background check runs once per Windows start-up, limited by the channel’s update-check interval, 20 hours by default. Background updates are applied silently, so users meet the older version the next time they start the app. See Updates on user machines.

What a rollback cannot fix

• Copies that do not check. A PC that stays off, or a channel set to manual checks only, keeps the bad build until someone checks.
• Copies whose updater cannot run. The Omnihorn updater is a separate program from your app, so a crash in your app does not stop it. A PC that cannot reach the update server, though, stays on the bad build until it can.
• Local data changes. Files your app wrote, such as a migrated settings or database format, are not part of the build and are not rolled back.
• All-users installs. Where the all-users install mode is enabled, its privileged updater refuses a build number lower than the installed one, as downgrade protection. For those machines, fix forward.

Postmortem checklist after a rollback

1. Write down the timeline: release, first signal, rollback, fix.
2. Record which channels and build numbers were involved.
3. Check how many copies took the bad build and how many have moved back, using your analytics.
4. Find why testing missed it: no beta stage, beta too short, or a case only real machines hit. See QA testing.
5. Check whether the bad build changed local data that the older version cannot read.
6. Decide whether the check interval is right for how fast you need fixes to land.
7. Add the failing case to your release checklist before the next release.