A deployment just failed in production. Users are seeing errors, alerts are firing, and someone on your team needs to make a call, fast. Do you revert to the last known good version, or push a fix forward as quickly as possible?
This decision, rollback versus roll-forward, is one of the clearest signals of how mature an engineering team's incident response actually is. Teams that have thought this through in advance recover in minutes. Teams that haven't spent that time arguing in a Slack channel while the outage continues.
This post breaks down both strategies, when each one makes sense, and how to decide in advance so you're not deciding under pressure.
What Rollback Means
Rollback means reverting the system to the last known good state, undoing the deployment that caused the problem and restoring the previous version that was working correctly.
How Rollback Works in Practice
In a well-designed pipeline, rollback can be close to instant: redeploying a previous container image, switching traffic back to an old version in a blue-green setup, or reverting a database migration alongside the code. In a poorly designed pipeline, rollback might mean manually reconstructing a previous state from scattered configuration changes, which can take longer than fixing the bug itself.
When Rollback Is the Right Call
-
The cause is unclear: If you don't know exactly what broke, rolling back removes the unknown variable immediately
-
The blast radius is large: If the issue affects core functionality or many users, restoring stability quickly matters more than understanding the root cause right away
-
A fix would take longer than a revert: If the previous version was stable and the new one isn't, going back is almost always faster than going forward correctly
What Roll-Forward Means
Roll-forward means pushing a new fix through the pipeline to correct the issue, rather than undoing the change that caused it.
How Roll-Forward Works in Practice
This typically involves identifying the specific bug, writing a targeted fix, and deploying it through the same pipeline as any other change, ideally with an expedited but still tested path. The goal is to move forward to a new stable state rather than backward to an old one.
When Roll-Forward Is the Right Call
-
The cause is well understood: If the team can pinpoint the exact issue and the fix is small and low-risk
-
Rolling back isn't simple: Some changes, especially database migrations or data transformations, can't be cleanly reversed without risking data loss or corruption
-
The previous version isn't actually viable: If the old version had its own known issues, going back may just trade one problem for another
The Database Problem
This is where the rollback versus roll-forward debate gets complicated for most teams.
Reverting application code is usually straightforward. Reverting a database schema change, especially one that's already been applied to production data, is a different story. If a deployment included a migration that altered table structures, reverting the code without reverting the database can leave the application and database out of sync, sometimes making things worse than the original failure.
This is why many mature teams design migrations to be backward-compatible by default, so that even if the application code is rolled back, the database schema still works with the previous version. This single practice removes a huge amount of risk from the rollback decision.
Why "It Depends" Isn't a Good Enough Answer
Every engineering team will tell you "it depends" when asked which strategy they prefer. That's technically true, but it's also a non-answer if it's the first time anyone has thought about it during an active incident.
Mature teams don't decide rollback versus roll-forward during the incident. They decide it in advance, as part of their deployment and incident response design:
-
Rollback is the default for most teams, because it's faster to reason about under pressure and doesn't require diagnosing the issue first
-
Roll-forward is the exception, reserved for cases where rollback is technically difficult or the fix is trivial and well-understood
Having this default agreed upon ahead of time means the first five minutes of an incident go toward executing a known plan, not debating strategy.
What This Looks Like in a Mature Pipeline
Teams that handle this well tend to share a few characteristics:
-
Rollback is a one-command operation, not a manual reconstruction process
-
Database migrations are designed to be reversible or backward-compatible as a standing engineering practice, not an afterthought
-
Feature flags decouple deployment from release, so a bad feature can be turned off instantly without any rollback at all
-
Post-incident reviews examine not just what broke, but whether the rollback or roll-forward decision was made quickly and correctly
This last point matters. The goal isn't just to recover from this incident, it's to make the next one faster to resolve, regardless of which path is taken.
A Quick Decision Framework
When a deployment fails, ask these questions in order:
-
Can we roll back in under five minutes without data risk? If yes, do it.
-
Is the cause already known with a fix ready? If yes and rollback is risky or slow, roll forward.
-
Are we unsure on both counts? Roll back. Buying time to investigate safely is almost always worth more than a fast but uncertain fix.
This framework won't cover every edge case, but it removes the paralysis that turns a five-minute outage into a thirty-minute one.
Where to Go From Here
Rollback versus roll-forward isn't really a technical debate, it's a planning debate that happens to surface during technical incidents. Teams that decide this in calm moments recover faster in chaotic ones.
