Hotfix or Not? Managing a Successful Release Process

It's obvious that good communication is critical to the success of any project. On a recent large project, the many layers of communication needed to coordinate some daily operations have become even more clear to me. In particular, I've become attuned to something that I never realized would be relevant to a PM: release management.

We've been working on a big, complex app for almost a year that has had up to eight front- and back-end developers working on it at once. On top of that, we've been simultaneously working on large features, small features, quick support fixes, and QA. With all that in play, it meant that once this app was in the wild, we would need a rock-solid release management plan.

The Challenges

In landing on a release process, our team first talked through the scenarios that our release process would need to support:

• Ongoing development of large features that would need to stay isolated until completion and approval by the client
• Frequent releases for small and medium tasks that aren’t urgent, but still require a separate QA/approval process by the client
• Critical fixes that need to go live ASAP

It was clear that flexibility needed to be at the core of our process. We didn't want to slow down ongoing development, but needed to be able to release frequently and react quickly to emergency situations.

The Solution

We ultimately landed on a modified version of this successful git branching model to handle our releases.

Main development happens in the "master" branch. The types of changes that go directly in master are low-risk changes that likely won't need any tweaking after QA and we're confident will get a thumbs up from the client. Additionally, only complete work should go in master—a work-in-progress or tasks that still need contributions from another team member shouldn't go in master. Commits in master are automatically deployed to our internal integration environment if they pass automated testing, so the PM can always QA the most up-to-date version of master.

When everything that's new in master has been QAed and we're ready for it to be part of the next release, it's time to split off a new release candidate branch from master. We deploy this release branch to a staging environment, where the client can check it out. Any bugs discovered during this period should be fixed directly in the release branch. Once we're clear to go live, we merge the release branch into production.

If we're developing a new feature that will be worked on over time and won't be ready to launch for a while, or if it's something that will need to be touched by multiple people, that will stay in a separate feature branch. We'll deploy that feature branch to a staging environment for client QA, and it won't go in master—and subsequently into a release candidate—until it is complete and has client sign-off.

When we need to deploy something to production right away, and we can’t wait on the next release candidate to either be complete or approved, we’ll do a "hotfix." A developer will create a hotfix branch, which is created off of the production branch rather than master. Once the hotfix is complete, it gets merged into production as well as master, or the existing release candidate if there is one. Hotfixes allow us to react quickly in stop-the-presses situations.

The Process

With a lot of hands touching this code, it's important to have clear communication and established protocol around this process. Here are a couple things our team has been doing to keep this smooth:

• We've established that all tickets go in master by default. If it shouldn't go in master, the PM should communicate on the ticket that it should remain in a separate features branch.
• If you work on a ticket that will need contribution from another team member, you should share the name of the branch you worked in when you resolve the ticket. Now the PM knows to open a new ticket referencing this branch for the remaining work.
• We follow a tagging convention to name releases. Release branches are named "release-X.Y.Z." "X" is the major version number—we started with 0, and bumped it up to 1 when our app moved from a private beta to a public launch. "Y" increments by one for every new release. The "Z" slot is used when it’s a hotfix.
• Before we deploy a release candidate to staging, a developer posts the commits that are included in that release candidate in Campfire. We use that as a final check that everything on that list has been QAed. It also serves as a good log to refer back to via Campfire's transcripts. Lastly, I use that list to generate the "here's what's in this release candidate" list I'll share with this client.
  • A protip from David: We use the git command git cherry -v 0.7.2 (where 0.7.2 is the last production deploy) to generate this list.

Why Does A PM Need To Care About This?

You may be thinking that this sounds pretty dev-specific. Why would a PM need to care? I'd argue that it's critical for a PM to be fully engaged in this process, especially if you're working on a living, breathing app in a client situation.

For starters, you have insight into the client's urgency that the development team may not have. For example, a copy edit may not seem like a big deal and a developer may think that it can wait to go out with the next release candidate. But you may know the real story: it's actually related to a typo on an ad, and your client has an advertiser who's livid about the mistake. That screams HOTFIX!

You're also likely to have the best judgment when it comes to telling whether the client will want or need to QA something for an extended period of time before approving it -- which would  mean it should stay out of master. You'll know what sorts of changes the client will be sensitive to, even if it's something that may seem like a trivial change to others.

I encourage PMs to get up to speed with their development teams on how to handle releases. Take it upon yourself to understand the process. It can and should impact how you think about tasks and communicate them to your team and the client. You can help make swift and smart decisions that will make releases fluid for your team and efficient for your client.

And the best part? You get to feel like a total badass and say things like “Let’s deploy release 0.4.0 to production,” or even better, “HOTFIX IT!” Really, it doesn’t get much better than that.