Make Your Continuous Integration Server STRIKE FIRST and STRIKE HARD..

If automated testing is a part of your process, you know how much it hurts when tests fail on your master branch. Every red test on master makes it more likely that further breakages will sneak in, for several reasons:

• If master has failures, when you’re starting to review a feature branch, rather then relying on color recognition (red versus green), you need to use brainpower to separate “new failures” from “old failures”. Whenever you need to use your brain, there’s the chance that you might not end up with the right idea.
• Even if you never make mistakes, if the first in a series of assertions fails, this can prevent the rest from even running! In other words, because tests are failing, you’re now testing less than you were.
• Most importantly, good people have a powerful psychological need to preserve beautiful things. A pristine test suite is beautiful, and tends to motivate developers to keep it that way.

How best to keep the master branch green? You could attempt to foster an organizational appreciation that breaking master is a Very Bad Thing, in the hopes that this will make it happen less often. Some folks take this to interesting extremes, including ritual humiliation (however tongue in cheek).

(source: http://www.flickr.com/photos/foca/6935569551/)

While this approach may encourage people to care about the integrity of the tests, it has some limitations:

• Even if the situation induces the proper sort of frowny faces, the build still got broken, and if you want to un-break it quickly, you now have have an emergency on your hands. In general, people are less likely to employ healthy long-term thinking in emergency situations.
• If your code is the result of a team effort (like, you pair program or perform detailed code reviews), who wears the hat?
• Is encouraging people to triple-check everything due to fear of dire consequences really the best tactic we have at our disposal? Fear leads people to focus on tests that pass rather than on tests that are meaningful.

Another approach is to enhance the role of your continuous integration server. At MojoTech, we don’t just monitor the mainline development branch with continuous integration; we run the entire test suite whenever anyone pushes to a remote branch. Therefore, there is no shame in seeing a feature branch break the tests; in fact, it’s expected.

The other side of the coin is the following policy: “Don’t submit a pull request until you can see that the tests pass on your branch!” This makes it really easy to ensure that the test suite is ALWAYS passing on your master branch; nothing gets merged unless you already know that it won’t break tests. (Of course, there’s always the possibility that tests could be passing in both the mainline branch and in the feature branch, but that merging one into the other could cause breakages, but this is a comparatively rare occurrence).

This makes the maintainer’s job easier, in that it eliminates a type of code review which would otherwise prove common and repetitive: “This change breaks some tests. Please adjust.”

One tool that facilitates this approach is Semaphore.

I’ll follow up later with more details about how we make Semaphore work for us.

—Matthew Forsyth