Engineering
The Bug Hiding Behind a Passing Test Suite
Mike Lewis ·
Ninety-five green tests and a broken feature are not a contradiction. They usually mean the tests are checking the thing next to the thing that matters.
A full test suite passing tells you the code does what the tests say. It tells you nothing about whether the tests describe what the software is for. Those come apart more often than anyone admits, and the gap is where the expensive bugs live.
The pattern
We keep meeting the same failure in different clothes: the test verifies something adjacent to the thing that matters. It checks the permission instead of the write. It checks that a function was called instead of what it produced. It checks the guard logic instead of whether anyone can get through the door.
On a recent build, an enquiry form was emailing every lead correctly and storing none of them. Every test around it passed, because they tested the logic that decides what to store, using a fake database that accepted anything. The real database rejected the whole record over a single optional field that was present but empty. The tests were right. They were just pointed slightly to the left of the problem.
Three questions worth asking about any test
- If this feature broke completely tomorrow, would this test fail? If not, it is documentation, not a test
- Does it exercise the real boundary, or a stand-in that is more forgiving than the real thing?
- Has it ever failed? A test that has never once gone red has never been proven capable of it
Break it on purpose
The cheapest habit we have adopted is deliberately breaking the thing a new test is supposed to catch, watching the test fail, then putting it back. It takes thirty seconds. It has caught several tests that would have passed against absolutely anything, including a configuration check that turned out to be reading a file it had already been handed.
A test you have never seen fail is a hope with a green tick next to it.
And look at the actual thing
The other half of this is unglamorous. Open the page. Sign in as a real user. Submit the form and go and look for the record. A surprising proportion of serious defects are visible in ten seconds to anyone who actually looks, and invisible forever to a suite that never does.