Engineering
Silent Failure Is the Expensive Kind
Mike Lewis ·
Software that breaks loudly gets fixed the same day. Software that breaks quietly can run wrong for months while every dashboard says it is fine.
There is a category of fault that never appears in an error log, never triggers an alert and never gets reported by a user, because from the outside everything looks like it worked. These are the ones that cost real money, and they are almost always caused by a well-meaning piece of defensive code.
Three ways it happens
The first is the empty catch. Something is wrapped in error handling so a failure will not take the page down, and the failure then goes nowhere. We found an audit log on one system that had silently recorded nothing for weeks, because every write threw and every throw was swallowed by the code protecting the user from seeing it.
The second is missing configuration. A setting that is absent does not usually announce itself; it simply reads as empty, and the feature depending on it behaves as though something else went wrong. We spent a long time investigating a sign-in problem that reported itself as the user cancelling, and turned out to be four configuration values that were never supplied to the deployment at all. Nothing failed at build time. Nothing appeared in any log.
The third is the forgiving converter. Code that moves data between two shapes and quietly drops anything it does not recognise. We hit this three times on one project in a single day: an import that discarded every bulleted list, an editor that would have deleted those lists the first time anyone pressed Save, and a mapping that emptied every headline figure on a case study because the source called the field one thing and the destination called it another. None of the three raised so much as a warning.
What we do instead
- Never catch an error without reporting it somewhere a person will actually see
- Fail loudly on missing configuration at startup, naming what is absent, rather than degrading into odd behaviour later
- For anything that converts between two formats, test the round trip: convert out, convert back, assert you got exactly what you started with
- Check the destination, not the operation. Confirm the record is in the database, not that the save function returned
The round trip test is the cheap one
Of all of these, the round trip is the best value. It is two lines, it needs no infrastructure, and it fails immediately on exactly the class of bug that is otherwise invisible until someone notices content has quietly gone missing. If your system converts data between formats anywhere, and most do, it is the first test worth writing.