Ex Post Facto Testing
No Bill of Attainder or ex post facto Law shall be passed.
Many constitutions in modern democracies prohibit the passing of ex post facto law, or laws that retroactively change the legal consequences of an action. This is a feature: it prevents new regimes from passing specific laws aimed to put the previous leaders in jail. As an individual, it would be difficult to exist in a society where you always worried that a future law would be passed criminalizing and punishing today’s actions.
Similar logic might be applied to the tests that we write for software. What is the cost of writing tests (laws) after the production code (actions) has happened?
This blog post proposes a new term for tests written after production code has been written. I call them Ex Post Facto Tests (EPFTs). We should look to avoid this practice and gently enourage folks away from the practice.
Let’s explore why.
Test-Driven Development vs. Ex Post Facto Testing
Test-Driven Development (TDD) continues to see friction in the workplace. We often don’t feel we have time to write the tests or the act of writing the tests beforehand is too difficult.
Test-Driven Development is difficult because it forces the developer to codify their understanding of the problem in code before shipping. TDD teases out the design questions and interaction patterns early in the process. TDD almost always results in better code in my experience. The process is more difficult to practice than alternatives but, like exercising or playing a musical instrument, we can improve with practice.
Ex Post Facto Testing happens in an environment that encourages writing tests for regression purposes, but may not encourage regular refactoring. In a more onerous environment, EPFT happens when deadlines are rigid and so are the requirements that “code must be tested” without much thought for the “Why do we test?”
Ex Post Facto Testing results in team members coming to distrust the practice of testing on the whole, because the tests are often a must-do rather than a value-add.
The two process loops have identical ingredients, but yield different outcomes. For TDD, we…
- Write a failing test (Red)
- Write the least amount of code to get the test case to pass (Green)
- Clean up the code (Refactor)
TDD feels pedantic to the new acolyte: “Why even implement the method add as def add(x, y); 4; end? That’s stupid.” Indeed, the non-generalized cases are silly. TDD gives the first impression of coming off silly and unpragmatic. Like investing $5 into a Vanguard index fund, it’s only after sustained investments will teams begin to reap the dividends.
With TDD, we also need to get comfortable leaving untested behavior undefined. If we haven’t written a test for a scenario, we have decided that that behavior is undefined.
Now let’s look to the loop for Ex Post Facto Testing. We…
- Write our production code
- Write tests until we are compelled to stop (we lose interest, we have a deadline, …)
- Clean up the code
The open-endedness of (2) kills us. When do we stop? We can always use our professional judgement, but adequate judgement might not exist on every team. EPFTs don’t give us any design feedback, they simply confirm what we’ve already written. They are the Yes Men of coding practices. The tests are tautological.
We see the complexity of our tests not match the complexity of the subject under test: Why am I bootstrapping a database to add a few numbers together?
When we start speaking in lines of coverage, we have already lost.
How Can We Identify Ex Post Facto Tests?
Ex Post Facto Tests can be identified in production by the following property: If you delete any line of code in a project and you have a green build, EPFTs were used in the development of the project.
Answering the question, “What can I delete and still have a passing build?” is the first question to answer when opening up some legacy code. Mutation testing is a great tool for doing exactly that.
In Pull Requests, EPFTs are easier to identify. A Pull Request that only modifies production code without corresponding test updates is an obvious offender. Taking a look at the commit sequencing can be the next clue. You’re looking for a steady beat of red-green-red-green-red-green in the commit history.
What Should We Do with Ex Post Facto Tests?
Delete them.
You probably still want tests but test at the seams of your modules, classes, and Bounded Contexts. If those seams don’t exist, create them.
EPFTs are a fine tool when putting seams in place, as they can be an effective form of Pinning Tests or Test Vices.
The key with EPFTs is that they are a tool with a specific requirement: they should never land on master.
What Should We Do with Production Code Created Using Ex Post Facto Testing?
Leave it alone.
If you must change it, change it carefully using Pinning Tests or Test Vices.
What If I Don’t Have Time?
Practice TDD going forward. If you find EPFTs while working through legacy code, make it your first goal to safely remove them.
Conclusion
Remember that any process or practice is merely a tool. Test-Driven Development is like a well-balanced and familiar hammer, where Ex Post Facto Testing might be closer to a loose razor blade. It can still do it’s job (cutting), but you’ll need to careful.
With anything, it would be silly to create a mandate “Use the hammer for all tasks” or “We must use the hammer once every three weeks.”
Use your discretion!
Special thanks to Justin Duke and Hector Virgen for reading early drafts of this post.