What are unit tests for?

Unit tests are the “largest” layer of tests. This means that you should have mostly unit tests in your baseline. Unit tests help us do the following:

1. Allow us to refactor and change our design/structure
2. Allow us to add features without breaking other features
3. Help demonstrate correctness
4. Serve as “living” documentation and use cases

Have you ever been in a situation where you fix a bug, but you inadvertently introduced another bug when you applied your fix? This is exactly the type of thing unit tests help prevent.

What is the definition of a unit test?

You’re bound to find many definitions on the internet of what a unit test is. Many of them are self-fulfilling, like “a test that tests a unit of code”. Well that’s a nice, minus the fact that the word “unit” is not well defined (which seems important, would you say?). Even wikipedia’s definition is like this. How large is a unit? Is it a function? A class? A module? The entire application? It turns out that defining what a “unit” constitutes is rather tricky, and not well agreed upon within the software community. Instead of trying to define what a unit test is precisely, I’ll list some necessary conditions I think unit tests must have.

Criteria for a unit test (in order of importance):

1) The tests must be automated:

If I have to run each unit test individually, then the tests are just not useful. The idea behind the unit tests is that after I make a change, I want to run all of the unit tests, and ensure that they are still passing. I want my unit tests to be able to tell me if I have broken any existing behavior, or if the new tests I have added are passing. If I have to do this manually, once I get to 10 or so test files, it becomes extremely burdensome, and hard for me to run all of the tests after a change.

2) Each test case must run in less than .1 (a tenth) of a second:

If you want to be able to refactor safely, you need to run all the unit tests after you make a change. If we run our test harness, and it takes an hour to run, we are unlikely to want to run it very often. This means that we may not be running the entire test suite after a change, which in turn means we are performing unsafe changes and defeating one of the purposes of having unit tests. A tenth of a second seems like an arbitrary line in the sand, but here is the logic: If I get up to 1000 test cases (which many projects will), then it will take 100 seconds (1 minute, 40 seconds) to run all the test cases. Note that it would only take 10 seconds to run if the test cases were .01 seconds. Lastly, observe that it would take 1000 seconds (about 20 minutes) for the tests to run if they were each a second long. 20 minutes is just way too long to wait after making a small change. 1-2 minutes is bearable. 10 seconds is ideal. Longer running tests aren’t necessarily bad, but you want to group them into your integration tests or acceptance tests, not your unit tests. (At some point, I’ll be adding some tips for making unit tests fast).

3) The test could have been written before the code

Copy/pasting your code’s output into a test case (which you can’t possibly do if you write the test before the code) creates fragile tests. In short, a fragile test is a test that is liable to break upon any change (even legitimate changes) and really bogs down the development process (every time you make a change, you have to change that test). Testing in this fashion also allows others to check that your test is actually correct.

4) You can run the tests from your best friend’s laptop (given your repository and a compiler)

If you can’t run your tests from your best friend’s laptop, then you probably haven’t mocked/faked your dependencies properly. If your test needs to connect to a database, read/write files only on your filesystem at work (and not in your repository), or access any other external resources, then the test that you’ve written is not a unit test. Unit tests need to be focused, and only test the behavior of the System Under Test (SUT). Poor use of mocks/fakes can also lead to slow running tests.

P.S. Some more in-depth info about a particular kind of automated test: Mocks (may) be a design smell/issue