The (potential) problem with London Style tests

First, it’s important to understand the difference between a London style and Chicago (or Classic) style test. A Classic style test is all what we’re used to. Let’s say we need to test a function, say add() (of course we’d never write a function so silly, this is just an easy to understand example). We might write some unit tests checking that this add() function works with some quick tests like this:

 

 

Of course, we want to check that this function is correct and also ensure that if someone tries to refactor add(), they can do so safely and without fear. As it was noted earlier, this function is particularly easy to understand. We rarely solve such simple problems like these while programming. Sometimes we solve problems that are more abstract, and sometimes the System Under Test (SUT) is attempting to do something that isn’t easily tested directly like this add() function.

Consider a module responsible for managing usernames and (hashed) passwords for a website. Say we’re trying to test that the module accurately passes the usernames and passwords to a database. How would we test such a module? If you follow the pattern that we did for the add function, your code might look something like this:

 

 

We create a database, pass it to our module in charge of handling usernames and passwords, and then check that the actual values exist in the database. Seems okay, right? Well, there are some problems with the way we’ve written this test. First, we failed to follow criteria #2 of a good unit test. Setting up a db connection, putting values into a database, and then query for them likely takes more than a tenth of a second. We also have violated criteria #4, since the only place this test would run is on our local machine at work (I couldn’t run this test on my best friend’s laptop).

We’ve clearly violated 2 critical unit testing criteria. So how do we fix it? Enter the London style of tests. London style tests are less concerned with algorithmic details, and more concerned with making sure that the correct functions were called. Here’s an example of a more “London” style test for our username/password module:

 

 

If we mock the database object, we can now just assert that our module is making the right functional calls instead of asserting that values are actually written to a database. This is the spirit of a London style test: asserts against behavior, not against output.

This test also rids us of the Database dependency. This gives us two advantages. The first is, the test will run very quickly. We no longer have to hit the (relatively) slow database. The second thing this test does is allow us to run it from anywhere. This test will now pass on my best friend’s laptop. Additionally, the test is more focused on the module we are trying to test.

London style tests do have some significant drawbacks, though. It is easier to introduce bugs with these types of tests. The trade-off for speed and portability can come at the cost of correctness. I wrote a unit test a while ago that seemed to be pretty solid. The module it was testing was responsible for moving output products (stored in a temporary directory) to where a user specifies. Instead of mocking out a directory and using a “Classic/Chicago” style unit test, I decided to use a “London” style unit test to see whether the OS call to move the files was actually being executed.

The test looked something like this:

 

 

Do you see any potential problems with this test? Try to take at least 5 or 10 minutes and think of what could be a bug in the real production code (Note that shutil.move has the same semantics as the Linux mv command. Also note that the patch decorator is simply a way of replacing the actual shutil.move function with a mocked version that keeps track of what arguments it is called with and how many times it’s called, among other things. I’d like to eventually put up a tutorial on the @patch decorator.)

Did you really stop and try to think of bugs? Please do, you’ll get more entertainment out of this post. Besides, it never hurts to exercise the bug finding part of your brain. It’s a useful skill to be able to look at tests and pinpoint what is wrong with them (or at least what is potentially wrong with them).

One potential bug that this London style test will not catch is that we could potentially be moving the same file twice. For example, if we move product2 to output_dir1 first, and then move product2 to output_dir2, this unit test will pass for that behavior, but it will obviously fail in real production code (the second move will error out since product2 has already been moved and is no longer in the temp directory where the module assumes it will be).

So what’s the solution? It turns out that we needed an additional assert. Namely,
assert shutil.move.call_count == 4. This asserts that we have only called the shutil.move function exactly four times. If we call it more than 4 times (like we were doing so sneakily in the original implementation), then our test will fail.

London style tests are powerful in terms of what they offer in speed and portability. However, this comes at the cost of having to do “extra” asserts, and being careful about your logic. As Software Engineers we can rarely get away from trade-offs. One approach is almost never strictly better than another. London style tests are no exception.