Unit Testing

The basic idea of unit testing is to verify that a piece of code works the way it is supposed to work. That is, have the code try to solve the a problem with a known answer, and if the code does not give the right answer, it does not work. There is a built-in module for doing this in Python called unittest, but PyTest is becoming the standard.

Why unit test?

When developing code, we always find ourselves running some "sanity checks" to see if it makes sense. Unit tests formalize that process so that we can methodically check the correctness of important pieces of the code, and most importantly, we can *recheck* them after making changes. Done correctly, unit tests should speed development by allowing you to make ambitious changes to you code and quickly verify that you haven't broken anything. Although unit tests are very helpful when you are the sole developer of a code base, they really shine when multiple people are making changes to the code. Continuous integration (whereby code is run through unit tests each time someone merges changes into a code base) is the modern mainstay of coordinated code development, and it is only possible because of unit testing.

Done incorrectly, they add a lot of extra hoops to jump through when editing your code without telling you anything meaningful about the functionality. This is why it is important to carefully choose the scope of your unit tests. If you choose your units (the block of code under test) too big, then failed tests don't illuminate where or why code broke and you run the risk of missing important failure modes. If you choose your units too small, then superficial code changes or refactorings end up requiring significant changes in the test code. Similarly, if the tests you write are too micro-managy (i.e. enforcing *how* a solution is computing, instead of the solution itself), you may find that tests do not survive code changes. On the other hand, if you only check final answers, you may find that the interfaces between code sections are not well enforced and are prone to break for non-obvious reasons.

At its best, unit testing helps authors write modular, factorized code with well-defined interfaces, while also providing on-going quality assurance. In the short term, it can feel like it takes extra time to write tests, but that time is earned back during the debugging and integration stages of code development.

When is the right time to write unit tests? For large applications, it can be *before* you've written your actual code, at the time when you have defined the interfaces and the objectives. For smaller applications (the ones we see more often in small-scale science projects), the right time is just after you've sketched out some exploratory code. You've written enough to convince yourself you know what the code should look like, and it's time to graduate that code into something a bit more stable. Ideally, you should write your tests as you write the code, using the unit tests themselves as the mechanism for convincing yourself that the code works. Whatever it was that you wanted to see that would tell you the code "makes sense"---that's what you should put in the unit test. As you make changes and refactor the code, you can extend these tests, evolving them in step with the maturity of your code.

Practical Unit Testing in Python

This section serves as a recipe for adding unit tests to a python software package you may be writing. If you need a primer on writing Python Packages, see Packaging Projects. For this section, we assume your package is laid out as follows:

• Package Directory (mypkg)
  • Setup Script (mypkg/setup.py)
  • Source Code (mypkg/mypkg_src)

In this case, the best place to put your unit tests is probably mypkg/mypkg_src/tests, although mypkg/tests is a reasonable alternative. Inside this directory, you should write a file with the following structure:

import unittest
import numpy as np
import mypkg

class TestMyPkg(unittest.TestCase):
    def test_method1(self):
        answer = mypkg.method1(parameters)
        self.assertEqual(answer, true_answer)
    def test_method2(self):
        ans_array = mypkg.method2(parameters)
        np.testing.assert_all_close(ans_array, true_ans_array)

if __name__ == '__main__':
    unittest.main()

Assuming you fill out the values of the above pseudo-code, you can then run the unit test from the command line as:

$ python test_mypkg.py
..
----------------------------------------------------------------------
Ran 2 tests in 0.072s

OK

If you are just developing this project yourself, that's about all there is to it! Add new unit test scripts to test different modules, and add new objects/methods within each script to test functions and objects for each module. Just remember to keep running your different unit tests when you make changes.

Setting Up Continuous Integration

Continuous integration is the merging of unit testing into a revision control workflow so that tests are automatically run when code changes are proposed. CI is well-supported on GitHub by setting up Actions which trigger, where you can configure a repository to run tests automatically before allowing branches to be merged into main. This tutorial provides a good step-by-step formula for setting it up.