{"id":23432,"date":"2022-11-11T12:31:38","date_gmt":"2022-11-11T20:31:38","guid":{"rendered":"https:\/\/coderpad.io\/?p=23432"},"modified":"2023-06-05T13:52:01","modified_gmt":"2023-06-05T20:52:01","slug":"testing-in-python-types-of-tests-and-how-to-write-them","status":"publish","type":"post","link":"https:\/\/coderpad.io\/blog\/development\/testing-in-python-types-of-tests-and-how-to-write-them\/","title":{"rendered":"Testing in Python: Types of Tests and How to Write Them"},"content":{"rendered":"\n

You\u2019ve finally <\/em>gotten your API set up. Now other applications can talk to your fully functioning REST API. Now what? What\u2019s the next step? Making sure it works! How do you know if your REST API<\/a> is functioning correctly<\/em>? Well, naturally, you test it.<\/p>\n\n\n\n

Why is testing important?<\/h2>\n\n\n\n

When you hear a developer refer to testing, we\u2019re typically talking about automated<\/em> testing. You may have done testing by manually making calls to your API using a tool such as Postman, Insomnia, or cURL. This works, but every time you go to change something then you have to go back and test that nothing else broke from your change.<\/p>\n\n\n\n

Now, imagine you have a massive<\/em> REST API that\u2019s hundreds of thousands of lines long. Think about doing that same manual testing on every endpoint for every change. My gut reaction is to cringe because I don\u2019t want to do that much busy work. <\/p>\n\n\n\n

Manually testing every change just doesn\u2019t scale to a massive API. You don\u2019t want to spend hours testing just because you changed a utility file. Or, even worse, you don\u2019t want to break something and not even know it.<\/p>\n\n\n\n

The solution to these problems is to automate all your manual testing, so that you can verify your API’s functionality significantly faster. This also removes the risk of forgetting edge cases if you have all use cases safely documented in existing tests.<\/p>\n\n\n\n

Types of testing<\/h2>\n\n\n\n

Automated tests come in many types and flavors. The different types of tests can be used as general guidelines and help you make sure you\u2019re writing comprehensive tests. The two most common types of tests are unit tests<\/strong> and integration tests<\/strong>.<\/p>\n\n\n\n

Unit tests<\/h3>\n\n\n\n

Unit tests ideally test a small unit (or rather, amount) of code. This can often mean one function or one if<\/code> statement. What level of granularity you want in your tests depends largely on what could reasonably go wrong in code you\u2019ve written. <\/p>\n\n\n\n

Your unit tests should be aimed at testing one unit of your code. Your code doing one, tiny, action<\/em>is what you want to test. For example, you want to test that your pet_the_cat<\/code> function returns the correct reaction from an imaginary cat. Given a class like this:<\/p>\n\n\n

import<\/span> requests\nfrom<\/span> typing import<\/span> Optional\n\n\nclass<\/span> Cat<\/span>:<\/span>\n    def<\/span> __init__<\/span>(self, cat: str)<\/span>:<\/span>\n        self.cat = cat\n        self.fed = 0<\/span>\n\n    def<\/span> ask_what_to_do<\/span>(self)<\/span> -> dict:<\/span>\n        return<\/span> requests.get(\"http:\/\/www.boredapi.com\/api\/activity\"<\/span>).json()\n\n    def<\/span> pet_the_cat<\/span>(self)<\/span> -> Optional[str]:<\/span>\n        actions = {\n            \"catname\"<\/span>: \"action\"<\/span>,\n            \"bean\"<\/span>: \"purr\"<\/span>,\n            \"hades\"<\/span>: \"scratch\"<\/span>,\n        }\n        return<\/span> actions.get(self.cat)<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
You could either manually call your pet_the_cat<\/code> function a bunch of times, giving it different input, OR you could write a unit test that look like:<\/p>\n\n\n
from<\/span> cat import<\/span> Cat\n\n\ndef<\/span> test_pet_the_cat<\/span>()<\/span>:<\/span>\n    tests = {\n        # \"input_parameter\": \"expected_result\",<\/span>\n        \"catname\"<\/span>: \"action\"<\/span>,\n        \"not_there\"<\/span>: None<\/span>,\n        1<\/span>: None<\/span>,\n    }\n    failed = 0<\/span>\n    for<\/span> input_value, expected_result in<\/span> tests.items():\n        cat = Cat(input_value)\n        output = cat.pet_the_cat()\n        if<\/span> output != expected_result:\n            print(\"FAILED: '{}' returned '{}'\"<\/span>.format(\n                input_value, output,\n            ))\n            failed += 1<\/span>\n\n    print(\"{}\/{} Tests succeeded\"<\/span>.format(\n        len(tests) - failed, len(tests),\n    ))\n\n\nif<\/span> __name__ == \"__main__\"<\/span>:\n    test_pet_the_cat()<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
Now, when you want to add more test cases, you add them to the tests<\/code> dict, and run all of them by running the test functions once. Now you don’t have to remember all your test cases, and you don’t have to spend time running them manually.<\/p>\n\n\n\n
The most fundamental rule of unit tests is that they should test only one unit of code at a time. Just like how in the example the feed_the_cat<\/code> function is tested individually.<\/p>\n\n\n\n
It\u2019s also important to note that, as part of testing small<\/em>units of code, you also don\u2019t want to rely on any other service (like a database). Your unit tests should all be able to run independent of each other (and in parallel) and not have dependencies on other services. This is why ask_what_to_do<\/code> isn\u2019t tested in this example: because it has an external dependency (the API call).<\/p>\n\n\n\n
While we can test small bits of code, you usually don\u2019t need to write unit tests for built-in functions. After all, why<\/em> test that 1+1<\/code> is 2<\/code>? You can trust that the addition functionality built into your language is trustworthy. Also, you probably don\u2019t need to write tests for third-party code that is well tested and used. Standard code used by everyoneshould<\/em> have its own tests.<\/p>\n\n\n\n
You also want comprehensive code coverage, meaning you want every line of your code to be used in at least one test. This means in an if \u2026 else<\/code> code block there is a unit test for the if<\/code> part and a unit test for the else<\/code> part.<\/p>\n\n\n\n
Ideally, because unit tests are cheap (in cost and time, partially due to the ability to parallelize them) to run, you want total code coverage using unit tests.<\/p>\n\n\n\n
Integration tests<\/h3>\n\n\n\n
Of course, you still need to test how all the units of code you just unit tested work together. Enter integration tests, which specifically test how units of code interact together.<\/p>\n\n\n\n
It\u2019s worth noting that integration tests can have dependencies on things such as databases, and as a result are often slower to run. Ideally they should be written so that they can run in parallel,  so even if they have dependencies on databases they\u2019ll use their own (probably temporary) database table.<\/p>\n\n\n
from<\/span> cat import<\/span> Cat\n\ndef<\/span> test_ask_what_to_do<\/span>()<\/span>:<\/span>\n    cat = Cat(\"cat\"<\/span>)\n    runs = 2<\/span>\n    tests_passed = 0<\/span>\n    for<\/span> _ in<\/span> range(2<\/span>):\n        try<\/span>:\n            result = cat.ask_what_to_do()\n            assert<\/span> isinstance(result, dict)\n            assert<\/span> \"activity\"<\/span> in<\/span> result\n            # Since the result is not always the same we<\/span>\n            # can't test everything about the result, but it's<\/span>\n            # an external dependency that has its own tests<\/span>\n            tests_passed += 1<\/span>\n        except<\/span>:\n            continue<\/span>\n\n    print(\"{}\/{} Tests succeeded\"<\/span>.format(\n        tests_passed, runs,\n    ))\n\n\nif<\/span> __name__ == \"__main__\"<\/span>:\n    test_ask_what_to_do()<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
How to write tests<\/h2>\n\n\n\n
The first and most important step to writing tests is to write testable code.<\/strong> This might seem obvious, but if you don\u2019t do it the first time you write your code, then you\u2019ll probably have to refactor it. The alternative is to have tests that are a nightmare to read and implement.<\/p>\n\n\n\n
What does writing testable code mean, though? Testable code is often best thought about from the unit test level first \u2013 can you call a small unit of code? If you have a function that\u2019s hundreds of lines long, that\u2019s probably not very testable. After all, how do you, using automation, isolate part of the function and run only that?<\/p>\n\n\n\n
For example, looking at the following code snippet, how would you write unit tests for this?<\/p>\n\n\n
import<\/span> requests\nfrom<\/span> typing import<\/span> Optional\n\n\nclass<\/span> Cat<\/span>:<\/span>\n    def<\/span> __init__<\/span>(self, cat: str)<\/span>:<\/span>\n        self.cat = cat\n        self.fed = 0<\/span>\n\n    def<\/span> pet_the_cat<\/span>(self)<\/span> -> Optional[str]:<\/span>\n        found_pet = requests.get(\"https:\/\/petstore.swagger.io\/v2\/pet\/1\"<\/span>).json()\n        if<\/span> found_pet.get(\"message\"<\/span>) == \"Pet Not Found\"<\/span>:\n            self.cat = \"UNKNOWN\"<\/span>\n\n        actions = {\n            \"catname\"<\/span>: \"action\"<\/span>,\n            \"bean\"<\/span>: \"purr\"<\/span>,\n            \"hades\"<\/span>: \"scratch\"<\/span>,\n            \"diana\"<\/span>: \"sleep\"<\/span>,\n        }\n        if<\/span> self.fed % 2<\/span> == 0<\/span> and<\/span> self.cat == \"diana\"<\/span>:\n            self.fed += 1<\/span>\n            return<\/span> \"fed\"<\/span>\n\n        return<\/span> actions.get(self.cat)<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
The answer is: You don’t. It would be difficult. Instead, you want to take that same code, and make it testable:<\/p>\n\n\n
import<\/span> requests\nfrom<\/span> typing import<\/span> Optional\n\n\nclass<\/span> Cat<\/span>:<\/span>\n    def<\/span> __init__<\/span>(self, cat: str)<\/span>:<\/span>\n        self.cat = cat\n        self.fed = 0<\/span>\n\n    def<\/span> feed_the_cat<\/span>(self)<\/span> -> Optional[str]:<\/span>\n        if<\/span> self.fed % 2<\/span> == 0<\/span> and<\/span> self.cat == \"diana\"<\/span>:\n            self.fed += 1<\/span>\n            return<\/span> \"fed\"<\/span>\n\n    def<\/span> get_cat_action<\/span>(self)<\/span> -> str:<\/span>\n        actions = {\n            \"catname\"<\/span>: \"action\"<\/span>,\n            \"bean\"<\/span>: \"purr\"<\/span>,\n            \"hades\"<\/span>: \"scratch\"<\/span>,\n            \"diana\"<\/span>: \"sleep\"<\/span>,\n        }\n        return<\/span> actions.get(self.cat)\n\n    def<\/span> find_the_pet<\/span>(self)<\/span> -> None<\/span>:<\/span>\n        found_pet = requests.get(\"https:\/\/petstore.swagger.io\/v2\/pet\/1\"<\/span>).json()\n        if<\/span> found_pet.get(\"message\"<\/span>) == \"Pet Not Found\"<\/span>:\n            self.cat = \"UNKNOWN\"<\/span>\n\n    def<\/span> pet_the_cat<\/span>(self)<\/span> -> Optional[str]:<\/span>\n        self.find_the_cat()\n\n        fed = self.feed_the_cat()\n        if<\/span> fed is<\/span> not<\/span> None<\/span>:\n            return<\/span> fed\n\n        return<\/span> self.get_cat_action()<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
Now, you can write separate unit tests for each function without dependencies and an integration test for the one that has dependencies.<\/p>\n\n\n\n
Ok, I have unit testable code. Now make it integration testable.<\/h3>\n\n\n\n
After you have code that\u2019s easy to unit test, think about your code at the integration test level. Are there clear lines, and ideally \u2018contracts\u2019 between modules of code?<\/p>\n\n\n\n
For instance, earlier in our Cat<\/code> example we had an ask_what_to_do<\/code> function that returns different results on every call. However, are there things we know we can expect to be true about the data returned by that function?<\/p>\n\n\n\n
Yes! We know the key names to expect in an error or a success. Because of this, we can test what we do<\/em> know about the results:<\/p>\n\n\n
assert<\/span> isinstance(result, dict)\nassert<\/span> \"activity\"<\/span> in<\/span> result<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
On the other hand, if we expected an error, we could have our test assert the keys expected:<\/p>\n\n\n
assert<\/span> isinstance(result, dict)\nassert<\/span> \u201cerror\u201d in<\/span> result\nassert<\/span> \u201cactivity\u201d not<\/span> in<\/span> result<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
\n
\u2139\ufe0f Notice that the unit tests shown here can run without anything other than the file and python, while the integration tests cannot. This is because the unit tests don\u2019t depend on the REST API to be something we can actually \u2018hit\u2019 or \u2018call\u2019 \u2013 it tests the underlying code. Meanwhile, integration tests try to actually call the API, so if the API isn\u2019t being hosted in some way then the integration tests cannot run.<\/p>\n<\/blockquote>\n\n\n\n
Testing code that has dependencies<\/h3>\n\n\n\n
So, we\u2019ve specified that unit tests shouldn\u2019t call to external dependencies, but it\u2019s just a fact that code is going to have external dependencies, like our database, and network calls, like making a network call to the API.<\/p>\n\n\n\n
So, how do we write unit tests for Python code that relies on dependencies? There are lots of options, but we\u2019re going to talk about two common ones – unittest<\/strong> and pytest<\/strong>. <\/p>\n\n\n\n
\n
\u2139\ufe0f Learn how you can test for code that has dependencies in this guide to database unit testing with pytest and SQLAlchemy<\/a>.<\/p>\n<\/blockquote>\n\n\n\n
Unittest<\/strong> is built into Python and has all the basic things you\u2019ll need to write tests. Pytest<\/strong> is often used in conjunction with unittest<\/strong>, as they share a fair bit of functionality. It\u2019s fair to think of pytest<\/strong> as a tool designed to make writing and running tests easier. In particular, pytest<\/strong> enables functionality like running tests in parallel. Meanwhile unittest<\/strong> is a tool designed to enable writing tests, but it, by default, does not have the ability to run tests in parallel.<\/p>\n\n\n\n
Using these tools, we can switch out dependencies with code that make it easier to test without modifying our source code. There are two ways to do this: dependency injection and mocking. In my experience people tend to be passionate about one or the other.<\/p>\n\n\n\n
Dependency injection<\/strong><\/h4>\n\n\n\n
Dependency injection is a complex topic, but here’s the gist: imagine writing your code in such a way that it\u2019s possible to pass in different code to any calls to external dependencies during test time, allowing for a cleaner unit test. In other words, you \u2018inject\u2019 a fake dependency, making the code not call out to an external dependency.<\/p>\n\n\n\n
To illustrate, let’s imagine when I\u2019m ordering something online, and I choose to order a test item instead of a real item. The website says \u201coh, ok\u201d, and doesn\u2019t bother to charge me or send any kind of order to its database.<\/p>\n\n\n\n
For example, we make a small change to our ask_what_to_do<\/code> function in our cat class: <\/p>\n\n\n
from<\/span> typing import<\/span> Callable, Optional\n\n\nclass<\/span> Cat<\/span>:<\/span>\n    def<\/span> __init__<\/span>(self, cat: str)<\/span>:<\/span>\n        self.cat = cat\n        self.fed = 0<\/span>\n\n    def<\/span> ask_what_to_do<\/span>(self, request_function: Callable)<\/span> -> dict:<\/span>\n        return<\/span> request_function(\"http:\/\/www.boredapi.com\/api\/activity\"<\/span>).json()\n\n    def<\/span> pet_the_cat<\/span>(self)<\/span> -> Optional[str]:<\/span>\n        actions = {\n            \"catname\"<\/span>: \"action\"<\/span>,\n            \"bean\"<\/span>: \"purr\"<\/span>,\n            \"hades\"<\/span>: \"scratch\"<\/span>,\n        }\n        return<\/span> actions.get(self.cat)<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
In ask_what_to_do<\/code>, rather than directly calling requests.get<\/code> we call request_function<\/code> which is an argument we pass in. Now, we have control over what request_function<\/code> is, rather than it always being a call out to requests.get<\/code> (a dependency). Thus we can pass in a different function fake_request_function<\/code> which does not call out to an external dependency, thus injecting<\/em> the dependency.<\/p>\n\n\n\n
And now, because we\u2019re injecting the dependency, we can write a unit test for the function, ask_what_to_do<\/code>, that previously had a dependency:<\/p>\n\n\n
from<\/span> cat_1 import<\/span> Cat\n\n\nclass<\/span> FakeFailResult<\/span>:<\/span>\n    def<\/span> json<\/span>(self)<\/span>:<\/span>\n        return<\/span> {}\n\n\nclass<\/span> FakeSuccessResult<\/span>:<\/span>\n    def<\/span> json<\/span>(self)<\/span>:<\/span>\n        return<\/span> {\n            \"activity\"<\/span>: \"Organize your pantry\"<\/span>,\n            \"type\"<\/span>:\"busywork\"<\/span>,\n            \"participants\"<\/span>:1<\/span>,\n            \"price\"<\/span>:0<\/span>,\n            \"link\"<\/span>:\"\"<\/span>,\n            \"key\"<\/span>:\"3954882\"<\/span>,\n            \"accessibility\"<\/span>:0<\/span>,\n        }\n\n\ndef<\/span> fake_request_function<\/span>(url)<\/span> -> dict:<\/span>\n    if<\/span> url != \"http:\/\/www.boredapi.com\/api\/activity\"<\/span>:\n        return<\/span> FakeFailResult()\n\n    return<\/span> FakeSuccessResult()\n\n\ndef<\/span> test_ask_what_to_do<\/span>()<\/span>:<\/span>\n    # Since all `ask_what_to_do` does is make this request, we can easily test<\/span>\n    # that nothing has been accidentally added<\/span>\n    cat = Cat(\"catname\"<\/span>)\n    output = cat.ask_what_to_do(fake_request_function)\n    assert<\/span> output == FakeSuccessResult().json()\n\n\nif<\/span> __name__ == \"__main__\"<\/span>:\n    test_ask_what_to_do()<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
\n
\ud83d\udca1 Dependency injection has more code up-front so it doesn\u2019t make as much sense on a small scale, but it gains a lot more value when you get to the scale of classes and multiple things being passed around or called.<\/p>\n<\/blockquote>\n\n\n\n
Mocking<\/strong><\/h4>\n\n\n\n
Mocking (and no, I\u2019m not mocking you\u2013it\u2019s actually called mocking) is about hijacking a call, be it a function call, or a class, or whatever, and giving back fake output. So, the external dependency is never called, even though the code thinks it was.<\/p>\n\n\n\n
This is a lot like if you told me to do something, and I told you I did it, even though I never did. I gave you the output of \u201cI did it\u201d, but I never had to actually get up and go do something, so you didn\u2019t have to wait for me to finish doing some action.<\/p>\n\n\n\n
This makes a lot of sense on a small scale because it\u2019s not much code, but at scale it gets unwieldy quickly. As you can imagine, you don\u2019t want 500 mocks for one test.<\/p>\n\n\n\n
A quick example:<\/p>\n\n\n\n
Using our original implementation of Cat:<\/p>\n\n\n
from<\/span> unittest import<\/span> mock\nfrom<\/span> cat import<\/span> Cat\n\n\n@mock.patch(\"cat.requests\")<\/span>\ndef<\/span> test_ask_what_to_do<\/span>(mocked_requests)<\/span>:<\/span>\n    mocked_result = mock.MagicMock()\n    mocked_requests.get.return_value = mocked_result\n\n    cat = Cat(\"catname\"<\/span>)\n    output = cat.ask_what_to_do()\n    assert<\/span> output == mocked_result.json()\n    mocked_requests.get.assert_called_once_with(\n        \"http:\/\/www.boredapi.com\/api\/activity\"<\/span>\n    )\n\n\nif<\/span> __name__ == \"__main__\"<\/span>:\n    test_ask_what_to_do()<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
Within Python mocks there are different types\u2013Mock<\/code>s, MagicMock<\/code>s, AsyncMocks<\/code>\u2013all with their own specific uses (there are also things called fixtures and different tools you can use for testing), but I\u2019ll save those for another blog post.<\/p>\n\n\n\n
Mocking vs dependency injection<\/strong><\/h4>\n\n\n\n
The reality is that \u2018mocking\u2019 is typically better for small scale projects without a lot of dependencies, because it doesn\u2019t require a lot of work up front. However, mocks are not very reusable, so they don\u2019t scale well.<\/p>\n\n\n\n
Meanwhile, \u2018dependency injection\u2019 is better for large scale projects, but typically requires more up front work setting up all the fake dependencies. After you have dependency injection setup it\u2019s easier to reuse fake dependencies across your code base, though, so it can be great for scaling.<\/p>\n\n\n\n
In my experience these two concepts can be polarizing because they result in fundamentally different programming styles – where do you put the dependencies, how are they called, etcetera. That said, one style can be converted to another, so which you should use is a question based on either:<\/p>\n\n\n\n
Other test types<\/h3>\n\n\n\n
There are many more types of tests, such as contract tests<\/a> and functional tests<\/a> (often confused with integration tests!), but unit and integration tests are far and away the most important tests to start with.<\/p>\n\n\n\n
There are a lot of tools out there that can help you measure code coverage to ensure that every possible code path is covered. For Python, I\u2019m partial to using the tool Coverage.py<\/a>, but there are different options for different languages. For example, Istanbul<\/a> is a popular tool for JavaScript, but I\u2019ve seen plenty of people use Cypress\u2019s code coverage plugin<\/a>.<\/p>\n\n\n\n