7  Testing Code

Tests are an essential part of software development. They help ensure that your code works as expected and that new changes do not introduce bugs or regressions. In this tutorial, you will learn how to write and run tests in Python using the pytest testing framework.

7.1 Why Write Tests?

Writing tests is an important practice in software development for several reasons:

  • Quality Assurance: Tests help ensure that your code works as expected and that new changes do not introduce bugs or regressions. They provide a safety net that allows you to make changes with confidence.
  • Documentation: Tests serve as documentation for your code by specifying the expected behavior of your functions and modules. They help you understand how your code should work and how it should be used.
  • Refactoring: Tests help you refactor your code with confidence by ensuring that existing functionality is not broken. They allow you to make changes to your code without worrying about introducing bugs.
  • Debugging: Tests help you identify and diagnose issues in your code by providing a way to reproduce and isolate problems. They help you narrow down the cause of the issue and fix it quickly.
  • Collaboration: Tests help improve collaboration among team members by providing a common language to discuss the expected behavior of the code. They help ensure that everyone is on the same page and understands how the code should work.
  • Continuous Integration: Tests are an essential part of continuous integration and continuous deployment (CI/CD) pipelines. They help automate the testing process and ensure that your code is tested automatically before deployment.
  • Confidence: Tests give you confidence in your code by providing a safety net that allows you to make changes without fear of breaking existing functionality. They help you write better code and improve the quality of your software.

Tests are meant to be run frequently during development to ensure that your code works as expected. They help you catch bugs early and fix them before they become more difficult to diagnose and resolve. By writing tests, you can improve the quality of your code and build more reliable software.

Figure 7.1: Do you trust your code?

7.2 Boundary Value Testing

Sometimes, testing the boundaries of your code is as important as testing the code itself. Boundary Value Testing is a software testing technique that focuses on testing the boundaries of input values. The idea is to test the minimum and maximum values of the input range, as well as the values just below and above the boundaries. This helps identify issues related to boundary conditions and ensures that the code handles edge cases correctly. For example:

  • If a function accepts a range of values from 1 to 10, you would test the values 0, 1, 10, and 11 to ensure that the function handles the boundaries correctly.
  • If a function accepts a list of values, you would test an empty list, a list with one element, and a list with multiple elements to ensure that the function handles different input sizes. Additionally, you would test the list with the maximum number of elements to ensure that the function can handle large inputs.
  • If a function accepts a string, you would test an empty string, a string with one character, and a string with multiple characters to ensure that the function handles different string lengths.

7.3 Test-Driven Development (TDD)

Test-Driven Development (TDD) is a software development technique that involves writing tests before writing the actual code. The TDD cycle consists of three steps: writing a failing test, writing the code to make the test pass, and refactoring the code. TDD helps you write better code by focusing on the expected behavior of your code and ensuring that it works as expected.

7.4 Black-Box, Functional, Unit Testing

There are several types of testing that you can perform on your code to ensure its quality and reliability. We focus on tests at the intersection of these three types:

  • Black-Box Testing: Black-Box Testing is a testing technique that focuses on testing the functionality of the software without looking at the internal code. The idea is to test the software based on its input and output without knowing how it works internally. Black-Box Testing helps you identify issues related to the functionality of the software and ensures that it behaves as expected.
  • Functional Testing: Functional Testing is a testing technique that focuses on testing the functionality of the software by providing input and checking the output. The goal of functional testing is to verify that the software works as expected and that it meets the requirements. Functional tests are typically larger and more complex than unit tests and test the functionality of the software as a whole.
  • Unit Testing: Unit Testing is a testing technique that focuses on testing individual units or components of your code in isolation. The goal of unit testing is to verify that each unit of your code works correctly and that it behaves as expected. Unit tests are typically small, focused tests that test a single function or method and its behavior. They help you identify issues in your code early and ensure that each unit of your code works as expected.

7.5 Writing Tests in Python

Python provides several tools and libraries for writing tests, including the built-in assert statement and the pytest testing framework. You can use these tools to write tests for your Python code and ensure its quality and reliability.

7.5.1 Assertions in Python

In Python, assert is used to add assertions to your code. An assertion is a statement that checks whether a particular condition is True, and if it’s not, it raises an AssertionError. This helps you identify and diagnose issues in your code during development and debugging. The syntax for an assertion is as follows:

assert condition, message

Where:

  • condition is the expression that you want to check. If the condition is False, an AssertionError is raised.
  • message is an optional message that provides more information about the assertion. It is displayed when the assertion fails.

For example, in Listing 7.1, several assertions are used to check the expected results of the code. Assertions are meant to be used for debugging and development purposes and should not be used for error handling in production code. If an assertion fails, it indicates a bug in the code that needs to be fixed. They are a “fail fast” mechanism that helps you identify issues early in the development process, you are not supposed to catch and handle them.

Listing 7.1: Example of using assertions in Python. The function divide checks if the divisor b is not zero before performing the division operation. The test function test_divide checks the expected results of the divide function using assertions. 

def divide(a, b):
    assert b != 0, "Division by zero is not allowed"
    return a / b

def test_divide():
    assert divide(10, 2) == 5
    assert divide(10, 0) == 0

Assertions are the foundation of Black Box Testing, which is a testing technique that focuses on the functionality of the software without looking at the internal code. In this case, we are testing the function divide without knowing how it works internally. You are only interested in the input and output of the function.

7.5.2 Writing Tests with Pytest

You can use the pytest testing framework to test your Python code. Pytest is a popular testing framework for Python that makes it easy to write simple tests. You can find more information about pytest in the official documentation.

Pytest provides a simple and intuitive way to write tests using the assert statement. You can write tests as regular Python functions and use assertions to check the expected results. To write tests with Pytest, follow these steps:

  1. Create a new Python file for your tests. You can name the file test_<module>.py, where <module> is the name of the module you want to test. For example, if you want to test a module called basic_math.py, you can name the test file test_basic_math.py. Alternatively, you can create a tests directory and place your test files inside it (e.g., tests/test_basic_math.py). Make sure to make this directory a package by adding an __init__.py file inside it.
  2. Write test functions inside the test file. Test functions should start with test_ to be recognized by Pytest as test functions. You can write multiple test functions to test different parts of your code. It is important to use test_ so that Pytest can automatically discover and run the test functions. Normally, you would have one test function for each function you want to test in your code.
  3. Use the assert statement inside the test functions to check the expected results of your code. If the assertion fails, Pytest will raise an AssertionError and mark the test as failed.
  4. Run the tests using the pytest command in the terminal. Pytest will automatically discover and run the test functions in the test file and display the results.

For example, in Listing 7.3, we have a simple Python module called basic_math.py that contains two functions: add_two and multiply_two. We want to write tests for these functions using Pytest. We create a test file called test_basic_math.py and write test functions to check the expected results of the add_two and divide functions.

Listing 7.2: Example of a simple Python module basic_math.py that contains two functions: add_two and multiply_two. 
# basic_math.py

def add_two(a, b):
    return a + b

def divide(a, b):
    return a / b
Listing 7.3: Example of a test file test_basic_math.py that contains test functions for the add_two and divide functions in the basic_math.py module. 
# test_basic_math.py

import basic_math

def test_add_two():
    assert basic_math.add_two(1, 2) == 3

def test_divide():
    assert basic_math.divide(10, 2) == 5
    assert basic_math.divide(10, -2) == -5

In this example, we have two test functions: test_add_two and test_divide. The test_add_two function checks the expected result of the add_two function, and the test_divide function checks the expected results of the divide function. If the expected result does not match the actual result, Pytest will raise an AssertionError and mark the test as failed.

7.5.3 How to Run Tests with Pytest

To run the tests, follow these steps:

  1. Open the terminal in Visual Studio Code by clicking Terminal > New Terminal or using the keyboard shortcut Ctrl + `.

  2. Run the following command to install the pytest package:

    pip install pytest

  3. Run the following command to run the tests:

     pytest

  4. Verify that the tests pass. A passing test will display a . (dot) for each test that passes and an F for each test that fails.

  5. If a test fails, review the error message to identify the issue in your code. The error message will provide information about the failed test and the expected result. For example, if the test expects the result to be True but the actual result is False, or if the test expects the result to be 5 but the actual result is 10, you will need to review your code to identify the issue.

  6. Make the necessary changes to your code to fix the issue.

  7. Run the tests again to verify that the issue has been resolved.

7.5.4 How to Inspect Pytest Results

When you run pytest, you will see a summary of the test results at the end of the output. The summary will show you how many tests passed, how many tests failed, and how many tests were skipped.

First, you will see a summary of the test results:

============================= test session starts ==============================
platform linux -- Python 3.13.0, pytest-7.0.1, pluggy-1.0.0 -- /usr/bin/python3

collected 3 items

tests/test_exercise_1.py FF [50%]
tests/test_exercise_2.py .  [75%]
tests/test_exercise_3.py F  [100%]

=================================== FAILURES ===================================

In this case, FF means that the file test_exercise_1.py has two failing tests, . means that the file test_exercise_2.py has one passing test, and F means that the file test_exercise_3.py has one failing test.

The percentages are cumulative, so the percentage of passed tests is calculated based on the total number of tests run. In this case, 4 tests were run (2 inside test_exercise_1.py, 1 inside test_exercise_2.py, and 1 inside test_exercise_3.py), and 1 test passed (the test inside test_exercise_2.py). Therefore, 50%, 75%, and 100% indicate that the first file has 50% of tests (i.e., 2/4), the first two files have 75% of tests (i.e., 3/4), and all files have 100% of tests (i.e., 4/4).

After the summary, you will see the detailed output of the tests. Search for the FAILURES section to see which tests failed and why they failed. The output will show you the name of the failed test, the line of code that caused the failure, and the reason for the failure (e.g., an assertion error).

For example, the following output shows that the test test_exercise_1.py::test_add_two failed because the expected result was 3 but the actual result was None:

=================================== FAILURES ===================================
______________________________ test_add_two ______________________________

    def test_add_two():
>       assert basic_math.add_two(1, 2) == 3
E       assert None == 3
E        +  where None = <function add_two at 0x7f8b3b3b6c10>(1, 2)
E        +    where <function add_two at 0x7f8b3b3b6c10> = basic_math.add_two

In this case, the test failed because the function add_two returned None instead of the expected result 3. You must review the code in the basic_math.py file to identify and fix the issue.

You do not need to understand all the details of the output, but you should be able to identify the failed test and its cause (check the E lines for more information).

At the end of the file, you will see a summary of the test results, including the number of tests that passed, the number of tests that failed, and the number of tests that were skipped:

=========================== short test summary info ============================
FAILED tests/test_exercise_1.py::test_add_two - assert None == 3
FAILED tests/test_exercise_1.py::test_multiply_two - assert None == 2
FAILED tests/test_exercise_3.py::test_hello_world_output - AssertionError: assert '' == 'Hello, world!'

In this case, the summary shows that three tests failed (test_add_two, test_multiply_two, and test_hello_world_output) and one test passed. The summary also briefly describes the reason for the failure (e.g., assert None == 3).

Finally, the output will show the total number of tests run and the time it took to run the tests:

========================= 3 failed, 1 passed in 0.10s ==========================

In this case, the output shows that 3 tests failed, 1 test passed, and the tests took 0.10 seconds to run.

7.5.5 Tunning The Verbose Level of Pytest

Pytest provides different verbosity levels to control the output displayed when running tests. By default, pytest will display a summary of the test results, showing the number of tests that passed, the number of tests that failed, and the number of tests that were skipped.

According to the Pytest documentation, you can control the verbosity level of the output by using the following flags:

pytest -q               # quiet - less verbose
pytest -v               # increase verbosity, display individual test names
pytest -vv              # more verbose, display more details from the test output

The following sections will highlight the differences between the different verbosity levels.

7.5.5.1 Less Verbose Output (pytest -q)

If you prefer a less verbose output, you can use the -q flag to run the tests in “quiet” mode:

pytest -q

The -q flag stands for “quiet” and provides a more concise output for each test. The details of the session (e.g., the platform, Python version, and pytest version) will not be displayed, and instead of the per-file summary, you will see something like this:

FF.F     [100%]

In this case, FF.F means that we have four tests, and the first, second, and fourth tests failed, while the third test passed. The rest of the output will be the same as before, showing the detailed output of the tests and the summary of the test results.

7.5.5.2 Verbose Output (pytest -v)

If you want to see more detailed output, you can use the -vv flag:

pytest -v

The -v flag stands for “verbose” and provides a more detailed output for each test. In this mode, you will see the name of each test and the result of the test (e.g., PASSED, FAILED). For example, the following output shows that the test test_add_two passed:

tests/test_exercise_1.py::test_add_two FAILED                 [25%]
tests/test_exercise_1.py::test_multiply_two FAILED            [50%]
tests/test_exercise_2.py::test_greeting_output PASSED         [75%]
tests/test_exercise_3.py::test_hello_world_output FAILED      [100%]

7.5.5.3 More Verbose Output (pytest -vv)

You can use the -vv flag if you want even more detailed output. This will increase the verbosity of the “short test summary” at the end of the output:

=========================== short test summary info ============================
FAILED tests/test_exercise_1.py::test_add_two - assert None == 3
 + where None = <function add_two at 0x7f8b3b3b6c10>(1, 2)
    +   where <function add_two at 0x7f8b3b3b6c10> = basic_math.add_two
FAILED tests/test_exercise_1.py::test_multiply_two - assert None == 2
    + where None = <function multiply_two at 0x7f8b3b3b6d30>(1, 2)
        +   where <function multiply_two at 0x7f8b3b3b6d30> = basic_math.multiply_two
FAILED tests/test_exercise_3.py::test_hello_world_output - AssertionError: assert '' == 'Hello, world!'

   - Hello, world!
========================= 3 failed, 1 passed in 0.10s ==========================

In this case, the output shows the same information as before but with more details about the failed tests. For example, it shows the expected result (3) and the actual result (None) for the test test_add_two.

7.6 Testing in VS Code

Visual Studio Code (VS Code) provides a Testing extension that allows you to run and debug tests directly from the editor. The Testing extension supports various testing frameworks, including pytest. You can find more information about the Testing extension in the official documentation.

To use the Testing extension in VS Code, follow these steps:

  1. Install the Testing extension in VS Code by clicking on the Extensions view (Ctrl + Shift + X) and searching for “Testing”. Click on the Install button to install the extension.
  2. Open the Test Explorer view by clicking on the Test icon in the Activity Bar on the side of the window. This will display the Test Explorer panel, which shows a list of tests in your workspace.

The extension provides several features to help you write and run tests in VS Code:

  • Test Explorer: The Test Explorer displays a list of tests in your workspace and allows you to run and debug them. You can view the test results, navigate to the test files, and filter the tests based on their status (e.g., passed, failed, skipped).
  • Run Tests: You can run tests directly from the editor by clicking the Run Test button next to the test function or file (a green play button). This will run the test and display the results in the Test Explorer.
  • Debug Tests: You can debug tests directly from the editor by clicking the Debug Test button next to the test function or file (a bug icon). This will run the test in debug mode and allow you to step through the code, inspect variables, and identify issues.
  • View Test Output: You can view the output of the tests in the Test Output panel. This panel displays the detailed output of the tests, including the test results, error messages, and stack traces.