Testing Guide

Testing Philosophy

Minimum Requirement

  • Demonstrate that a particular functionality is working

  • Show that the effect of new code is understood, and less likely to interact with the rest of code base in unexpected ways

  • Validate that a bug has been fixed and we can detect the problem in the future

What’s the Benefit to the Team?

  • Reading test code allows others to more easily understand and work with our code

  • Ensures a consistent level of functionality testing for all subsequent changes

  • Speeds up future changes because we can instantly test to see if a change works as expected

How do we Test if a Feature Works?

Since we are dealing with data, it helps to have a toy model which has the following characteristics:

  • Has enough variation to test the different cases which arise due to branching in the code (e.g., include generators of different types to test that the code handles scaling for the right type of generators)

  • Yet is still small enough to be able to verify overall correctness manually

  • Particular values in the data set are easy to work with and confirm manually (e.g., use 5 instead of 4.624754)

  • Includes just the data needed for the code to run - in testing it is better to have an error if the code ever attempts to change something unexpectedly

Prerequisite for Writing Tests

In order to be discoverable by pytest, all the test files need to be prefixed with test. In addition, within the test files themselves test functions should have name starting with test, e.g., test_scale_capacity_argument_type, where scale_capacity is the function being tested and argument_type is the test that is performed on this function.

Examples

Let’s Start Easy

You develop a new feature and you need a helper function that calculates the square root of a number. The function will be declared in a helpers module and the tests will be written in a test_helpers module located in a tests folder at the same level as the helpers module. Three tests should be written for this function:

  • Test the argument type

  • Test the argument value

  • Test the returned value

The function would read:

def calculate_square_root(number):
    """Calculate the nth root of a number.

    :param int/float number: positive number.
    :return: (*float*) -- principal square root of number.
    :raises TypeError: if number has wrong type.
    :raises ValueError: if number is negative.
    """
    if not isinstance(number, (int, float)):
      raise TypeError("number must be a int or float")

    if number < 0:
      raise ValueError("number must be positive")

    return number ** (1.0 / 2)

and the tests will look like:

def test_calculate_square_root_argument_type():
    arg = ((4), [16], {9})
    for a in arg:
        with pytest.raises(TypeError):
            calculate_square_root(a)


def test_calculate_square_root_argument_value():
    with pytest.raises(ValueError):
        calculate_square_root(-9)


def test_calculate_square_root():
    assert calculate_square_root(64) == 8.0
    assert calculate_square_root(49.0) == 7.0
    assert calculate_square_root(36.0) != 5

Mock Objects

Let’s now consider a more complex example. One of the simulation outputs is the power generated by a generator in the grid in a given hour. We would like to summarize this result by calculating the total power generated in an hour per bus. To do so, we need to relate each generator to a connected bus. This information is enclosed in the Grid class. The function will read:

def summarize_plant_to_bus(pg, grid):
    """Take a plant-column data frame and sum to a bus-column data frame.

    :param pandas.DataFrame pg: indices are UTC timestamp and columns are plant id in grid.
    :param powersimdata.input.grid.Grid grid: a Grid instance.
    :return: (*pandas.DataFrame*) -- indices as input data frame, columns are buses.
    """
    # build a data frame mapping plant id to bus id.
    all_buses_in_grid = grid.plant["bus_id"]

    # keep only the rows matching the column of the pg data frame
    buses_in_df = all_buses_in_grid.loc[pg.columns]

    # transpose the pg data frame to get the plant id as indices, group all rows whose
    # indices correspond to the same bus id and sum the values in these rows for each
    # timestamp (column). Finally, transpose back.
    bus_data = pg.T.groupby(buses_in_df).sum().T

    return bus_data

To test this function we need a couple of [mock objects]. One for the pg data frame and one for the grid object. We have developed a catalog of mock objects in the powersimdata/tests/.

We show below how the MockGrid object can be used to test the summarize_plant_to_bus function:

import pandas as pd
from numpy.testing import assert_array_equal
from powersimdata.tests.mock_grid import MockGrid

# plant_id is the index
mock_plant = {
    "plant_id": ["A", "B", "C", "D"],
    "bus_id": [1, 1, 2, 3],
}

# bus_id is the index
mock_bus = {
    "bus_id": [1, 2, 3],
    "lat": [47.6, 37.8, 40.7],
    "lon": [122.3, 122.4, 74],
}

mock_pg = pd.DataFrame(
    {"A": [1, 2, 3, 4], "B": [1, 2, 4, 8], "C": [1, 1, 2, 3], "D": [1, 3, 5, 7]},
    index=pd.date_range(start="2018-04-24", freq="H", periods=4),
)

grid = MockGrid({"plant": mock_plant, "bus": mock_bus})


def test_summarize_plant_to_bus():
    expected_return = pd.DataFrame(
        {1: [2, 4, 7, 12], 2: [1, 1, 2, 3], 3: [1, 3, 5, 7]},
        index=pd.date_range(start="2018-04-24", freq="H", periods=4),
    )
    bus_data = summarize_plant_to_bus(mock_pg, grid)
    assert_array_equal(bus_data.to_numpy(), expected_return.to_numpy())