Parameterized Tests: An Underused Technique

Over the past several years I've come to really value Parameterized Tests. I think they are an underused technique that should be considered more often. They make it easier to cover more of the test space and reduce cognitive load by being succinct.

So what are Parameterized Tests? Well they allow developers to run the same test multiple times over a set of different values. Here is a simple Java example using Java's Stream class with AssertJ:

Stream.of(null, "", " ", "false", "FALSE")
        .forEach(value -> assertThat(Boolean.valueOf(value)).isFalse());

Cucumber Inspiration

I first discovered this technique with Cucumber, which has the ability to run a scenario multiple times over a set of different values using a Scenario Outline. Prior to this I was in the habit of copying and pasting a Scenario and tweaking the Given and Then steps; or I wouldn't test all the sad path combinations in an effort to reduce test maintenance. Learning about the Scenario Outline changed my world. I was able to combine multiple scenarios into a single scenario and the result was more comprehendible and easier to maintain. For example, this is a common Scenario Outline we might use to test an API endpoint to ensure it validates the input. Previously, this would have been spread across multiple scenarios or not tested at all.

Scenario Outline: Should return a 400 when signing up with an invalid email address
  When I attempt to sign up with email "<email>"
  Then I should be returned a "400 Bad Request" status code

  Examples:
    | Email                       |
    | ${absent}                   |
    | ${blank}                    |
    | ${whitespace}               |
    | mailto:john.doe@example.com |
    | john.doe@example            |
    | john.doe@example.           |
    | john.doe@example.com.       |
    | john.doe@@example.com       |
    | john.doe@example..com       |
    | john doe@example.com        |

Recognizing Pattern

Once I discovered this concept and got comfortable with it, I explored ways to introduce it in lower level tests like unit and integration tests. But the trick was recognizing when to apply it. Basically, any time you find yourself copying and pasting a test and tweaking the arrange and assert statements you've probably got a good candidate for a Parameterized Test.

For example, let's say you have a class that determines if a number is a prime number or not. Before Parameterized Tests you might have been tempted to write something like the following:

public class PrimeNumberCheckerTestOldSchool {
    @Test
    public void shouldReturnTrueForPrimeNumber() {
        assertThat(PrimeNumberChecker.check(2)).isTrue();
    }

    @Test
    public void shouldReturnFalseForNonPrimeNumber() {
        assertThat(PrimeNumberChecker.check(6)).isFalse();
    }

    @Test
    public void shouldReturnFalseForAnotherNonPrimeNumber() {
        assertThat(PrimeNumberChecker.check(9)).isFalse();
    }

    @Test
    public void shouldReturnTrueForAnotherPrimeNumber() {
        assertThat(PrimeNumberChecker.check(17)).isTrue();
    }
}

Here you can see we are just repeating the test with different inputs and a different result. And since it's spread out across multiple tests it's hard to comprehend and this solution doesn't scale well if we want to add additional tests.

The ideal solution would be to define a sort of truth table, like Cucumber's Example table, that includes a combination of the inputs and expected result in a single test. This is where Parameterized Tests comes in.

JUnit 5

My preferred way to write Parameterized Tests is with JUnit 5 (see Parameterized Tests with JUnit 5). It's a big improvement over the JUnit 4 way of doing Parameterized Tests. We can rewrite the earlier example using the @ParameterizedTest and @CsvSource annotations (see other source annotations):

public class PrimeNumberCheckerTestJUnit5 {
    @ParameterizedTest
    @CsvSource({
            "2,  true",
            "6,  false",
            "9,  false",
            "17, true"
    })
    public void shouldCheckPrimality(final int number, final boolean expected) {
        assertThat(PrimeNumberChecker.check(number)).isEqualTo(expected);
    }
}

Here are some of the benefits of this approach:

• Easier to comprehend since it's not spread out across 4 test methods.
• Easier to add/remove additional scenarios.
• Easier to delete if the Method Under Test (MUT) is removed; all we will have to do is delete one test method!

To write Parameterized Tests with JUnit 5 you need to include the following dependencies. Note the JUnit 5 documentation states this is an "experimental" feature.

<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-engine</artifactId>
    <version>5.5.2</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-params</artifactId>
    <version>5.5.2</version>
    <scope>test</scope>
</dependency>

JUnit 4

A quick note about JUnit 4. If your team/project is still using JUnit 4, you can use both JUnit 4 and JUnit 5 simultaneously. JUnit 5 provides a gentle migration path. Just include the following dependency. I also recommend reading the Migration Tips.

<dependency>
    <groupId>org.junit.vintage</groupId>
    <artifactId>junit-vintage-engine</artifactId>
    <version>5.5.2</version>
    <scope>test</scope>
</dependency>

Now you are safe to write JUnit 5 tests without having to migrate all your existing JUnit 4 tests and be able to take advantage of the new @ParameterizedTest annotation.

Conclusion

Well hopefully I've convinced you of the power of Parameterized Tests and you'll look for the right opportunity to try them out. And keep in mind while these examples use Java, this technique should also be applicable to other languages. So do some research and see if your language provides them and if not maybe create your own. That's what we did before JUnit 5 was out and we weren't happy with the JUnit 4 way of doing Parameterized Tests.