Thursday, February 27, 2020

Parameterized Tests: An Underused Technique

Over the past several years I've come to really value Parameterized Tests. I think they are an underused technique that should be considered more often. They make it easier to cover more of the test space and reduce cognitive load by being succinct.

So what are Parameterized Tests? Well they allow developers to run the same test multiple times over a set of different values. Here is a simple Java example using Java's Stream class with AssertJ:

Stream.of(null, "", " ", "false", "FALSE")
        .forEach(value -> assertThat(Boolean.valueOf(value)).isFalse());

Cucumber Inspiration


I first discovered this technique with Cucumber, which has the ability to run a scenario multiple times over a set of different values using a Scenario Outline. Prior to this I was in the habit of copying and pasting a Scenario and tweaking the Given and Then steps; or I wouldn't test all the sad path combinations in an effort to reduce test maintenance. Learning about the Scenario Outline changed my world. I was able to combine multiple scenarios into a single scenario and the result was more comprehendible and easier to maintain. For example, this is a common Scenario Outline we might use to test an API endpoint to ensure it validates the input. Previously, this would have been spread across multiple scenarios or not tested at all.

Scenario Outline: Should return a 400 when signing up with an invalid email address
  When I attempt to sign up with email "<email>"
  Then I should be returned a "400 Bad Request" status code

  Examples:
    | Email                       |
    | ${absent}                   |
    | ${blank}                    |
    | ${whitespace}               |
    | mailto:john.doe@example.com |
    | john.doe@example            |
    | john.doe@example.           |
    | john.doe@example.com.       |
    | john.doe@@example.com       |
    | john.doe@example..com       |
    | john doe@example.com        |

Recognizing Pattern


Once I discovered this concept and got comfortable with it, I explored ways to introduce it in lower level tests like unit and integration tests. But the trick was recognizing when to apply it. Basically, any time you find yourself copying and pasting a test and tweaking the arrange and assert statements you've probably got a good candidate for a Parameterized Test.

For example, let's say you have a class that determines if a number is a prime number or not. Before Parameterized Tests you might have been tempted to write something like the following:

public class PrimeNumberCheckerTestOldSchool {
    @Test
    public void shouldReturnTrueForPrimeNumber() {
        assertThat(PrimeNumberChecker.check(2)).isTrue();
    }

    @Test
    public void shouldReturnFalseForNonPrimeNumber() {
        assertThat(PrimeNumberChecker.check(6)).isFalse();
    }

    @Test
    public void shouldReturnFalseForAnotherNonPrimeNumber() {
        assertThat(PrimeNumberChecker.check(9)).isFalse();
    }

    @Test
    public void shouldReturnTrueForAnotherPrimeNumber() {
        assertThat(PrimeNumberChecker.check(17)).isTrue();
    }
}

Here you can see we are just repeating the test with different inputs and a different result. And since it's spread out across multiple tests it's hard to comprehend and this solution doesn't scale well if we want to add additional tests.

The ideal solution would be to define a sort of truth table, like Cucumber's Example table, that includes a combination of the inputs and expected result in a single test. This is where Parameterized Tests comes in.


JUnit 5


My preferred way to write Parameterized Tests is with JUnit 5 (see Parameterized Tests with JUnit 5). It's a big improvement over the JUnit 4 way of doing Parameterized Tests. We can rewrite the earlier example using the @ParameterizedTest and @CsvSource annotations (see other source annotations):

public class PrimeNumberCheckerTestJUnit5 {
    @ParameterizedTest
    @CsvSource({
            "2,  true",
            "6,  false",
            "9,  false",
            "17, true"
    })
    public void shouldCheckPrimality(final int number, final boolean expected) {
        assertThat(PrimeNumberChecker.check(number)).isEqualTo(expected);
    }
}

Here are some of the benefits of this approach:
  • Easier to comprehend since it's not spread out across 4 test methods.
  • Easier to add/remove additional scenarios.
  • Easier to delete if the Method Under Test (MUT) is removed; all we will have to do is delete one test method!

To write Parameterized Tests with JUnit 5 you need to include the following dependencies. Note the JUnit 5 documentation states this is an "experimental" feature.

<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-engine</artifactId>
    <version>5.5.2</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-params</artifactId>
    <version>5.5.2</version>
    <scope>test</scope>
</dependency>


JUnit 4


A quick note about JUnit 4. If your team/project is still using JUnit 4, you can use both JUnit 4 and JUnit 5 simultaneously. JUnit 5 provides a gentle migration path. Just include the following dependency. I also recommend reading the Migration Tips.

<dependency>
    <groupId>org.junit.vintage</groupId>
    <artifactId>junit-vintage-engine</artifactId>
    <version>5.5.2</version>
    <scope>test</scope>
</dependency>

Now you are safe to write JUnit 5 tests without having to migrate all your existing JUnit 4 tests and be able to take advantage of the new @ParameterizedTest annotation.


Conclusion


Well hopefully I've convinced you of the power of Parameterized Tests and you'll look for the right opportunity to try them out. And keep in mind while these examples use Java, this technique should also be applicable to other languages. So do some research and see if your language provides them and if not maybe create your own. That's what we did before JUnit 5 was out and we weren't happy with the JUnit 4 way of doing Parameterized Tests.

Friday, January 9, 2015

Setting Environment Variables for Docker with Fig

For the past few months I've been playing around with Docker, and so far I've had a ton of fun. The documentation is excellent, and in one simple command you can start experimenting. After going through the tutorial, one of my first goals was to figure out the best way to create a Docker image for a spring-boot service. My initial goals were to make it easy to set environment variables, since our projects follow the twelve-factor methodology. One of the several factors we follow is the third factor (III. Config), which recommends storing the config in the environment. While this has many benefits, one of the downsides is the tendency to create a lot of environment variables because it's quick, easy, and well defined. This makes it difficult to configure, test, and run the service. But as we will see, Fig will not only make it easy to set environment variables, it will also provide many other benefits.

Docker Image
Let's first start with a pretend service called the logging service. It's a Java service created with spring-boot. Here is a basic Dockerfile:

FROM dockerfile/java:oracle-java8
COPY logging-service-0.1.0.jar /data/
EXPOSE 8080
CMD ["java", "-jar", "logging-service-0.1.0.jar"]
FROM - the base image I start with. In this case it's the dockerfile/java base image with the Oracle JDK 8 tag.
COPY - here I copy over the jar so it's present in the image
EXPOSE - this tells Docker the container will be listening on port 8080 at runtime
CMD - here I've defined a default command to run which will start the service

Next we need to build the image:
docker build --tag="jlorenzen/logging-service:v1" .

Docker Run
Now we could run our new service by executing this command:
docker run -dP jlorenzen/logging-service

That's great but let's imagine the logging-service requires the following environment variables: ENV_1 and ENV_2. Here is how you would run the service while also setting the environment variables:
docker run -dP -e ENV_1=value1 -e ENV_2=value2 jlorenzen/logging-service

That's a basic example, but you can image how nasty it could get if your service required a dozen or more environment variables. The docker run command also has some other nice options for setting environment variables. For example, when using the -e option, if you provide just the name like -e ENV_1 without a value, than that variables current value will be used. Or you can use the --env-file option to specify a file that contains a list of environment variables. While this all works, it's really not enjoyable having to remember all those options and commands. That is where Fig can help. And it not only helps us easily set environment variables, but it also makes creating containers simpler and reproducable by anyone anywhere.

Fig
Fig is basically a simple utility that wraps Docker making it easier to create and manage Docker containers. In our case we will use it to run our logging-service image and set the environment variables. Here is a simple fig.yml file:
logging-service:
  image: jlorenzen/logging-service
  ports: 
   - "8080"
  environment:
   - ENV_1
   - ENV_2
As you can see I didn't specify any values for the environment variables. That's because I already have them defined in my host using direnv and Fig will just automatically use them. So in my case I have a local .envrc file that contains the following:
export ENV_1=value1
export ENV_2=value2
This allows me to set all my environment variables in one place. Here is the command I can use to start the container:
fig up

That's it! Much simpler than the corresponding docker run command.

Ideal World
What would be the best of both worlds is if Fig supported the docker run --env-file option and that it could read in a file containing export commands which is required by direnv. It seems support for the --env-file option in Fig is coming soon, so we are halfway there

Tuesday, November 18, 2014

Example Using Grails Promises

I was recently playing around with the Asynchronous Programming features in Grails using Promises, and wanted to share an example that went a little beyond a simple example. In case you are using an older version of Grails, the asynchronous features where added in Grails 2.3. While there are a lot of useful asynchronous features in Grails, for this article I'll only focus on using Promises. Promises are a common concept being introduced in many concurrency frameworks. They are similar to Java's java.util.concurrent.Future class, but like all things with Grails/Groovy, Grails has made them easier to use.

First, before showing you an example, go ahead and run grails console under an existing grails project. If you don't have one, install grails (see GVM) and run grails create-app. Using the grails console will allow you to quickly run these examples and experiment on your own.

Basic Example

import static grails.async.Promises.task
import static grails.async.Promises.waitAll

def task1 = task {
    println "task1 - starting"
    Thread.sleep(5000)
    println "task1 - ending"
}

def task2 = task {
    println "task2 - starting"
    Thread.sleep(1000)
    println "task2 - ending"
}

waitAll(task1, task2)
This would output:
task1 - starting
task2 - starting
task2 - ending
task1 - ending

More Complex Example
Let's say you wanted to list the states of 5 zip codes. Here is what that would look like if we did it synchronously:

["74172", "64840", "67202", "68508", "37201"].each { z ->
    println "getting state for zip code: $z"
    def response = new URL("http://zip.getziptastic.com/v2/US/$z").content.text
    def json = grails.converters.JSON.parse(response)
    println "zip code $z is in state $json.state"
}



And the output for that would look like:
getting state for zip code: 74172
zip code 74172 is in state Oklahoma
getting state for zip code: 64840
zip code 64840 is in state Missouri
getting state for zip code: 67202
zip code 67202 is in state Kansas
getting state for zip code: 68508
zip code 68508 is in state Nebraska
getting state for zip code: 37201
zip code 37201 is in state Tennessee

And here is what it would look like using Grails Promises to make it asynchronous:

import static grails.async.Promises.task
import static grails.async.Promises.waitAll

def tasks = ["74172", "64840", "67202", "68508", "37201"].collect { z ->
    task {
        println "getting state for zip code: $z"
        def response = new URL("http://zip.getziptastic.com/v2/US/$z").content.text
        def json = grails.converters.JSON.parse(response)
        println "zip code $z is in state $json.state"
    }
}

waitAll(tasks)

The asynchronous output would look like this:
getting state for zip code: 37201
getting state for zip code: 68508
getting state for zip code: 67202
getting state for zip code: 64840
getting state for zip code: 74172
zip code 74172 is in state Oklahoma
zip code 37201 is in state Tennessee
zip code 64840 is in state Missouri
zip code 68508 is in state Nebraska
zip code 67202 is in state Kansas

Each time you run the asynchronous version it will output a different order because the tasks are running asynchronously. The waitAll() method will block until all tasks complete.

Thanks to jeremydanderson for helping me figure out how best to use the collect method.