Moved to blog.jlo.fail

I've moved! I've switched blogging platforms (i.e. hashnode) and bought my own domain: https://blog.jlo.fail. Check out my first blog "Why a .fail domain?". It will be the home to all my new articles, so subscribe to my newsletter. I plan on keeping all the articles here on blogpost without migrating them to my new domain. Blogspot was great and it met the need of being a low-barrier way to start blogging. It was just time to buy my own domain and have better support for writing developer-focused articles that use markdown. Anyways, see ya there.

H2 URL with PostgreSQL mode and jOOQ

Just a quick code snippet to show my future self how I was able to finally get H2 working with a jOOQ query that in production hits a PostgreSQL database. I spent hours trying to get this to work. H2 kept complaining about not being able to find the public schema when our jOOQ query was attempting to execut a query like select * from "public"."customer". It's wasn't the schema it was having trouble with but the quotes. I figured this out by pulling up the H2 console and trying the query with and without quotes and without the quotes it worked. So then I figured out that I needed to somehow tell H2 this was for PostgreSQL. H2 has a MODE option that you can set.

The following is the code snippet that finally worked. I have not optimized it. The url value contains all my attempts to get this thing to work. I think it can definitely be trimmed down but that's out of scope for this blog. I'm just trying to save others some time including my future self.

import org.springframework.boot.jdbc.DataSourceBuilder;

DataSourceBuilder.create()
	.driverClassName("org.h2.Driver")
	.url("jdbc:h2:~/.h2/testdb"
		+ ";TRACE_LEVEL_SYSTEM_OUT=3"
		+ ";INIT=create schema if not exists public AUTHORIZATION sa\\"
		+ ";RUNSCRIPT FROM 'classpath:schema.sql'"
		+ ";DB_CLOSE_ON_EXIT=FALSE"
		+ ";SCHEMA=public"
		+ ";MODE=PostgreSQL"
		+ ";DATABASE_TO_LOWER=TRUE")
	.username("sa")
	.password("")
	.build();

Parameterized Tests: An Underused Technique

Over the past several years I've come to really value Parameterized Tests. I think they are an underused technique that should be considered more often. They make it easier to cover more of the test space and reduce cognitive load by being succinct.

So what are Parameterized Tests? Well they allow developers to run the same test multiple times over a set of different values. Here is a simple Java example using Java's Stream class with AssertJ:

Stream.of(null, "", " ", "false", "FALSE")
        .forEach(value -> assertThat(Boolean.valueOf(value)).isFalse());

Cucumber Inspiration

I first discovered this technique with Cucumber, which has the ability to run a scenario multiple times over a set of different values using a Scenario Outline. Prior to this I was in the habit of copying and pasting a Scenario and tweaking the Given and Then steps; or I wouldn't test all the sad path combinations in an effort to reduce test maintenance. Learning about the Scenario Outline changed my world. I was able to combine multiple scenarios into a single scenario and the result was more comprehendible and easier to maintain. For example, this is a common Scenario Outline we might use to test an API endpoint to ensure it validates the input. Previously, this would have been spread across multiple scenarios or not tested at all.

Scenario Outline: Should return a 400 when signing up with an invalid email address
  When I attempt to sign up with email "<email>"
  Then I should be returned a "400 Bad Request" status code

  Examples:
    | Email                       |
    | ${absent}                   |
    | ${blank}                    |
    | ${whitespace}               |
    | mailto:john.doe@example.com |
    | john.doe@example            |
    | john.doe@example.           |
    | john.doe@example.com.       |
    | john.doe@@example.com       |
    | john.doe@example..com       |
    | john doe@example.com        |

Recognizing Pattern

Once I discovered this concept and got comfortable with it, I explored ways to introduce it in lower level tests like unit and integration tests. But the trick was recognizing when to apply it. Basically, any time you find yourself copying and pasting a test and tweaking the arrange and assert statements you've probably got a good candidate for a Parameterized Test.

For example, let's say you have a class that determines if a number is a prime number or not. Before Parameterized Tests you might have been tempted to write something like the following:

public class PrimeNumberCheckerTestOldSchool {
    @Test
    public void shouldReturnTrueForPrimeNumber() {
        assertThat(PrimeNumberChecker.check(2)).isTrue();
    }

    @Test
    public void shouldReturnFalseForNonPrimeNumber() {
        assertThat(PrimeNumberChecker.check(6)).isFalse();
    }

    @Test
    public void shouldReturnFalseForAnotherNonPrimeNumber() {
        assertThat(PrimeNumberChecker.check(9)).isFalse();
    }

    @Test
    public void shouldReturnTrueForAnotherPrimeNumber() {
        assertThat(PrimeNumberChecker.check(17)).isTrue();
    }
}

Here you can see we are just repeating the test with different inputs and a different result. And since it's spread out across multiple tests it's hard to comprehend and this solution doesn't scale well if we want to add additional tests.

The ideal solution would be to define a sort of truth table, like Cucumber's Example table, that includes a combination of the inputs and expected result in a single test. This is where Parameterized Tests comes in.

JUnit 5

My preferred way to write Parameterized Tests is with JUnit 5 (see Parameterized Tests with JUnit 5). It's a big improvement over the JUnit 4 way of doing Parameterized Tests. We can rewrite the earlier example using the @ParameterizedTest and @CsvSource annotations (see other source annotations):

public class PrimeNumberCheckerTestJUnit5 {
    @ParameterizedTest
    @CsvSource({
            "2,  true",
            "6,  false",
            "9,  false",
            "17, true"
    })
    public void shouldCheckPrimality(final int number, final boolean expected) {
        assertThat(PrimeNumberChecker.check(number)).isEqualTo(expected);
    }
}

Here are some of the benefits of this approach:

• Easier to comprehend since it's not spread out across 4 test methods.
• Easier to add/remove additional scenarios.
• Easier to delete if the Method Under Test (MUT) is removed; all we will have to do is delete one test method!

To write Parameterized Tests with JUnit 5 you need to include the following dependencies. Note the JUnit 5 documentation states this is an "experimental" feature.

<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-engine</artifactId>
    <version>5.5.2</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-params</artifactId>
    <version>5.5.2</version>
    <scope>test</scope>
</dependency>

JUnit 4

A quick note about JUnit 4. If your team/project is still using JUnit 4, you can use both JUnit 4 and JUnit 5 simultaneously. JUnit 5 provides a gentle migration path. Just include the following dependency. I also recommend reading the Migration Tips.

<dependency>
    <groupId>org.junit.vintage</groupId>
    <artifactId>junit-vintage-engine</artifactId>
    <version>5.5.2</version>
    <scope>test</scope>
</dependency>

Now you are safe to write JUnit 5 tests without having to migrate all your existing JUnit 4 tests and be able to take advantage of the new @ParameterizedTest annotation.

Conclusion

Well hopefully I've convinced you of the power of Parameterized Tests and you'll look for the right opportunity to try them out. And keep in mind while these examples use Java, this technique should also be applicable to other languages. So do some research and see if your language provides them and if not maybe create your own. That's what we did before JUnit 5 was out and we weren't happy with the JUnit 4 way of doing Parameterized Tests.

Setting Environment Variables for Docker with Fig

For the past few months I've been playing around with Docker, and so far I've had a ton of fun. The documentation is excellent, and in one simple command you can start experimenting. After going through the tutorial, one of my first goals was to figure out the best way to create a Docker image for a spring-boot service. My initial goals were to make it easy to set environment variables, since our projects follow the twelve-factor methodology. One of the several factors we follow is the third factor (III. Config), which recommends storing the config in the environment. While this has many benefits, one of the downsides is the tendency to create a lot of environment variables because it's quick, easy, and well defined. This makes it difficult to configure, test, and run the service. But as we will see, Fig will not only make it easy to set environment variables, it will also provide many other benefits.

Docker Image
Let's first start with a pretend service called the logging service. It's a Java service created with spring-boot. Here is a basic Dockerfile:

FROM dockerfile/java:oracle-java8
COPY logging-service-0.1.0.jar /data/
EXPOSE 8080
CMD ["java", "-jar", "logging-service-0.1.0.jar"]

FROM - the base image I start with. In this case it's the dockerfile/java base image with the Oracle JDK 8 tag.
COPY - here I copy over the jar so it's present in the image
EXPOSE - this tells Docker the container will be listening on port 8080 at runtime
CMD - here I've defined a default command to run which will start the service

Next we need to build the image:
docker build --tag="jlorenzen/logging-service:v1" .

Docker Run
Now we could run our new service by executing this command:
docker run -dP jlorenzen/logging-service

That's great but let's imagine the logging-service requires the following environment variables: ENV_1 and ENV_2. Here is how you would run the service while also setting the environment variables:
docker run -dP -e ENV_1=value1 -e ENV_2=value2 jlorenzen/logging-service

That's a basic example, but you can image how nasty it could get if your service required a dozen or more environment variables. The docker run command also has some other nice options for setting environment variables. For example, when using the -e option, if you provide just the name like -e ENV_1 without a value, than that variables current value will be used. Or you can use the --env-file option to specify a file that contains a list of environment variables. While this all works, it's really not enjoyable having to remember all those options and commands. That is where Fig can help. And it not only helps us easily set environment variables, but it also makes creating containers simpler and reproducable by anyone anywhere.

Fig
Fig is basically a simple utility that wraps Docker making it easier to create and manage Docker containers. In our case we will use it to run our logging-service image and set the environment variables. Here is a simple fig.yml file:

logging-service:
  image: jlorenzen/logging-service
  ports: 
   - "8080"
  environment:
   - ENV_1
   - ENV_2

As you can see I didn't specify any values for the environment variables. That's because I already have them defined in my host using direnv and Fig will just automatically use them. So in my case I have a local .envrc file that contains the following:

export ENV_1=value1
export ENV_2=value2

This allows me to set all my environment variables in one place. Here is the command I can use to start the container:
fig up

That's it! Much simpler than the corresponding docker run command.

Ideal World
What would be the best of both worlds is if Fig supported the docker run --env-file option and that it could read in a file containing export commands which is required by direnv. It seems support for the --env-file option in Fig is coming soon, so we are halfway there

Example Using Grails Promises

I was recently playing around with the Asynchronous Programming features in Grails using Promises, and wanted to share an example that went a little beyond a simple example. In case you are using an older version of Grails, the asynchronous features where added in Grails 2.3. While there are a lot of useful asynchronous features in Grails, for this article I'll only focus on using Promises. Promises are a common concept being introduced in many concurrency frameworks. They are similar to Java's java.util.concurrent.Future class, but like all things with Grails/Groovy, Grails has made them easier to use.

First, before showing you an example, go ahead and run grails console under an existing grails project. If you don't have one, install grails (see GVM) and run grails create-app. Using the grails console will allow you to quickly run these examples and experiment on your own.

Basic Example

import static grails.async.Promises.task
import static grails.async.Promises.waitAll

def task1 = task {
    println "task1 - starting"
    Thread.sleep(5000)
    println "task1 - ending"
}

def task2 = task {
    println "task2 - starting"
    Thread.sleep(1000)
    println "task2 - ending"
}

waitAll(task1, task2)

This would output:

task1 - starting
task2 - starting
task2 - ending
task1 - ending

More Complex Example
Let's say you wanted to list the states of 5 zip codes. Here is what that would look like if we did it synchronously:

["74172", "64840", "67202", "68508", "37201"].each { z ->
    println "getting state for zip code: $z"
    def response = new URL("http://zip.getziptastic.com/v2/US/$z").content.text
    def json = grails.converters.JSON.parse(response)
    println "zip code $z is in state $json.state"
}

And the output for that would look like:

getting state for zip code: 74172
zip code 74172 is in state Oklahoma
getting state for zip code: 64840
zip code 64840 is in state Missouri
getting state for zip code: 67202
zip code 67202 is in state Kansas
getting state for zip code: 68508
zip code 68508 is in state Nebraska
getting state for zip code: 37201
zip code 37201 is in state Tennessee

And here is what it would look like using Grails Promises to make it asynchronous:

import static grails.async.Promises.task
import static grails.async.Promises.waitAll

def tasks = ["74172", "64840", "67202", "68508", "37201"].collect { z ->
    task {
        println "getting state for zip code: $z"
        def response = new URL("http://zip.getziptastic.com/v2/US/$z").content.text
        def json = grails.converters.JSON.parse(response)
        println "zip code $z is in state $json.state"
    }
}

waitAll(tasks)

The asynchronous output would look like this:

getting state for zip code: 37201
getting state for zip code: 68508
getting state for zip code: 67202
getting state for zip code: 64840
getting state for zip code: 74172
zip code 74172 is in state Oklahoma
zip code 37201 is in state Tennessee
zip code 64840 is in state Missouri
zip code 68508 is in state Nebraska
zip code 67202 is in state Kansas

Each time you run the asynchronous version it will output a different order because the tasks are running asynchronously. The waitAll() method will block until all tasks complete.

Thanks to jeremydanderson for helping me figure out how best to use the collect method.

Groovy Spring Bean for Static Factory

I started playing around with Grails again recently and ran into a problem trying to create a bean in the Grails resources.groovy file for a static factory. After several frustrating hours trying to find the right combination, I eventually stumbled upon an answer.

The factory I was trying to create a bean from was the JAX-RS Client API class ClientBuilder.newClient() which returns a Client object.

Here is what the bean definition looks like in my Grails resources.groovy file:

import javax.ws.rs.client.ClientBuilder

beans = {
    httpClient(ClientBuilder) { bean ->
        bean.factoryMethod = 'newClient'
        bean.destroyMethod = 'close'
    }
}

Then in your Grails service or controller you can autowire or inject the bean by doing the following:

class FooService {
    def httpClient

    def get(url) {
        return httpClient.target(url).request().get()
    }
}

Hope it helps the next person.

Disable Spring Boot Production Ready Services

For the past two months I've had the pleasure of working with The Lampo Group developing Hypermedia-Driven REST services using Spring Boot. Spring Boot makes it super simple to get started writing REST services. One of its biggest advantages is by default it embeds tomcat as the servlet container, allowing the developer to focus on other important things. Another great thing about Spring Boot is it includes the ability to easily enable what they call Production-Ready or Production-Grade Services. These services allow you to monitor and manage your application when its pushed to production and it's as easy as adding a dependency to your project. Unfortunately, it wasn't well documented on how to disable some of these services. But before I show you how to disable them, let me first show you how to enable them.

Enabling Spring Boot's Production-Ready Services
One of the reasons we wanted to enable some of the production-ready services was our target production environment was Amazon Web Services (AWS). As a part of that they support Elastic Load Balancing which allows one to configure a health check endpoint. It's basically an endpoint you configure in AWS that gets pinged to make sure the EC2 instance is up and running. As luck would have it, one of the services included in Spring Boot's production-ready services was a health endpoint.

To enable the production-ready services all you have to do is add a dependency to your project. If you are using maven all you have to do is add the following to your pom.xml.

<dependency>
    <groupid>org.springframework.boot</groupid>
    <artifactid>spring-boot-starter-actuator</artifactid>
    <version>1.1.4.RELEASE</version>
</dependency>

After you make the change to your pom.xml, just rebuild your project and you should be able to access http://localhost:8080/health and see something like this:

Not only does it add a health endpoint, but it also adds: autoconfig, beans, configprops, dump, env, info, metrics, mappings, shutdown (not enabled by default over HTTP), and trace. Like us, you might not want to expose all these endpoints in a production environment. In fact, all we wanted to enable was the health and info endpoints. The following will show you how to disable each service individually and how to re-enable them dynamically at runtime.

Disabling Spring Boot's Production-Ready Services
When you include the spring-boot-starter-actuator dependency in your project, it automatically exposes 11 different endpoints in your project. What I wanted to be able to do was disable all endpoints except health and info, but also have the ability to enable the other services at runtime via environment variables.

To disable some of the production-ready services add the following to your /src/main/resources file. These will be your default settings for your project.

endpoints.autoconfig.enabled=false
endpoints.beans.enabled=false
endpoints.configprops.enabled=false
endpoints.dump.enabled=false
endpoints.env.enabled=false
endpoints.health.enabled=true
endpoints.info.enabled=true
endpoints.metrics.enabled=false
endpoints.mappings.enabled=false
endpoints.shutdown.enabled=false
endpoints.trace.enabled=false

To enable/disable endpoints externally at runtime you can follow any one of these steps. Since our project is following the 12-factor app rules we needed the ability to enable/disable endpoints by setting environment variables. So for example, if I wanted to enable the metrics endpoint at runtime, I would set the following environment variable.

export ENDPOINTS_METRICS_ENABLED=true

After restarting Sprint Boot you can access the metrics endpoint at http://localhost:8080/metrics. Note, that as of version 1.0.1.RELEASE, you are unable to disable the mappings endpoint, but this was quickly fixed in 1.1.3.RELEASE.

In summary, I've been very impressed with how easy it is to work with Spring Boot and the production-ready services. In another article I'll cover how to use maven filtering, along with the git-commit-id-plugin, to display project information in the info endpoint.

Remember Target URL with Spring Security and Jasig CAS

I recently ran into a unique issue when combining Spring Security, Jasig CAS, and the Ozone Widget Framework (OWF). Basically, the original target URL was not being remembered with all the redirects from the CAS client filters, to CAS server, and then back to Spring Security. For example, if a user browsed to https://localhost/myapp/widget.jsp, they would be redirected to https://localhost/cas. Upon successful login, the user would be incorrectly redirected back to https://localhost/myapp instead of the original target URL.

SavedRequestAwareAuthenticationSuccessHandler almost worked
I thought I had found a solution by using Spring Security's SavedRequestAwareAuthenticationSuccessHandler, and that did work for single requests, but it did not work in an environment like OWF where multiple widgets are loaded simultaneously. The reason is because SavedRequestAwareAuthenticationSuccessHandler extracts the original target URL from the session, which is originally set by the ExceptionTranslationFilter. Since it used the session there could only be one original target URL. So if I had a workspace in OWF that loaded two different widgets from the same WAR (/myapp/widget1.jsp and /myapp/widget2.jsp), then there will only be one target URL saved in the session and both widgets would load the last saved URL, which is obviously not good.

There were several ideas out there, but I really didn't like any of them. Some required you to modify the CAS login form or return some weird javascript in one of the responses. What I wanted was the ability to preserve the original target URL within all the redirect URLs via a parameter. The only problem was, to my knowledge, nothing like this existed in Spring Security. So the following will show you how I extended Spring Security to preserve the original target URL via a parameter. As a simple example, here is a sequence of URLs that I was attempting to support (Note, the URL in the params are typically encoded but I kept them decoded for readability):

1. User browses to: https://localhost/myapp/widget.jsp
2. CAS Client filters redirect to: https://localhost/cas/login?service=https://localhost/myapp/j_spring_cas_security_check&spring-security-redirect=/widget.jsp
3. After authentication user is redirected to the URL defined in the service paramter https://localhost/myapp/j_spring_cas_security_check&spring-security-redirect=/widget.jsp which is monitored by Spring Security.
4. Once Spring Security does it's thing, it needs to redirect to the value in the spring-security-redirect parameter.

The following example works with Spring Security 3.1.4.RELEASE and Tomcat 7.0.21.

Spring Security Application Context File
What Spring Security example wouldn't be complete without some XML? The following is the application context file (note, for simplicity I have hardcoded the CAS urls and other values, but these would typically be read in from a properties file):

<sec:http use-expressions="true" entry-point-ref="casEntryPoint">
    <sec:intercept-url pattern="/css/**" access="permitAll" />
    <sec:intercept-url pattern="/images/**" access="permitAll" />
    <sec:intercept-url pattern="/scripts/**" access="permitAll" />
    <sec:intercept-url pattern="/**" access="hasRole('ROLE_USER')" requires-channel="https"/>
    
    <sec:custom-filter ref="casFilter" after="CAS_FILTER"/>
</sec:http>

<sec:authentication-manager alias="authenticationManager">
    <sec:authentication-provider ref="casAuthProvider" />
</sec:authentication-manager>

<bean id="casEntryPoint" class="com.example.security.cas.web.RememberCasAuthenticationEntryPoint">
 <property name="loginUrl" value="https://localhost/cas/login" />
 <property name="serviceProperties" ref="serviceProperties" />
 <property name="targetUrlParameter" value="spring-security-redirect" />
</bean>

<bean id="casAuthProvider" class="com.example.security.cas.authentication.RememberCasAuthenticationProvider">
 <property name="userDetailsService" ref="userService" />
 <property name="serviceProperties" ref="serviceProperties" />
 <property name="ticketValidator" ref="ticketValidator" />
 <property name="key" value="an_id_for_this_auth_provider_only" />
 <property name="targetUrlParameter" value="spring-security-redirect" />
</bean>

<!--
 - This is the filter that monitors all incoming requests for the url /myapp/j_spring_cas_security_check (sequence #7).
 - Sets the targetUrlParameter to redirect to the target URL after authentication.
 - Also sets the authenticationDetailsSource to our custom one in order to have access to the HttpServletRequest
 - in RememberCasAuthenticationProvider for ticket validation.
 - Tried setting the authenticationSuccessHandler to SavedRequestAwareAuthenticationSuccessHandler, and that works
 - for a single request, but in OWF if you have two widgets loading different URLs, it doesn't work because
 - it loads the saved url from the session object, so both widgets load the same url.
-->
<bean id="casFilter" class="org.springframework.security.cas.web.CasAuthenticationFilter">
 <property name="authenticationManager" ref="authenticationManager" />
 <property name="authenticationDetailsSource">
  <bean class="com.example.security.web.authentication.RememberWebAuthenticationDetailsSource"/>
 </property>
 <property name="authenticationFailureHandler">
  <bean class="org.springframework.security.web.authentication.SimpleUrlAuthenticationFailureHandler">
   <property name="defaultFailureUrl" value="/cas_failed.jsp" />
  </bean>
 </property>
    <property name="authenticationSuccessHandler">
        <bean class="org.springframework.security.web.authentication.SimpleUrlAuthenticationSuccessHandler">
            <property name="defaultTargetUrl" value="/" />
            <property name="targetUrlParameter" value="spring-security-redirect" />
        </bean>
    </property>
 <property name="proxyGrantingTicketStorage" ref="proxyGrantingTicketStorage" />
 <property name="proxyReceptorUrl" value="/secure/receptor" />
</bean>

CAS Entry Point
When dealing with Spring Security it all starts with the http entry-point-ref attribute. Here is the code for RememberCasAuthenticationEntryPoint:

package com.example.security.cas.web;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.jasig.cas.client.util.CommonUtils;

import org.springframework.security.cas.web.CasAuthenticationEntryPoint;

/**
 * Class which is responsible for remembering the original target url specified by the client.
 * Takes the original target url and appends that to the service param used by CAS.
 * This will later be used to redirect to the target URL after authentication.
 */
public class RememberCasAuthenticationEntryPoint extends CasAuthenticationEntryPoint {
    String targetUrlParameter = "spring-security-redirect";
    
    protected String createServiceUrl(final HttpServletRequest request, final HttpServletResponse response) {
        String service = this.serviceProperties.getService();
        
        String servletPath = request.getServletPath();        
        if (servletPath) {
            service += String.format("?%s=%s", this.targetUrlParameter, servletPath);
        }
        
        return CommonUtils.constructServiceUrl(null, response, service, null, this.serviceProperties.getArtifactParameter(), this.encodeServiceUrlWithSessionId);
    }
}

CAS Authentication Provider
Next up is the CAS authentication provider which we have defined as RememberCasAuthenticationProvider. This bean is given to the CAS Custom Filter under the alias authenticationManager. Here is the code for RememberCasAuthenticationProvider:

package com.example.security.cas.authentication;

import com.example.security.web.authentication.RememberWebAuthenticationDetails;

import org.jasig.cas.client.validation.Assertion;
import org.jasig.cas.client.validation.TicketValidationException;

import org.springframework.security.authentication.BadCredentialsException;
import org.springframework.security.authentication.UsernamePasswordAuthenticationToken;
import org.springframework.security.authentication.AccountStatusUserDetailsChecker;
import org.springframework.security.cas.authentication.CasAuthenticationProvider;
import org.springframework.security.cas.authentication.CasAuthenticationToken;
import org.springframework.security.cas.ServiceProperties;
import org.springframework.security.cas.web.CasAuthenticationFilter;
import org.springframework.security.core.Authentication;
import org.springframework.security.core.AuthenticationException;
import org.springframework.security.core.authority.mapping.GrantedAuthoritiesMapper;
import org.springframework.security.core.authority.mapping.NullAuthoritiesMapper;
import org.springframework.security.core.userdetails.*;

/**
 * CasAuthenticationProvider that tries to remember the original target url requested by the client.
 * The trick is having access to the HttpServletRequest in the authenticateNow() method.
 * This is accomplished via the RememberWebAuthenticationDetails class.
 * Since authenticateNow() was marked as private in CasAuthenticationProvider I had to also override
 * the authenticate() method. Created spring security jira https://jira.springsource.org/browse/SEC-2188
 * to address making authenticateNow protected so we don't have to duplicate authenticate().
 */
public class RememberCasAuthenticationProvider extends CasAuthenticationProvider {

    UserDetailsChecker userDetailsChecker = new AccountStatusUserDetailsChecker();
    ServiceProperties serviceProperties;
    GrantedAuthoritiesMapper authoritiesMapper = new NullAuthoritiesMapper();
    String targetUrlParameter = "spring-security-redirect";

    /**
     * Straight copy and paste from CasAuthenticationProvider
     * @see spring security jira https://jira.springsource.org/browse/SEC-2188
     */
    public Authentication authenticate(Authentication authentication) throws AuthenticationException {
        ...
    }

    /**
     * The service URL used in ticketValidator.validate() needs to match the service URL given to CAS when
     * the ticket was granted.
     */
    protected CasAuthenticationToken authenticateNow(final Authentication authentication) throws AuthenticationException {
        try {
            String targetPath = this.getTargetPath(authentication.getDetails());
        
            def service = String.format("%s?%s", serviceProperties.getService(), targetPath);
            
            final Assertion assertion = this.ticketValidator.validate(authentication.getCredentials().toString(), service);
            final UserDetails userDetails = loadUserByAssertion(assertion);
            userDetailsChecker.check(userDetails);
            return new CasAuthenticationToken(this.key, userDetails, authentication.getCredentials(),
                    authoritiesMapper.mapAuthorities(userDetails.getAuthorities()), userDetails, assertion);
        } catch (final TicketValidationException e) {
            throw new BadCredentialsException(e.getMessage(), e);
        }
    }
    
    /**
     * Extracts the original target url form the query string.
     * Example query string: spring-security-redirect=/widget.jsp&ticket=ST-112-RiRTVZmzghHO7az5gpJF-cas
     */
    protected String getTargetPath(Object authenticationDetails) {
        String targetPath = "";
        
        if (authenticationDetails instanceof RememberWebAuthenticationDetails) {
            RememberWebAuthenticationDetails details = (RememberWebAuthenticationDetails) authenticationDetails;
            String queryString = details.getQueryString();
            
            if (queryString) {
                int start = queryString.indexOf(this.targetUrlParameter);
                if (start >= 0) {
                    int end = queryString.indexOf("&", start);
                    if (end >= 0) {
                        targetPath = queryString.substring(start, end);
                    } else {
                        targetPath = queryString.substring(start);
                    }
                }
            }
        }
        
        return targetPath;
    }
}

Authentication Details Source
Since I needed access to the requests query string in the RememberCasAuthenticationProvider.getTargetPath() method, I needed to provide a different WebAuthenticationDetails class. This was accomplished by setting the authenticationDetailsSource property on the CAS Filter. Here is the code for RememberWebAuthenticationDetailsSource:

package com.example.security.web.authentication;

import javax.servlet.http.HttpServletRequest;

import org.springframework.security.authentication.AuthenticationDetailsSource;
import org.springframework.security.web.authentication.WebAuthenticationDetails

public class RememberWebAuthenticationDetailsSource implements AuthenticationDetailsSource<HttpServletRequest, WebAuthenticationDetails> {

    public WebAuthenticationDetails buildDetails(HttpServletRequest request) {
        return new RememberWebAuthenticationDetails(request);
    }
}

Here is the code for RememberWebAuthenticationDetails:

package com.example.security.web.authentication;

import javax.servlet.http.HttpServletRequest;

import org.springframework.security.web.authentication.WebAuthenticationDetails;

public class RememberWebAuthenticationDetails extends WebAuthenticationDetails {
    private final String queryString;

    public RememberWebAuthenticationDetails(HttpServletRequest request) {
        super(request);
        
        this.queryString = request.getQueryString();
    }
    
    public String getQueryString() {
        return this.queryString;
    }
}

Summary
That pretty much does it. I know it's a lot of code, but once I got familiar with Spring Security it really wasn't that much. Once you get this all configured it should just work and the users original target URL can be seen when getting redirected to the CAS login page and then after authentication, the user should be redirected back to the original target URL. And in an OWF environment where multiple URLs from the same WAR are being loaded simultaneously this solution seems to work.

I do want to mention that we have since noticed that there are conditions where the entire URL is not remembered. Simple URLs like /myapp/widget.jsp work great, but REST URLs like /myapp/api/events/1 are not completely preserved. Nor are params remembered either like /myapp/widget.jsp?id=1. At this time we really don't need that capability but I don't think it would be hard to add it. Most of the work has already been done.

One other thing. I think this experiment begs the question: why doesn't Spring Security already support something like this out of the box? It would seem like a very common use case. While researching it seems there might be a reluntance to support this feature due to security concerns of a malicious user gaining access to a resource they are not authorized for. I didn't do super extensive testing, but in the testing I did do, I was not able to gain access to resources I was not authorized for. Perhaps when/if I come back to these classes and add support for the noted issues above I might submit a patch back to Spring Security so this can get incorporated into Spring Security.

Setting Gradle home directory and proxy in Jenkins

Real quick. Spent the past few hours working around some nasty issues with gradle and jenkins. It seems due to a bug, the jenkins gradle plugin puts the dependency/artifact cache under the jobs workspace. This really isn't a good idea as every job would then download all of the projects artifacts taking up large amounts of space. At the same time, I also needed to setup the proxy information for gradle, which sadly doesn't reuse the jenkins proxy information.

I was able to finally figure out a good place to define the gradle user home and proxy information in a single place to prevent each job from having to define it.

Go into Manage Jenkins > Configure System. Under Global properties check Environment variables and fill in the following for name and value:

name: GRADLE_OPTS
value: -Dgradle.user.home=/home/tomcat/.gradle -Dhttp.proxyHost=101.10.10.10 -Dhttp.proxyPort=3128

For the gradle.user.home property, I tried using ~/.gradle, but that didn't work which means most likely my $HOME environment variable was not set for whatever reason. My guess is it has something to do with all the troubles I've had lately using the bitnami jenkins amazon ami. I also tried setting the environment variable GRADLE_USER_HOME, but that didn't seem to work. Either way, hopefully this will help others.

Resource Filtering with Gradle

My team has recently started a new Java web application project and we picked gradle as our build tool. Most of us were extremely familiar with maven, but decided to give gradle a try.
Today I had to figure out how to do resource filtering in gradle. And to be honest it wasn't as easy as I thought it should be; as least coming from a maven background. I eventually figured it out, but wanted to post my solution to make it easier for others.

What is Resource Filtering?
First, for those that may not know, what is resource filtering? It's basically a way to avoid hard coding values in files and make them more dynamic. For example, I may want to display the version of my application in my application. The version is usually defined in your build file and this value can be injected or replaced in your configuration file during assembly. So I could have a file called config.properties under src/main/resources with the following content:
application.version=${application.version}. With resource filtering the ${application.version} value gets replaced with 1.0.0 during assembly, then my application can load config.properties and display the application version.

It's an extremely valuable and powerful feature in build tools like maven and one that I took advantage of often.

Resource Filtering in Gradle
With this being my first gradle project, I needed to find the recommended way to enable resource filtering in gradle. My first problem I had to figure out was where to define the property. In maven this would typically be defined in the project's pom.xml file as a maven property:

<properties>
    <application.version>1.0.0</application>
</properties>

For gradle the appropriate place seemed to be the project's gradle.properties file. So you would add the following to your project's gradle.properties file (Note, I'm not suggesting you would hardcode the modules version in the gradle.properties file. Obviously the value would be derived from the version property in your project. I'm just using this for a simple example):

application.version=1.0.0

The next, and most difficult, problem I had to track down was how to actually enable resource filtering. I was hoping to just set some enableFiltering option and define the includes/excludes list, but that doesn't seem to be the case (extra tip: don't do filtering on binary files like images). I did find some resources online, but this one seemed to be the best approach. So you will need to add the following to your build.gradle file:

import org.apache.tools.ant.filters.*

processResources {
    filter ReplaceTokens, tokens: [
        "application.version": project.property("application.version")
    ]
}

Next you need to update your resource file. So put a config.properties file under src/main/resources and add this:

application.version=@application.version@

Note, the use of @ instead of ${}. This is because gradle is based on ant, and ant by default uses the @ character as the token identifier whereas maven uses ${}.

Finally, if you build your project you can look under build/resources/main and you should see a config.properties file with a value of 1.0.0. You can also open up your artifact and see the same result.

Dot notation
One thing to note is I typically use a period or dot to separate words for properties: application.version instead of applicationVersion. So you will notice the surrounding quotes around "application.version" in the build.gradle file. This is required as failing to surround the key by quotes will fail the build. Probably because groovy's dynamic nature thinks you are traversing an object.

Overriding
I also investigated the best approach to overriding properties in gradle, as this appeared to be slightly different then how it's done in maven. In maven, properties can be overridden by properties defined in the user's setting.xml file or on the command line with the -D option. To override application.version in gradle on the command line I had to run the following:

gradle assemble -Papplication.version=2.0.0

If you want to override it for all projects you can add the property in your gradle.properties file under /user_home/.gradle.

Also, if you are overriding the value via the command line and your property value contains special characters like a single quote, you can wrap the value with double quotes like the following to get it to work:

gradle assemble -Papplication.version="2.0.0'6589"

Summary
Well I hope this helps and if anyone from the gradle community sees a better way to perform resource filtering I'd love to hear about it. I'd also like to see something as important as resource filtering becoming easier to perform in gradle. I think it's crazy having to add an import statement to perform something so simple.

Configuring CAS Externally using Spring Import

Over the past few months, my team has been working on integrating the Jasig CAS (Central Authentication Service) framework with our application. CAS is a feature rich enterprise Single Sign On (SSO) service. Most importantly, it's very flexible. We need this flexibility because we have several customers, each with different authentication requirements:. Some with and without Active Directory, and others needing PKI (X.509) authentication. Beyond that, we also need to support light weight options for local development. So the problem we had to solve was how to configure CAS externally so we could dynamically change authentication mechanisms without having to rebuild CAS.

Let me first explain that the recommended way to extend CAS is using maven overlays. It's real simple to modify the default behavior in CAS by copying the files from the CAS WAR into your own WAR directory. For example, most of the authentication configuration exists in /WEB-INF/deployerConfigContext.xml. To change the default authentication behavior, you copy this file to your WARs /WEB-INF directory and modify it to fit your requirements. This is the file that defines all the authentication handlers for things like LDAP (BindLdapAuthenticationHandler), X.509 (X509CredentialsAuthenticationHandler), and my personal favorite the simple username equals password (SimpleTestUsernamePasswordAuthenticationHandler) authentication handler.

So the real problem became how can we enable the SimpleTestUsernamePassword authentication handler for local development and test servers, while disabling it for production systems? We wanted to avoid creating multiple CAS WARs containing configuration for their specific purpose: cas-ldap, cas-x509, cas-simple. Also, we wanted to avoid having a single CAS WAR that contained all methods of authentication. Ironically the solution was really simple and is actually used in other places in CAS.

Spring Import
The solution was spring imports (section 3.2.2.1). This was a feature I was not previously familiar with, but now think is one of the coolest features in spring because it lets you change the behavior of the system without having to rebuild. Spring lets you load in other bean xml files via the spring import tag. And since CAS uses spring, it was easy to modify CAS to be configured externally.

Configuring CAS Externally
First, I assume you have already setup your maven project that is performing the maven overlay. Once you have built your project do the following:

• copy
  /cas/target/war/work/org.jasig.cas/cas-server-webapp/WEB-INF/deployerConfigContext.xml
  to
  /cas/src/main/webapp/WEB-INF
• You'll also need to copy this same file to your application servers classpath for your application. For us this would be under the jboss conf directory: /jboss/server/default/conf. I also renamed this file to custom-deployerConfigContext.xml to avoid confusion.
• Open the file /cas/src/main/webapp/WEB-INF/deployerConfigContext.xml
• Remove all the bean tags in between the beans tag. This leaves you with nothing but the beans tag with nothing in between.
• Paste in the following spring import tag:
  <import resource="classpath:custom-deployerConfigContext.xml"/>
• Save your changes, rebuild and redeploy your CAS WAR

That's it. Now you can edit the custom-deployerConfigContext.xml and add/remove the SimpleTestUsernamePasswordAuthenticationHandler, or any other authentication handlers, without having to rebuild your CAS WAR.

Final Thoughts
Hopefully, you learned a little about how to configure CAS externally and a way to use springs import ability. For the most part this works great, but it isn't 100% fool proof. There are several CAS configuration files, so for the more complex authentication scenarios, you might have to modify the cas-servlet.xml or login-webflow.xml files. For example, if you want to do X.509 authentication, you have to include a bean in the cas-servlet.xml file and modify the login-webflow.xml file. But you could apply the same concepts to these files as well as cas-servlet.xml is just another spring file and login-webflow.xml is a spring webflow file that supports the bean-import tag.

What's missing from our REST Services?

While reading the excellent book RESTful Web Services I discovered something. A technique used on the web since its beginning. Twitter and Facebook seem to use it. Yet it was absent in the REST Services I had been developing and if I had to guess most REST developers aren't including it either. What is so valuable the Internet would be useless without it? The answer is hypermedia.

Hypermedia is the technical term used to describe how web pages (resources) are linked or connected. The author of RESTful Web Services calls it connectedness: how the server guides the client from one application state to another by providing links. A good example of a well-connected application is Wikipedia. It's very powerful when you could pick a random page and click through every entry on Wikipedia without ever editing your browsers URL. Performing a search on Google is another great example, as it wouldn't be very useful if the search result didn't include any links. Without these links, the client must know and create predefined rules to build every URL it wants to visit.

So what do links on a website have to do with REST Services? REST was built on the foundation of the web and just because your not returning HTML doesn't mean you shouldn't create relationships between your resources. In fact, it's amazing how powerful embedding links in your responses can be. For instance, it prevents tight coupling between your clients and services as the clients don't have to construct or even know the exact URLs because the services are providing them.

Let's use Twitter as an example. Assume the following is the URL to get the 20 most recent tweets for my account:

http://api.twitter.com/1/statuses/user_timeline.json?screen_name=jlorenzen

And here is a shortened fictitious response:

[
    {
        "text": "Finished watching Battlestar Galatica"
        "id": "55947977415667712"
        "url": "http://api.twitter.com/1/statuses/show/55947977415667712.json"
    },
    {
        "text": "Started watching Battlestar Galatica"
        "id": "45947977415667712"
        "url": "http://api.twitter.com/1/statuses/show/45947977415667712.json"
    }
]

Notice each tweet includes a direct URL. Now clients can use that URL value verses constructing it, and if it changes in the future, clients don't have to make any changes.

Example using Jersey
So what is the best way to include links in your REST Services? Since I use Jersey on a daily basis, I'll go ahead and show an example of how to embed links in your responses using Jersey. Most likely, any REST framework is going to provide the same kind of features.

Using the twitter example above, the User Status Service with links may look something like this (using groovy):

import javax.ws.rs.GET
import javax.ws.rs.Path
import javax.ws.rs.Produces
import javax.ws.rs.QueryParam
import javax.ws.rs.core.Context
import javax.ws.rs.core.UriInfo

import com.test.UserStatuses

@Path("/statuses")
class StatusesResource {
 
    @Context 
    UriInfo uriInfo

    @GET
    @Path("/user_timeline")
    @Produces(["application/json", "application/xml"])
    def UserStatuses getUserTimeline(@QueryParam("screen_name")String screen_name) {
        def statuses = getStatuses(screen_name)
        
        statuses.each {
            it.url = "${uriInfo.baseUri}statuses/show/$it.id"
        }
        
        return new UserStatuses(statuses: statuses)
    }
}

In this simple example, Jersey injects the UriInfo object which we use to get the baseUri of the request. It's really that simple.


Potential Issues
Well for some it may not be that simple. For example, we discovered an issue when using a reverse proxy (apache httpd). In our production environments, we typically setup apache on port 80 to proxy a localhost JBoss on port 8080. Unfortunately, in this setup the UriInfo.getBaseUri() returns localhost:8080 and not the actual original URL the client used; which is obviously not good. Now if you don't use a reverse proxy then no worries. However, if you do or might potentially in the future, a easy solution seems to be to set the Apache Proxy module option ProxyPreserveHost to On. Setting this to On and restarting Apache seems to fix the issue.

JSONView Firefox Plugin
Once you've started embedding URLs in your REST responses, you might find it useful to install the JSONView firefox plugin. It's got some really slick features like formatting the JSON and creating clickable links for URLs.

How to Change Extjs PieChart Colors

When using Sencha's, or extjs, charting capability, most likely your going to want to change the default color scheme. I was faced with this issue today and it's not documented as well as you'd expect. I had to piece together a few articles and I'm still not 100% it's the right way as it's using an undocumented config option. But I wanted to document how I did get it to work to same others some time.

Sencha is very well documented and their examples are great. Here is the Pie Chart example we are going to be updating (using version 3.2.1). For those that don't know, extjs charts are based off of Yahoo's YUI charts which is also requires Flash. Not only is it pretty simple to create charts with extjs, but we already are using extjs so we decided to prototype some charts using it.

Here is the code that produces a simple basic PieChart:

new Ext.Panel({
    width: 400,
    height: 400,
    title: 'Pie Chart with Legend - Favorite Season',
    renderTo: 'container',
    items: {
        store: store,
        xtype: 'piechart',
        dataField: 'total',
        categoryField: 'season',
        extraStyle:
        {
            legend:
            {
                display: 'bottom',
                padding: 5,
                font:
                {
                    family: 'Tahoma',
                    size: 13
                }
            }
        }
    }
}); 

This produces the following PieChart.

To change the default colors provide a series config option:

new Ext.Panel({
    width: 400,
    height: 400,
    title: 'Pie Chart with Legend - Favorite Season',
    renderTo: 'container',
    items: {
        store: store,
        xtype: 'piechart',
        dataField: 'total',
        categoryField: 'season',
        series: [{
            style: {
                colors: ["#ff2400", "#94660e", "#00b8bf", "#edff9f"]
            }
        }],
        extraStyle:
        {
            legend:
            {
                display: 'bottom',
                padding: 5,
                font:
                {
                    family: 'Tahoma',
                    size: 13
                }
            }
        }
    }
});

The chart it produces may not look the most appealing but at least we figured out how to change the colors.

Pie Chart with New Colors

Again, I'm not certain this is the best way to accomplish this as the series config option isn't documented in 3.2.1. But by piecing together this PieChart question and this article, I was able to figure it out.

Missing Office Equipment Prank

Here is Part 2 of my series where I reminisce about the good times I had at Williams. As I stated previously, I came across these printed off emails in our attic while moving to our new home. I'm so glad I printed them off. #goodtimes

Anyway, this was a prank Keith Stanek, Josh Guthman (Guthy), and myself pulled on Michael Brotherman (Are you Miking Me?). Before showing you the email though I need to set the stage. The year was 2003, and our company had just gone through 3 rounds of layoffs. Morale was pretty low. We worked on the 32nd floor of the 52 story BOK building in downtown Tulsa, Oklahoma. During fire drills we had to take the stairs down a floor. During one of the fire drills we noticed the entire 31st floor was void of humans, but a lot of very nice unused office furniture and white boards were still present. We joked around about repurposing some of the nicer equipment, but nothing ever come of it. Until one morning, we noticed Mike had a new chair. A very nice new chair. We gave him junk about it all day, but decided a prank would be better. So we had Josh Guthman (Guthy) write us up a believable email that Keith and I would spoof using the Facilities email address. Here it is:

From: Facilities Services-Tulsa
Sent: Thursday, March 13, 2003 7:39AM
Subject: Missing Office Equipment

During a recent audit for unused office equipment we discovered black reclining office chairs were missing from several of the floors in the BOK tower. Upon further investigation we discovered those chairs had been procured by current employees looking to upgrade from their existing chairs. While facilities appreciates your desire to utilize existing assets instead of requisitioning new ones, please remember that Williams is undergoing a cost savings campaign at this time and that all unused equipment needs to remain on its respective floor for proper accounting and redistribution either to the lessee or to an employee in need. If we have not already reacquired your chair, please return it to the floor from which it came by the end of business Friday, March 14.

Mike smelled it out, but noone ever confessed, and I'm pretty sure deep down, even though his gut was telling him it was a prank, he still didn't want to take the chance in the possibility that it might have been true. The best part about it was the morning we sent the email, Mike and I had a group breakfast meeting with one of the Senior Executives. As we were going down the elevator, he mentioned to me that he was going to bring it up during the meeting, and with a complete straight face I called his bluff and told him that I think he should, knowing that he wouldn't anyway; and if he did it would only make the prank that much better.

Mike returned the chair.

Real man of Genius

I usually keep this blog professional, but I'm in the process of moving and came across some emails that I just have to share and they are too long to post on facebook. All 3 emails where from my time at The Williams Company in Tulsa, OK where I had the privilege to work with some great friends. People like Keith Stanek, Michael Brotherman (Are you Miking me?), Jason Randall, Josh Guthman (Guthy), Erin Nylund, Becca Fairchild, Jennifer Brandt, and a host of others. I was so fortunate to work with these people and I'm not sure it could ever be duplicated. We had a ton of fun together. During off hours we played a lot of cards (Rook mainly), Starcraft I, and lots and lots of Halflife. Over time, I got to be pretty good at Halflife (HL). At some point we started playing 2 (Keith and Mike) on one and I still would usually win.

Now to the email. Wednesday August 20, 2003 at 9:45 AM, Michael Brotherman emailed Keith and me his Real man of Genius in my honor. It was done just like the Real Men Of Genius commercials. Here it is in it's original form:

Mr real man of genius...
We solute you, Mr. Total Conqueror in HL.
You who drops bombs that are not in the bathroom.
Who shoots your double barrel in our face.

Your eyes keep us from getting caught,
But your play keeps us from coming back.

Here's to you...Mr crossbow no miss,
Mr. Ray gun who makes it no fun.

We solute you.


My Favorite weapon was the crossbow. Oh good times. Stayed tuned. I've got 2 more coming.

Hash tags in Commit Comments

I've been using Yammer, Twitter, and Facebook consistently for awhile now. One of the things I really like are hash tags, where yams or tweets include additional meta information in the comment such as #groovy, #hudson, or #maven. One of the main purposes of hash tags, is it allows others to subscribe to an area of interest verses subscribing to hundreds of individual people. Another purpose it serves, is determining interest value; sort of like a subject heading. Since hash tags are typically at the end of a tweet or yam, I usually read the end first before I commit to reading the whole yam or tweet. I don't follow a ton of people yet, but I do consume a lot of information in a day and in order to find the good I have to wade through the bad. Using hash tags aides in this process.

I also think hash tags could help in another area: commit comments. It's something important I've mentioned before, and I think hash tags can be useful in commit comments even if there aren't any tools yet to mash it up. A few days ago, our of habit I accidentally started including some high level hash tags in an svn commit comment and it occurred to me that it might be useful to others, if not myself in 6 months. If we find hash tags useful in yams and tweets, why not commit comments?

Including tokens in commit comments isn't new. In fact, we already include a Jira number in most of our commit comments and this allows us to view all the commits for a Jira issue. There is even a Jira plugin that allows you to perform actions by specifying hashes in commit comments. For example, if I want to resolve a Jira I can include #resolve in my commit comment, and Jira will automatically Resolve that Jira. And don't feel like you can't include the #resolve tag only if your using that jira plugin. I could see value in seeing a #resolve tag in the final commit of a Jira.

As an example, here is the exact commit comment I used that includes some hash tags for geoserver and installer.

"Jira: AC-4207. Got the filtered geowebcache.xml file correctly moved to the production and staging data directories. These files point to localhost with the correct geo port and stage geo port. Also commented out some fixpath.cmd lines to get the installer to work. Finally, I also change the ProcessPanel to not have a condition: changed to . This should allow us to be really selective in what we install and still allow the process panel to run, whereas before it wasn't running. #geoserver #installer"

Now the really cool part is if someone else in the near future notices an issue with geoserver in our installer, this comment will stick out more than a comment without those hashes.

Another cool thing that could be done is a team subscribing to certain hash tags in the svn commit emails. For example, someone responsible for peer reviewing all DAO changes could subscribe to a hash like #dao. Then when developers are modifying DAO's all they need to do is include the #dao tag.

I guess what I am saying is perhaps we could also benefit from putting extra hash tags in our commit comments. My brain has already been trained to read them so personally I think it's useful.

Restoring a Clonezilla Image using VirtualBox

Ubuntu 10.04 has been out for a few months, and I'm still on 09.10. I have had some success in the past upgrading, but I still prefer doing fresh installs. I guess it comes from my windows days, when an occasional fresh install was good for the computer soul. However, this time I'm also starting a new project at work doing .net instead of java, and I really wanted the ability to "come back" to my old setup. Basically, I wanted to convert my host machine to a virtual one or what's called P2V (Physical to Virtual). I tried VMware Converter but didn't get very far. With some advice from several co-workers though, I did come up with a method that did work and it was fairly easy.

The basic steps are:

• Use Clonezilla to Save a Disk Image to a external USB drive. This essentially clones my host machine, where I can restore it later. My hard drive is around 120GB, so I put the image on my 500GB external USB drive. This took about 1.5 hours.
• Create a new virtual machine on another external USB drive. The nice thing about using Clonezilla, is for this step you can use VMware or VirtualBox. I used VirtualBox. Obviously, you can't create the virtual machine on your laptop because you don't have enough space. And you can't use the same USB drive because when restoring, Clonezilla needs it to be unmounted. So instead I used another external USB drive.
• Start the new virtual machine and boot up Clonezilla to begin restoring the image. You need to change the mode because the default view doesn't work very well when restoring. So at the Clonezilla menu, choose "Other modes of Clonezilla live". Then choose "Clonezilla live (Safe graphic settings, vga=normal)".
• When you get to the point where Clonezilla needs to point to the external USB drive that contains the Clonezilla image, remember to enable the USB drive in VirtualBox. To do this, go to the Devices menu option in your virtual machine and select USB Devices and check the appropriate USB drive. Restoring my 120GB image took about 24 hours so make sure to do it when you have time.
• Once Clonezilla has finished restoring the image your ready to poweroff the virtual machine, remove the clonezilla CD, and restart.

I still had a few adjustments to make in order to get it to work. When I first started my virtual machine, it complained about not having PAE (Physical Address Extension) enabled. I enabled PAE in ubuntu about a month ago so I could use all 4GB of RAM. Fixing it was easy. Under your machines settings go to System and click on the Processor tab. Check the "Enable PAE/NX" checkbox and restart.

Once it booted up, it complained about my graphics configuration. I tried selecting "Reconfigure Graphics", but that didn't work. Instead I was able to get passed it by selecting "Run in low graphics for one session". This allowed me to finish booting where I installed the virtualbox guest additions which seemed to solve the graphics issue.

That is all there is to it. It was all rather easy. Now I can install Ubuntu 10.04 and have the ability to go back to my previous development environment. I could also see lots of different use cases for this. Combined with the ability to clone virtual machines, all your virtual needs are met.

Upgrading to Maven 3

I've been playing around with maven 3 lately on our legacy maven 2 multi-module project via mvnsh. Like advertised, maven 3 is backwards compatible with maven 2. In fact, most everything worked out of the box when switching to maven 3. In this post, I'm going to highlight the required and currently optional items I changed so you can start preparing to migrate your project to maven 3. But first, what's so special about maven 3 and why would you upgrade? Polyglot maven, mvnsh, and improved performance (50%-400% faster) are just a few. And since it's so easy to migrate to maven 3, you really don't have any excuses.

Currently, I build our project using maven 2.2.1. This article was tested with mvnsh 0.10 which includes maven 3.0-alpha-6. The current release of maven 3 is 3.0-beta-1, while maven 3.1 is due out Q1 of 2011.

Profiles.xml no longer supported
I haven't really figured out the reasoning, but it doesn't really matter; maven 3.0 no longer supports a profiles.xml. Instead you place your profiles in your ~/.m2/settings.xml. Some of our database processes and integration tests require properties from our profiles.xml. It was simple to solve by just moving my profiles to my settings.xml and everything worked.

Upgrade GMaven Plugin
We depend pretty heavily on the gmaven plugin for testing, simple groovy scripts, and some ant calls. In order to build some modules I had to upgrade gmaven. The current version we were using was 1.0-rc-3. Our projects built perfectly after changing it to org.codehaus.gmaven:gmaven-plugin:1.2.

${pom.version} changing to ${project.version}
Here maven 3 kindly warned me that uses of the maven property pom.version may no longer be supported in future versions and should be changed to project.version. My modules still built, but thought it was nice of maven to inform me of the potential change.

Version and Scope Issues
We had a few places where we needed to define a dependency version and another place where we shouldn't have defined a scope. Both instances prevented maven 3.0 from building our modules, but fixing them was easy. The first instance was we defined a version for a plugin in the pluginManagement section, but maven 3 required it also where it was used in the reporting plugin section. Not exactly sure about this one, ideally you would only define your plugin versions in the pluginManagement section but oh well.

We had some WAR projects using jetty. In the jetty plugin definition we had a dependency on geronimo and had defined a scope of provided. Maven 3 complained about it and since it's really not necessary, just removing it fixed the issue.

modelVersion
Maven 3.0 kept warning about using ${modelVersion} instead of ${project.modelVersion}. I was still able to build though, so my guess is the value for modelVersion, 4.0.0, most likely will change when maven 3.1 comes out.

Weird Surefire Output
This wasn't necessarily an issue with the surefire plugin, but I wanted to comment about it's output when tests failed as I thought it might have been a maven 3 issue. Below is a screenshot of the output when you have failed tests. At first I thought it was a maven 3 issue, but I built the same project using the same commands with maven 2.2.1 and got the same test failures. Hopefully, they can clean this type of thing up, because I could image lots of people getting confused.

Failed test output

That's essentially it. Happily, there really wasn't much required to change, which goes to show the great lengths the maven team has gone through to ensure backwards compatibility. Finally, here is a Compatibility Notes maven has provided on the subject of migrating maven 2 projects to maven 3.

My First Groovy DSL

It's no secret I'm a groovy homer. I love it. One of the things that makes using groovy so fun, is it's syntax. Being able to get the contents of a file by just saying new File("/home/james/test.log").text is refreshing compared to it's java counterpart. Another thing that makes groovy enjoyable is it's ability to support Domain Specific Languages (DSL). MarkupBuilder is a great example. With Groovy, you can create simple or very complex DSLs for your purposes. To my knowledge there are a few ways you can create your own DSL: extending BuilderSupport or using methodMissing/propertyMissing. In my opinion, extending BuilderSupport is more involved while methodMissing/propertyMissing is kind of the poor man's way of creating a DSL.

Up to this point though, I've never actually came across a good use case for creating a DSL until this past week. We have a large set of automated tests that run against our REST Services. Since our application is now multi-tenant, all of our tests need a valid organization (tenant). In our case, an organization contains multiple roles and locations. Each test has different requirements on the types of organizations it needs. Some might need 2 unique organizations, while another might need an organization with at least 2 roles and 2 locations. It was this use case that I thought a groovy DSL would fit perfectly.

My end goal was to have something like this:

def orgs = OrganizationService.getOrganizations().withRoles().withLocations()

This would return a list of organizations that had at least 1 role and 1 location. The nice thing about this DSL is it's scalable. Meaning, if we add new lists of information to an organization, we won't have to update our class. Also, an important feature, is the method name Roles and Locations correlate to the JSON named arrays of the organization. So my JSON looks something like this:

{"organizations": {"name": "James", "roles": ["R1", "R2"], "locations": ["Tulsa", "Omaha"]}}

When writing my DSL I decided to go the poor man's way and use the methodMissing approach combined with the @Delegate annotation. Here it is:

import net.sf.json.JSONArray

class OrganizationFilterArray {
    @Delegate private JSONArray array
    
    OrganizationFilterArray(array) {
        this.array = array
    }
      
    def methodMissing(String name, args) {        
        if (name.startsWith("with")) {
            def length = (args.length == 0) ? 1 : args[0]
            def arrayName = name[4..5].toLowerCase() + name[6..-1]
            
            return filterByLength(arrayName, length)
        } else {
            throw new MissingMethodException(name, this.class, args)
        }
    }
    
    private filterByLength(listName, length) {        
        def filteredArray = array.findAll {
            it."$listName"?.size() >= length
        }
        
        return new OrganizationFilterArray(filteredArray)
    }
}

I could have just as easily extended JSONArray since it's not final, but I was following the @Delegate guide initially and just thought it was an interesting alternative. The big key here is how I used methodMissing to support an infinite amount of possibilities with how to filter an organization. Everything else I think is pretty self explanatory. When it comes across a method that is missing, withRoles(), it calls my methodMissing method. From there I filter out all the organizations that don't fit the criteria. Eventually, this class could be refactored to support more than just the size of an array. Note, I did have to upgrade the gmaven plugin version to 1.0 to get it work in our maven project.

I knew from the beginning I wasn't going to use BuilderSupport. It did take me some time to figure out how I was going to support filtered (getOrganizations().withRoles()) and non-filtered versions (getOrganizations()). That is when I decided to extend List or JSONArray, as both method calls had to return my custom List/JSONArray. Overall, I'm pretty happy with the outcome and how long it took me. It was pretty trivial and very fun thanks to groovy.

Tip Debugging External Java Dependencies

Ever spent time debugging 3rd party java libraries? Decompiling is usually the first step. Attempting to walk through the code can be tedious but it's usually the first line of defense. But what if you want to deploy a slightly modified version? In the past, I've checked out the project and built it with my modifications. Since most open source projects don't support "virgin builds", this has a success rate of about 10%. Fortunately, there is a better way. I'm just disappointed I didn't think of it.

In our project we deploy a wiki that is based on JSPWiki using maven overlays. In the version we are using, there isn't any support for being able to configure the wiki files directory outside of a properties file in the WAR. In order to point JSPWiki to a different directory, you would basically have to unzip the WAR, update the file, and then zip the WAR back together (#fail). So, someone on our team discovered we could basically override this behaviour by providing our own implementation of the same class.

To be more specific, the class under question is com.ecyrd.jspwiki.PropertyReader. It's included in the JSPWiki.jar file under /WEB-INF/lib. It's default behaviour is not suitable for our needs, so we get an original copy of PropertyReader.java, and place it under our maven projects /src/main/java directory under the same package of com.ecyrd.jspwiki. Once the projects builds, we now have our version of PropertyReader.class under /WEB-INF/classes, which is important because the ClassLoader will first look under /WEB-INF/classes first before looking in /WEB-INF/lib. This means our class is used instead of the one provided by JSPWiki in /WEB-INF/lib/JSPWiki.jar.

Now I know what your thinking: that's a horrible idea James. And for the most part I agree, but it's not my fault this ability doesn't already exist in JSPWiki. So if you want to keep your conscience clean, go ahead and continue unpacking and repacking that WAR. I'll be happy getting important things done. Obviously, practicing this is the exception and not the rule. And one should provide the patch as an improvement back to the 3rd party for all to enjoy. And before you start asking yourself why you can't just extend the real PropertyReader and override the necessary methods, which I agree would be more ideal, it's not possible because you'd basically be extending yourself since the modified class is the first class in the classpath.

This technique has actually helped me twice debug environment specific issues. It'd saved me a huge amount of time not having to build an external library. In fact, if you check out the exact version, you could even perform remote debugging with breakpoints.

So next time you need to debug an external 3rd party library, consider using this technique before attempting to build it.