James Lorenzen's Blog

What's missing from our REST Services?

2011-04-05T23:11:00.006-05:00

While reading the excellent book RESTful Web Services I discovered something. A technique used on the web since its beginning. Twitter and Facebook seem to use it. Yet it was absent in the REST Services I had been developing and if I had to guess most REST developers aren't including it either. What is so valuable the Internet would be useless without it? The answer is hypermedia.

Hypermedia is the technical term used to describe how web pages (resources) are linked or connected. The author of RESTful Web Services calls it connectedness: how the server guides the client from one application state to another by providing links. A good example of a well-connected application is Wikipedia. It's very powerful when you could pick a random page and click through every entry on Wikipedia without ever editing your browsers URL. Performing a search on Google is another great example, as it wouldn't be very useful if the search result didn't include any links. Without these links, the client must know and create predefined rules to build every URL it wants to visit.

So what do links on a website have to do with REST Services? REST was built on the foundation of the web and just because your not returning HTML doesn't mean you shouldn't create relationships between your resources. In fact, it's amazing how powerful embedding links in your responses can be. For instance, it prevents tight coupling between your clients and services as the clients don't have to construct or even know the exact URLs because the services are providing them.

Let's use Twitter as an example. Assume the following is the URL to get the 20 most recent tweets for my account:

http://api.twitter.com/1/statuses/user_timeline.json?screen_name=jlorenzen

And here is a shortened fictitious response:

[
    {
        "text": "Finished watching Battlestar Galatica"
        "id": "55947977415667712"
        "url": "http://api.twitter.com/1/statuses/show/55947977415667712.json"
    },
    {
        "text": "Started watching Battlestar Galatica"
        "id": "45947977415667712"
        "url": "http://api.twitter.com/1/statuses/show/45947977415667712.json"
    }
]

Notice each tweet includes a direct URL. Now clients can use that URL value verses constructing it, and if it changes in the future, clients don't have to make any changes.

Example using Jersey
So what is the best way to include links in your REST Services? Since I use Jersey on a daily basis, I'll go ahead and show an example of how to embed links in your responses using Jersey. Most likely, any REST framework is going to provide the same kind of features.

Using the twitter example above, the User Status Service with links may look something like this (using groovy):

import javax.ws.rs.GET
import javax.ws.rs.Path
import javax.ws.rs.Produces
import javax.ws.rs.QueryParam
import javax.ws.rs.core.Context
import javax.ws.rs.core.UriInfo

import com.test.UserStatuses

@Path("/statuses")
class StatusesResource {
 
    @Context 
    UriInfo uriInfo

    @GET
    @Path("/user_timeline")
    @Produces(["application/json", "application/xml"])
    def UserStatuses getUserTimeline(@QueryParam("screen_name")String screen_name) {
        def statuses = getStatuses(screen_name)
        
        statuses.each {
            it.url = "${uriInfo.baseUri}statuses/show/$it.id"
        }
        
        return new UserStatuses(statuses: statuses)
    }
}

In this simple example, Jersey injects the UriInfo object which we use to get the baseUri of the request. It's really that simple.

Potential Issues
Well for some it may not be that simple. For example, we discovered an issue when using a reverse proxy (apache httpd). In our production environments, we typically setup apache on port 80 to proxy a localhost JBoss on port 8080. Unfortunately, in this setup the UriInfo.getBaseUri() returns localhost:8080 and not the actual original URL the client used; which is obviously not good. Now if you don't use a reverse proxy then no worries. However, if you do or might potentially in the future, a easy solution seems to be to set the Apache Proxy module option ProxyPreserveHost to On. Setting this to On and restarting Apache seems to fix the issue.

JSONView Firefox Plugin
Once you've started embedding URLs in your REST responses, you might find it useful to install the JSONView firefox plugin. It's got some really slick features like formatting the JSON and creating clickable links for URLs.

How to Change Extjs PieChart Colors

2010-10-01T13:07:00.000-05:00

When using Sencha's, or extjs, charting capability, most likely your going to want to change the default color scheme. I was faced with this issue today and it's not documented as well as you'd expect. I had to piece together a few articles and I'm still not 100% it's the right way as it's using an undocumented config option. But I wanted to document how I did get it to work to same others some time.

Sencha is very well documented and their examples are great. Here is the Pie Chart example we are going to be updating (using version 3.2.1). For those that don't know, extjs charts are based off of Yahoo's YUI charts which is also requires Flash. Not only is it pretty simple to create charts with extjs, but we already are using extjs so we decided to prototype some charts using it.

Here is the code that produces a simple basic PieChart:

new Ext.Panel({
    width: 400,
    height: 400,
    title: 'Pie Chart with Legend - Favorite Season',
    renderTo: 'container',
    items: {
        store: store,
        xtype: 'piechart',
        dataField: 'total',
        categoryField: 'season',
        extraStyle:
        {
            legend:
            {
                display: 'bottom',
                padding: 5,
                font:
                {
                    family: 'Tahoma',
                    size: 13
                }
            }
        }
    }
});

This produces the following PieChart.

To change the default colors provide a series config option:

new Ext.Panel({
    width: 400,
    height: 400,
    title: 'Pie Chart with Legend - Favorite Season',
    renderTo: 'container',
    items: {
        store: store,
        xtype: 'piechart',
        dataField: 'total',
        categoryField: 'season',
        series: [{
            style: {
                colors: ["#ff2400", "#94660e", "#00b8bf", "#edff9f"]
            }
        }],
        extraStyle:
        {
            legend:
            {
                display: 'bottom',
                padding: 5,
                font:
                {
                    family: 'Tahoma',
                    size: 13
                }
            }
        }
    }
});

The chart it produces may not look the most appealing but at least we figured out how to change the colors.

Pie Chart with New Colors

Again, I'm not certain this is the best way to accomplish this as the series config option isn't documented in 3.2.1. But by piecing together this PieChart question and this article, I was able to figure it out.

Missing Office Equipment Prank

2010-08-19T21:49:00.000-05:00

Here is Part 2 of my series where I reminisce about the good times I had at Williams. As I stated previously, I came across these printed off emails in our attic while moving to our new home. I'm so glad I printed them off. #goodtimes

Anyway, this was a prank Keith Stanek, Josh Guthman (Guthy), and myself pulled on Michael Brotherman (Are you Miking Me?). Before showing you the email though I need to set the stage. The year was 2003, and our company had just gone through 3 rounds of layoffs. Morale was pretty low. We worked on the 32nd floor of the 52 story BOK building in downtown Tulsa, Oklahoma. During fire drills we had to take the stairs down a floor. During one of the fire drills we noticed the entire 31st floor was void of humans, but a lot of very nice unused office furniture and white boards were still present. We joked around about repurposing some of the nicer equipment, but nothing ever come of it. Until one morning, we noticed Mike had a new chair. A very nice new chair. We gave him junk about it all day, but decided a prank would be better. So we had Josh Guthman (Guthy) write us up a believable email that Keith and I would spoof using the Facilities email address. Here it is:

From: Facilities Services-Tulsa
Sent: Thursday, March 13, 2003 7:39AM
Subject: Missing Office Equipment

During a recent audit for unused office equipment we discovered black reclining office chairs were missing from several of the floors in the BOK tower. Upon further investigation we discovered those chairs had been procured by current employees looking to upgrade from their existing chairs. While facilities appreciates your desire to utilize existing assets instead of requisitioning new ones, please remember that Williams is undergoing a cost savings campaign at this time and that all unused equipment needs to remain on its respective floor for proper accounting and redistribution either to the lessee or to an employee in need. If we have not already reacquired your chair, please return it to the floor from which it came by the end of business Friday, March 14.

Mike smelled it out, but noone ever confessed, and I'm pretty sure deep down, even though his gut was telling him it was a prank, he still didn't want to take the chance in the possibility that it might have been true. The best part about it was the morning we sent the email, Mike and I had a group breakfast meeting with one of the Senior Executives. As we were going down the elevator, he mentioned to me that he was going to bring it up during the meeting, and with a complete straight face I called his bluff and told him that I think he should, knowing that he wouldn't anyway; and if he did it would only make the prank that much better.

Mike returned the chair.

Real man of Genius

2010-08-18T21:28:00.000-05:00

I usually keep this blog professional, but I'm in the process of moving and came across some emails that I just have to share and they are too long to post on facebook. All 3 emails where from my time at The Williams Company in Tulsa, OK where I had the privilege to work with some great friends. People like Keith Stanek, Michael Brotherman (Are you Miking me?), Jason Randall, Josh Guthman (Guthy), Erin Nylund, Becca Fairchild, Jennifer Brandt, and a host of others. I was so fortunate to work with these people and I'm not sure it could ever be duplicated. We had a ton of fun together. During off hours we played a lot of cards (Rook mainly), Starcraft I, and lots and lots of Halflife. Over time, I got to be pretty good at Halflife (HL). At some point we started playing 2 (Keith and Mike) on one and I still would usually win.

Now to the email. Wednesday August 20, 2003 at 9:45 AM, Michael Brotherman emailed Keith and me his Real man of Genius in my honor. It was done just like the Real Men Of Genius commercials. Here it is in it's original form:

Mr real man of genius...
We solute you, Mr. Total Conqueror in HL.
You who drops bombs that are not in the bathroom.
Who shoots your double barrel in our face.

Your eyes keep us from getting caught,
But your play keeps us from coming back.

Here's to you...Mr crossbow no miss,
Mr. Ray gun who makes it no fun.

We solute you.

My Favorite weapon was the crossbow. Oh good times. Stayed tuned. I've got 2 more coming.

Hash tags in Commit Comments

2010-08-14T23:20:00.000-05:00

I've been using Yammer, Twitter, and Facebook consistently for awhile now. One of the things I really like are hash tags, where yams or tweets include additional meta information in the comment such as #groovy, #hudson, or #maven. One of the main purposes of hash tags, is it allows others to subscribe to an area of interest verses subscribing to hundreds of individual people. Another purpose it serves, is determining interest value; sort of like a subject heading. Since hash tags are typically at the end of a tweet or yam, I usually read the end first before I commit to reading the whole yam or tweet. I don't follow a ton of people yet, but I do consume a lot of information in a day and in order to find the good I have to wade through the bad. Using hash tags aides in this process.

I also think hash tags could help in another area: commit comments. It's something important I've mentioned before, and I think hash tags can be useful in commit comments even if there aren't any tools yet to mash it up. A few days ago, our of habit I accidentally started including some high level hash tags in an svn commit comment and it occurred to me that it might be useful to others, if not myself in 6 months. If we find hash tags useful in yams and tweets, why not commit comments?

Including tokens in commit comments isn't new. In fact, we already include a Jira number in most of our commit comments and this allows us to view all the commits for a Jira issue. There is even a Jira plugin that allows you to perform actions by specifying hashes in commit comments. For example, if I want to resolve a Jira I can include #resolve in my commit comment, and Jira will automatically Resolve that Jira. And don't feel like you can't include the #resolve tag only if your using that jira plugin. I could see value in seeing a #resolve tag in the final commit of a Jira.

As an example, here is the exact commit comment I used that includes some hash tags for geoserver and installer.

"Jira: AC-4207. Got the filtered geowebcache.xml file correctly moved to the production and staging data directories. These files point to localhost with the correct geo port and stage geo port. Also commented out some fixpath.cmd lines to get the installer to work. Finally, I also change the ProcessPanel to not have a condition: changed to . This should allow us to be really selective in what we install and still allow the process panel to run, whereas before it wasn't running. #geoserver #installer"

Now the really cool part is if someone else in the near future notices an issue with geoserver in our installer, this comment will stick out more than a comment without those hashes.

Another cool thing that could be done is a team subscribing to certain hash tags in the svn commit emails. For example, someone responsible for peer reviewing all DAO changes could subscribe to a hash like #dao. Then when developers are modifying DAO's all they need to do is include the #dao tag.

I guess what I am saying is perhaps we could also benefit from putting extra hash tags in our commit comments. My brain has already been trained to read them so personally I think it's useful.

Restoring a Clonezilla Image using VirtualBox

2010-07-25T21:09:00.001-05:00

Ubuntu 10.04 has been out for a few months, and I'm still on 09.10. I have had some success in the past upgrading, but I still prefer doing fresh installs. I guess it comes from my windows days, when an occasional fresh install was good for the computer soul. However, this time I'm also starting a new project at work doing .net instead of java, and I really wanted the ability to "come back" to my old setup. Basically, I wanted to convert my host machine to a virtual one or what's called P2V (Physical to Virtual). I tried VMware Converter but didn't get very far. With some advice from several co-workers though, I did come up with a method that did work and it was fairly easy.

The basic steps are:

Use Clonezilla to Save a Disk Image to a external USB drive. This essentially clones my host machine, where I can restore it later. My hard drive is around 120GB, so I put the image on my 500GB external USB drive. This took about 1.5 hours.
Create a new virtual machine on another external USB drive. The nice thing about using Clonezilla, is for this step you can use VMware or VirtualBox. I used VirtualBox. Obviously, you can't create the virtual machine on your laptop because you don't have enough space. And you can't use the same USB drive because when restoring, Clonezilla needs it to be unmounted. So instead I used another external USB drive.
Start the new virtual machine and boot up Clonezilla to begin restoring the image. You need to change the mode because the default view doesn't work very well when restoring. So at the Clonezilla menu, choose "Other modes of Clonezilla live". Then choose "Clonezilla live (Safe graphic settings, vga=normal)".
When you get to the point where Clonezilla needs to point to the external USB drive that contains the Clonezilla image, remember to enable the USB drive in VirtualBox. To do this, go to the Devices menu option in your virtual machine and select USB Devices and check the appropriate USB drive. Restoring my 120GB image took about 24 hours so make sure to do it when you have time.
Once Clonezilla has finished restoring the image your ready to poweroff the virtual machine, remove the clonezilla CD, and restart.

I still had a few adjustments to make in order to get it to work. When I first started my virtual machine, it complained about not having PAE (Physical Address Extension) enabled. I enabled PAE in ubuntu about a month ago so I could use all 4GB of RAM. Fixing it was easy. Under your machines settings go to System and click on the Processor tab. Check the "Enable PAE/NX" checkbox and restart.

Once it booted up, it complained about my graphics configuration. I tried selecting "Reconfigure Graphics", but that didn't work. Instead I was able to get passed it by selecting "Run in low graphics for one session". This allowed me to finish booting where I installed the virtualbox guest additions which seemed to solve the graphics issue.

That is all there is to it. It was all rather easy. Now I can install Ubuntu 10.04 and have the ability to go back to my previous development environment. I could also see lots of different use cases for this. Combined with the ability to clone virtual machines, all your virtual needs are met.

Upgrading to Maven 3

2010-07-25T00:09:00.001-05:00

I've been playing around with maven 3 lately on our legacy maven 2 multi-module project via mvnsh. Like advertised, maven 3 is backwards compatible with maven 2. In fact, most everything worked out of the box when switching to maven 3. In this post, I'm going to highlight the required and currently optional items I changed so you can start preparing to migrate your project to maven 3. But first, what's so special about maven 3 and why would you upgrade? Polyglot maven, mvnsh, and improved performance (50%-400% faster) are just a few. And since it's so easy to migrate to maven 3, you really don't have any excuses.

Currently, I build our project using maven 2.2.1. This article was tested with mvnsh 0.10 which includes maven 3.0-alpha-6. The current release of maven 3 is 3.0-beta-1, while maven 3.1 is due out Q1 of 2011.

Profiles.xml no longer supported
I haven't really figured out the reasoning, but it doesn't really matter; maven 3.0 no longer supports a profiles.xml. Instead you place your profiles in your ~/.m2/settings.xml. Some of our database processes and integration tests require properties from our profiles.xml. It was simple to solve by just moving my profiles to my settings.xml and everything worked.

Upgrade GMaven Plugin
We depend pretty heavily on the gmaven plugin for testing, simple groovy scripts, and some ant calls. In order to build some modules I had to upgrade gmaven. The current version we were using was 1.0-rc-3. Our projects built perfectly after changing it to org.codehaus.gmaven:gmaven-plugin:1.2.

${pom.version} changing to ${project.version}
Here maven 3 kindly warned me that uses of the maven property pom.version may no longer be supported in future versions and should be changed to project.version. My modules still built, but thought it was nice of maven to inform me of the potential change.

Version and Scope Issues
We had a few places where we needed to define a dependency version and another place where we shouldn't have defined a scope. Both instances prevented maven 3.0 from building our modules, but fixing them was easy. The first instance was we defined a version for a plugin in the pluginManagement section, but maven 3 required it also where it was used in the reporting plugin section. Not exactly sure about this one, ideally you would only define your plugin versions in the pluginManagement section but oh well.

We had some WAR projects using jetty. In the jetty plugin definition we had a dependency on geronimo and had defined a scope of provided. Maven 3 complained about it and since it's really not necessary, just removing it fixed the issue.

modelVersion
Maven 3.0 kept warning about using ${modelVersion} instead of ${project.modelVersion}. I was still able to build though, so my guess is the value for modelVersion, 4.0.0, most likely will change when maven 3.1 comes out.

Weird Surefire Output
This wasn't necessarily an issue with the surefire plugin, but I wanted to comment about it's output when tests failed as I thought it might have been a maven 3 issue. Below is a screenshot of the output when you have failed tests. At first I thought it was a maven 3 issue, but I built the same project using the same commands with maven 2.2.1 and got the same test failures. Hopefully, they can clean this type of thing up, because I could image lots of people getting confused.

Failed test output

That's essentially it. Happily, there really wasn't much required to change, which goes to show the great lengths the maven team has gone through to ensure backwards compatibility. Finally, here is a Compatibility Notes maven has provided on the subject of migrating maven 2 projects to maven 3.

My First Groovy DSL

2010-07-19T00:38:00.001-05:00

It's no secret I'm a groovy homer. I love it. One of the things that makes using groovy so fun, is it's syntax. Being able to get the contents of a file by just saying new File("/home/james/test.log").text is refreshing compared to it's java counterpart. Another thing that makes groovy enjoyable is it's ability to support Domain Specific Languages (DSL). MarkupBuilder is a great example. With Groovy, you can create simple or very complex DSLs for your purposes. To my knowledge there are a few ways you can create your own DSL: extending BuilderSupport or using methodMissing/propertyMissing. In my opinion, extending BuilderSupport is more involved while methodMissing/propertyMissing is kind of the poor man's way of creating a DSL.

Up to this point though, I've never actually came across a good use case for creating a DSL until this past week. We have a large set of automated tests that run against our REST Services. Since our application is now multi-tenant, all of our tests need a valid organization (tenant). In our case, an organization contains multiple roles and locations. Each test has different requirements on the types of organizations it needs. Some might need 2 unique organizations, while another might need an organization with at least 2 roles and 2 locations. It was this use case that I thought a groovy DSL would fit perfectly.

My end goal was to have something like this:

def orgs = OrganizationService.getOrganizations().withRoles().withLocations()

This would return a list of organizations that had at least 1 role and 1 location. The nice thing about this DSL is it's scalable. Meaning, if we add new lists of information to an organization, we won't have to update our class. Also, an important feature, is the method name Roles and Locations correlate to the JSON named arrays of the organization. So my JSON looks something like this:

{"organizations": {"name": "James", "roles": ["R1", "R2"], "locations": ["Tulsa", "Omaha"]}}

When writing my DSL I decided to go the poor man's way and use the methodMissing approach combined with the @Delegate annotation. Here it is:

import net.sf.json.JSONArray

class OrganizationFilterArray {
    @Delegate private JSONArray array
    
    OrganizationFilterArray(array) {
        this.array = array
    }
      
    def methodMissing(String name, args) {        
        if (name.startsWith("with")) {
            def length = (args.length == 0) ? 1 : args[0]
            def arrayName = name[4..5].toLowerCase() + name[6..-1]
            
            return filterByLength(arrayName, length)
        } else {
            throw new MissingMethodException(name, this.class, args)
        }
    }
    
    private filterByLength(listName, length) {        
        def filteredArray = array.findAll {
            it."$listName"?.size() >= length
        }
        
        return new OrganizationFilterArray(filteredArray)
    }
}

I could have just as easily extended JSONArray since it's not final, but I was following the @Delegate guide initially and just thought it was an interesting alternative. The big key here is how I used methodMissing to support an infinite amount of possibilities with how to filter an organization. Everything else I think is pretty self explanatory. When it comes across a method that is missing, withRoles(), it calls my methodMissing method. From there I filter out all the organizations that don't fit the criteria. Eventually, this class could be refactored to support more than just the size of an array. Note, I did have to upgrade the gmaven plugin version to 1.0 to get it work in our maven project.

I knew from the beginning I wasn't going to use BuilderSupport. It did take me some time to figure out how I was going to support filtered (getOrganizations().withRoles()) and non-filtered versions (getOrganizations()). That is when I decided to extend List or JSONArray, as both method calls had to return my custom List/JSONArray. Overall, I'm pretty happy with the outcome and how long it took me. It was pretty trivial and very fun thanks to groovy.

Tip Debugging External Java Dependencies

2010-07-14T23:21:00.000-05:00

Ever spent time debugging 3rd party java libraries? Decompiling is usually the first step. Attempting to walk through the code can be tedious but it's usually the first line of defense. But what if you want to deploy a slightly modified version? In the past, I've checked out the project and built it with my modifications. Since most open source projects don't support "virgin builds", this has a success rate of about 10%. Fortunately, there is a better way. I'm just disappointed I didn't think of it.

In our project we deploy a wiki that is based on JSPWiki using maven overlays. In the version we are using, there isn't any support for being able to configure the wiki files directory outside of a properties file in the WAR. In order to point JSPWiki to a different directory, you would basically have to unzip the WAR, update the file, and then zip the WAR back together (#fail). So, someone on our team discovered we could basically override this behaviour by providing our own implementation of the same class.

To be more specific, the class under question is com.ecyrd.jspwiki.PropertyReader. It's included in the JSPWiki.jar file under /WEB-INF/lib. It's default behaviour is not suitable for our needs, so we get an original copy of PropertyReader.java, and place it under our maven projects /src/main/java directory under the same package of com.ecyrd.jspwiki. Once the projects builds, we now have our version of PropertyReader.class under /WEB-INF/classes, which is important because the ClassLoader will first look under /WEB-INF/classes first before looking in /WEB-INF/lib. This means our class is used instead of the one provided by JSPWiki in /WEB-INF/lib/JSPWiki.jar.

Now I know what your thinking: that's a horrible idea James. And for the most part I agree, but it's not my fault this ability doesn't already exist in JSPWiki. So if you want to keep your conscience clean, go ahead and continue unpacking and repacking that WAR. I'll be happy getting important things done. Obviously, practicing this is the exception and not the rule. And one should provide the patch as an improvement back to the 3rd party for all to enjoy. And before you start asking yourself why you can't just extend the real PropertyReader and override the necessary methods, which I agree would be more ideal, it's not possible because you'd basically be extending yourself since the modified class is the first class in the classpath.

This technique has actually helped me twice debug environment specific issues. It'd saved me a huge amount of time not having to build an external library. In fact, if you check out the exact version, you could even perform remote debugging with breakpoints.

So next time you need to debug an external 3rd party library, consider using this technique before attempting to build it.

Avatar Maven

2010-07-13T16:51:00.001-05:00

Today I gave a quick presentation to some coworkers about maven. It's a broad topic, so I kept it fairly limited. Most of my audience was very familiar with maven, so I tried not boring them with stuff they already knew. I tried making it a little engaging by comparing the Avatar, master of all 4 elements, to Maven, master of the build (it's a stretch I know). It's a quick presentation (15 slides) providing some helpful maven tips, what's coming in maven 3, and mvnsh. Hope you like it.

Avatar Maven

View more presentations from jlorenzen.

Sharing Resources in Maven

2010-07-09T13:55:00.002-05:00

Today I needed to figure out the best way to share resources across multiple maven modules. We have previously done it 2 different ways, neither of which I thought were very good. The first way was using a relative path to reach across to the modules resource directory (usually not a good practice in maven). It went something like this:


    
        ../module1/src/main/resources

The second way was using the infamous maven assembly plugin. I typically avoid the assembly plugin like I avoid writing Assembly. Plus I prefer avoiding 100 extra lines of XML on something so trivial. Luckily, the Sonatype guys apparently knew this and have come up with a more efficient way of sharing resources using the maven-remote-resources-plugin. It has the advantages of requiring a lot less XML lifting and it's nicely integrated into the maven lifecycle. I did run into one small issue trying to get it to work. By default it only copies **/*.txt files from src/main/resources. For several minutes, I couldn't figure how why it wasn't working until I added an includes for **/*.xml. Then it worked perfectly. Here is the end result:

Creating a resource bundle
Add the following to your POM which is going to create the resource bundle.

      
    maven-remote-resources-plugin
    1.1
    
        
            
                bundle
            
            
                
                    **/*.xml

You now should see the following message in your mvn output while running mvn clean install.

[remote-resources:bundle {execution: default}]

This produces a /target/classes/META-INF/maven/remote-resources.xml file which contains references to the resource files. For example,


    test.xml

Consuming Resource Bundle
Add the following to the POM which needs to consume the new resource bundle.

      
    maven-remote-resources-plugin
    1.1
    
        
            
                process
            
            
                
                    com.lorenzen:lorenzen-core:${pom.version}

You now should see the following message in your mvn output while running mvn clean install.

[remote-resources:process {execution: default}]

You should now be able to look into your second modules /target/classes directory and see test.xml.

RSS, Lucene, and REST

2010-07-01T23:22:00.002-05:00

Sorry for the horrible title. I struggled trying to come up with a worthy title, but after a few minutes I decided to not let perfection get in the way of good.

My team has recently worked on a new feature I am pretty excited about: adding support for RSS/Atom in our application. I know your thinking so what. It's not really the what I am excited about but the how. What I'm really excited about was how the story was defined and implemented.

Approach
We had the simple requirement from a newer customer to provide an RSS feed for newly created items. This actually wasn't the first time for this requirement. We prototyped a similar capability a long time ago using OpenESB and the RSS BC, but for multiple reasons it just didn't work out.

So our first decision had to answer: how we were going to implement it......again, but better. Before the sprint began, a few of us got together and hashed out a potential solution: how about we use the Search REST Service, which is backed by Lucene, to support Advanced searches and return RSS?

Why does this excite me so much? To understand that I need to explain our application at a high level. It's a completely javascript-based application using ExtJS (now sencha), backed by REST Services using Jersey. Consequently, we have a lot of REST Services. Right now those REST Services support returning XML or JSON using a custom Response Builder we have created internally.

I'm excited because this single user story could have a huge improvement on the entire system:

If we modified the Search Service to return RSS, then all our REST Services could support RSS.
The REST Service would now support Advanced searches. Previously, it only really supported basic keyword searches.
Any search they perform could now be subscribe to via RSS.

Implementation
I'm not going to go into every detail on how it was done. I wasn't even actually the one who implemented it (see Matt White. He did a fantastic job.). We did have one major hurdle we had to overcome, and that was how to index items to enable advanced searches like Status=New.

Previously this wasn't possible given how we were indexing our items. We were basically indexing the item by building up a large String containing all the item information like the following:

import org.apache.lucene.document.Document
import org.apache.lucene.document.Field

def Document createDocument(item) {
    Document d = new Document()
    
    doc.add(new Field("content",
        getContent(item),
        Field.Store.NO,
        Field.Index.ANALYZED))
        
    return d
}

def String getContent(item) {
    def s = new StringBuilder()
    
    s.append(item.getTitle()).append(" ")
    s.append(item.getStatus()).append(" ")
    s.append(item.getPriority()).append(" ")
    s.append(item.getDescription()).append(" ")
    
    return s.toString()
}

The problem with this is performing a search for "New" would have returned any item with a status of New as well as any items that contained the word New. The solution was to just add another Field to the Document.

doc.add(new Field("Status",
    item.getStatus(),
    Field.Store.NO,
    Field.Index.NOT_ANALYZED));

Now the Search Service could support advanced searches like: Status:"New". You should put the value in quotes in case the value contains spaces (ie Status:"In Progress"). And since Lucene is so powerful, it also means the follow search would work: Status:"New" AND Priority:"High" AND "Hurricane". Now users have the freedom to subscribe to a near limitless amount of RSS feeds based on Advanced Searches.

Start to Finish
I think there were several reasons why this story was a success in my eyes. Most importantly where the two really smart co-workers who worked on it: Matt White and Chuck Hinson. All three of us knew of this user story ahead of time and we were able to discuss it technically days before backlog selection. This allowed us to brainstorm some ideas. Once we narrowed it down, we spent some more time separately looking into the code to find out the level of difficulty and if Advanced Searches like Status:New would be possible. Overall, together I'd say we spent 3-4 hours doing the preliminary work. Doing that preliminary work I think really enabled us to give a proper WAG for the story.

I really can't speak for how the development went (I was at Disney World for 10 days with the family), but I was really impressed with the tests Matt wrote. He wrote a number of unit tests making sure advanced searches worked and basic searches still worked. On top of that, he wrote an overall functional test using HttpBuilder executing the REST Service just as our javascript client would.

Finally, once the main work was finished, we uploaded a diff file to our internal instance of Review Board. From there I was able to perform a peer review where we found a minor bug in the changes.

Summary
I am sure it's not an original idea, but I thought it was a fun User Story that hopefully will provide a lot of value beyond what was originally estimated. Ideally, this might help others who are in similar situations.

How to create a release without the maven2 release plugin

2010-05-14T22:40:00.000-05:00

One of the most referenced articles I have written is "How to create a release using the maven release plugin". But what if you can't get the maven release plugin to work with your project? Perhaps like our team, you've got a legacy maven2 multi-module project that's been nigh impossible to use with the release plugin. Our project has a mix of WAR modules combined with some Flex modules. I believe our last issue was some googlecode flex mojo wasn't working with the release plugin. Consequently, for the past year or so, we've been manually creating our releases. This actually hasn't been that much of a pain since we really only do it once a sprint at the end. Combined with my favorite perl script it doesn't really take that long. However, it does have the disadvantage of requiring some knowledge of what and now to do it. Ideally, it would be a job in Hudson, anyone on the team could run as many times as they like.

In an effort to try and automate as much as possible, I decided to try and automate releasing our legacy multi-module project using bash. This has several benefits: create a release faster, done consistently each time, turn-key solution anyone on the team can run that doesn't require stale documentation on how to do it.

It took my several hours to essentially duplicate the maven release plugin process. Thanks to our new intern Scott Rogers and linux master Ron Alleva, I was eventually able to get it finished. It's my first "official" bash script so pardon the mess. If you've never attempted to automate your release project, first consider reading my article on How to effectively use SNAPSHOT.

Here is the script available as a gist on github: project-release.sh. Here is what it does:

Copies the current working branch (i.e. trunk) into another branch. It uses the pom.xml value to get the current working branch.
Updates all the pom.xml version sections of the current working branch
Commits the pom.xml changes
Checks out the release branch
Updates all the pom.xml version sections of the release branch (basically stripping off -SNAPSHOT)
Commits the pom.xml changes

To run this script all you have to do is run: project-release.sh 2 false. The first parameter (2) is the increment position that the current working branch needs to be next. For example, if trunk was on 1.2.0-SNAPSHOT and the position passed in was 2, then trunk gets updated to 1.3.0-SNAPSHOT. If the position was 3 then trunk would be updated to 1.2.1-SNAPSHOT. The second parameter is used when testing. It's like the dryRun option in the maven release plugin. When set to true, nothing gets copied or committed.

A few notes about the script:

The base branch URL is hardcoded but could easily be passed in as another parameter or placed and read from some external file.
It uses the cmd xpath to extract out the pom version, project name, and scm url. I'm on ubuntu 9.10 and according to synaptic I have libxml-xpath-perl version 1.13-6 installed.
It doesn't run any maven commands like mvn deploy. Other jobs in CI can accomplish that or you can easily add them into the script.
To run from Hudson:

Create a New Job
In the Build section Add a Execute Shell Step
Update the Command text with: $WORKSPACE/trunk/project-release.sh 2 false

Overall, I'm pretty happy with the outcome. And as we start to perform more releases among multiple projects I think it's going to really come in handy. I think ideally you should try and release your project using the maven release plugin, but if that isn't possible then don't give up. Just clone.

Thoughts on Fowler's Continuous Integration

2010-04-14T22:43:00.004-05:00

It's always kind of nice to go back to the basics. I've always enjoyed re-reading basic programming practices and patterns. I tend to forget the things I don't use on a daily basis. That's why I enjoyed reading Martin Fowler's article on Continuous Integration. The article says the last significant update occurred May 2006, but it's withstood the test of time; much like The Cathedral and the Bazaar. But if you don't have the time to read this rather long article, here are a few of the favorites I pulled out as I read it over the course of a few days. Before that, let me explain a little bit of my experience.

At my first programming job we didn't really have a VCS (Version Control System) like CVS or SVN nor did we have a CI (Continuous Integration) server; we really didn't know any better. We essentially did all of our work straight off a shared drive (I know). But that was before I came to Gestalt, now Accenture, 5 years ago. Since then I've been exposed to CVS-->SVN-->Git, Ant-->Maven 1-->Maven 2, CruiseControl-->Hudson, and finally TDD (Test Driven Development). Being exposed to all of this has been a huge improvement to my career. More importantly it's been a huge benefit to how I write software and the tools our teams use such as VCS and CI. I can't image developing software without them.

Here are some points out of Continuous Integration that I would think applies to any project java or not:

Work does not stop on your commit

"However my commit doesn't finish my work. At this point we build again, but this time on an integration machine based on the mainline code. Only when this build succeeds can we say that my changes are done."

So true. Just because you ran some tests locally or manually tested it, doesn't mean your done just because you checked in your changes. You've got to monitor CI to ensure it passes. This has been a topic of discussion on my team lately as we've come in in the morning with a few broken builds. Solution: check in often during the day, but don't checkin and leave and not verify CI passed. Either stay late, sign in at home, come in early, or checkin first thing the next day.

Simple checkout build rule

"The basic rule of thumb is that you should be able to walk up to the project with a virgin machine, do a checkout, and be able to fully build the system."

This is a very important point. Not only will this improve the productivity of new team members but also reduce the amount of time it takes to create new CI jobs. This rule is even more important for open source projects. I've had several issues in the past trying to patch open source projects and wasted several hours just trying to build their code. If you want people do contribute to your project, make it easy for them to build your software. For example, I've been wanting to write a simple Docky plugin for Hudson, but have ran into several issues (New Plugin and Missing Package) trying to build the Do project. Have those questions really been Answered? NO! What have I done about it? I haven't retried it since. To restate Mr. Fowler, I should be able to easily checkout your code and at a minimum build it. As an added bonus it'd be nice to run unit tests as well.

Automate everything

"However like most tasks in this part of software development it can be automated - and as a result should be automated. Asking people to type in strange commands or clicking through dialog boxes is a waste of time and a breeding ground for mistakes."

If your just getting started with CI this can often be difficult. But your long term goal should be to automate everything. This includes creating/destroying your database, deploying/undeploying your application, automating your tests, copying configuration files around. I'd even go as far as to say automate the creation of the development environment: installing maven and java for example. Again this not only speeds up new team members productivity but also those virgin CI servers.

Two great examples of this. Before we had a internal CI team, our team was manually setting up multiple CI servers with maven, java, jboss, and a database. These new servers couldn't be used until all of this stuff was manually configured. Then our internal CI team helped automate some of this stuff and we can very easily use hudson to point jobs at different servers within minutes. Something that wasn't really possible before without manually intervention. And all they really did was call a few simple ant copy commands from maven.

Another good example of this comes back from our old CruiseControl and Ant days. At one point in our project we were constantly breaking a major piece of functionality and one of the main reasons was it was very difficult to test. It was a distributed test with multiple servers communicating with multiple clients via SIP. The build process called for building the latest code, stopping 2 instances of weblogic (1 local, 1 remote), starting weblogic, deploying the latest code, waiting for weblogic to finish starting (not easy mind you), and then running our automated test. This was rather huge undertaking, but given a few weeks we had the core of it automated. It was amazing. I never thought it would have been possible, but it was and anytime that test failed we knew immediately we broke something. We were able to accomplish the difficult parts by calling remote bash scripts via ssh from ant.

Imperfect Tests

"Imperfect tests, run frequently, are much better than perfect tests that are never written at all. "

Not exactly sure what he means by imperfect tests, but this is one place I currently disagree. It takes practice to write good tests. Once you refactor and maintain tests over a long period of time you start getting pretty good at writing tests that require less refactoring. One of the things that is killing the productivity of our team right now is what I call "cronically failing tests" or tests that randomly fail for no reason. You check the change log and nothing changed in the build which means it shouldn't have failed. You rebuild the job and it passes. Here lately this can be attributed to date comparison asserts and issues with timing. For example, the test passes when the database is local, but fails when the database is remote. Or you get different results when the time on the database server is not sync'd. The end result is this produces false negatives that really hurt the validity of CI; developers just start ignoring all failures. Once you've identified one of these cronically failing tests, it's important the author of that test, or the person who last modified it, refactor the test to be flexible. If the author doesn't do it, they will continue producing these types of imperfect tests.

Good Build Characteristics
He had several comments I would wrap into good general build characteristics. Two of which are fast builds and accessible artifacts. As a general rule he suggests keeping build times to around 10 minutes. Which is usually achievable for compile/unit test jobs, but database related and above can usually take longer. My general guideline is try to keep those longer running builds to around 30 minutes, but definitely no longer than an hour. Unfortunately right now, we have several of those 40-55 minute builds I'd like to trim down some. It'd be great to see a hudson plugin that could show me how long each part of my build took.

With a combination of our company maven repository and hudson, it's pretty easy to make our artifacts accessible. This is really huge as sometimes I don't waste time building certain things that take forever to build; I'll just download them from hudson. I know a lot of times our DBA will just download the zip he wants to test and prevents him from updating his source and building, etc. Another related topic is we have several nightly jobs that deploy the latest code to jboss/websphere that can be used the next day by everyone to see/test/verify the latest code.

Rollback Deployment

"If you deploy into production one extra automated capability you should consider is automated rollback."

This was a pretty new concept for me and one we don't necessarily follow. I've heard of Continuous Deployment, but never really heard about a rollback feature. I know we've accidentally benefited from a build failing and not deploying the latest nightly code thus allowing us to perform diff-debugging to track down a bug. We had 2 servers that built the night before, 1 passed and the other failed so it contained the previous days build. A bug was detected on the passing server and we were unable to reproduce on the outdated server. This told us it had been introduced in the past 24 hours. This isn't exactly rolling back but maybe the morale of the story is keeping a server around that is behind a day.

Summary
There is a lot of good general information in this article and I would encourage anyone to take the time to read it. I only highlighted the things that really stuck out at me; there were a lot more useful things I passed mentioning.

Possible solution for WebSphere issue NMSV0310E

2010-04-08T11:01:00.003-05:00

It's been quite a ride porting our EAR from JBoss 4.2.1 to WebSphere 6.1. Installation issues on CentOS and Precompiling JSPs were just a few of the issues we encountered. We are still producing 2 different EARs, but both are based on identical WARs.

The latest issue had to do with receiving a WebSphere exception from a Init Servlet that started a separate Thread. This Thread contained code that indirectly performed a JNDI lookup to get a Datasource. Unlike JBoss, WebSphere apparently doesn't like Unmanaged Threads performing JNDI lookups.

Here is the WebSphere exception:

"javaURLContex E NMSV0310E: A JNDI operation on a "java:" name cannot be completed because the server runtime is not able to associate the operation's thread with any J2EE application component. This condition can occur when the JNDI client using the "java:" name is not executed on the thread of a server application request. Make sure that a J2EE application does not execute JNDI operations on "java:" names within static code blocks or in threads created by that J2EE application. Such code does not necessarily run on the thread of a server application request and therefore is not supported by JNDI operations on "java:" names"

This seems to be a rather common issue in WebSphere with multiple possible solutions. I guess ideally you should try and configure a container-managed Thread using CommonJ (see section Scheduling and thread pooling). Unfortunately, the configuration is different for JBoss.

Fortunately, I think I stumbled upon another solution. While reviewing the WAR Spring applicationContext.xml file that configured our Datasource I noticed a property called lookupOnStartup. It was set to false and when setting it to true, the exception went away.

<bean id="dataSource" class="org.springframework.jndi.JndiObjectFactoryBean">
    <property name="jndiName" value="java:comp/env/jdbc/MY-DS"/>
    <property name="lookupOnStartup" value="true"/>
    <property name="cache" value="true"/>
    <property name="proxyInterface" value="javax.sql.DataSource"/>
</bean>

When setting it back to false the exception appeared again; setting it back to true the exception was gone. Unfortunately, I can only speculate as to why this solved the issue. My guess is, when lookupOnStartup was false, the first attempt to get the Datasource was from a separate Thread which WebSphere didn't like. However, when setting lookupOnStartup to true, the first time the Datasource was retrieved was by a container-managed Thread and once my separate Thread needed the Datasource it was already looked up and cached. According to the javadocs for JndiObjectFactoryBean.setLookupOnStartup the default is true, so it can't be that bad, right? I can't think of a reason why someone would want this delayed.

If you ever run into this issue, you might consider setting lookupOnStartup to true and see if that fixes your issue.

How to Speed up Maven

2010-03-17T00:11:00.005-05:00

How many times do you invoke maven in a day? How much time could you save if you shaved 15 seconds off each execution? That doesn't really sound like a lot but when executed a 100 times a day it adds up quickly; and 15 seconds is probably conservative. Now stop being selfish and think collectively as team how often maven is executed. 15 seconds for 50-100 builds a day across a team of 10-15 programmers adds up even quicker. This was my experience experimenting with the maven-cli-plugin with a simple sub-module running mvn clean install with unit tests.

I had never actually heard of maven providing this ability. I had seen and used it for other technologies like grails and scala, but I never considered if maven did as well. It wasn't until James Strachen recently tweeted about attempting to use a new feature provided by Sonatype called mvnsh: "a CLI Interface that enables you to speed up your builds because project information and Maven plugins are loaded into a single, always-ready JVM instance". From the FAQ, "The Maven engine is only started up (bootstrapped) one time and then held in a "ready" state waiting for user input of Maven or other shell command". Since I spend most of my day basically running maven, I was very intrigued.

Unfortunately, it appears mvnsh only supports maven 3 projects and our projects are still using maven 2. Even though I've read maven 3 is backwards compatible, I'm still waiting for a more official stable release (maven 3 is currently at 3.0-alpha-7). On the bright side, it appears mvnsh was set to improve on the maven-cli-plugin that supports maven 2 projects.

So I just wanted to share my experience getting the maven-cli-plugin set up and get the word out to fellow maven users to improve their productivity. Overall it seems to work as expected and does improve my local build times. I'm on Ubuntu 9.10, maven 2.0.10, and JDK 1.5. As I mentioned in the beginning, using cli:execute-phase decreased my build times approximately 15 seconds. The biggest disadvantage so far is I don't have access to my alias's, which I depend on heavily for just about everything I do with maven.

Getting started with the maven-cli-plugin
I basically followed the Common Setup instructions on the github wiki User Guide.

I added the pluginGroup to my settings.xml file:

<pluginGroups>
    <pluginGroup>org.twdata.maven</pluginGroup>
</pluginGroups>

I then added a new profile to my settings.xml file:

<profile>
    <id>cli</id>
    <pluginRepositories>
        <pluginRepository>
            <id>twdata-m2-repository</id>
            <name>twdata.org Maven 2 Repository</name>
            <url>http://twdata-m2-repository.googlecode.com/svn/</url>
        </pluginRepository>
    </pluginRepositories>
</profile>

And finally I enabled the new profile in settings.xml:

<activeProfiles>
    <activeProfile>cli</activeProfile>
</activeProfiles>

Using the maven-cli-plugin
The maven-cli-plugin basically supports 2 goals: execute and execute-phase. Use execute when you want to run goals and use execute-phase when you want to run phases. It's unfortunate the plugin forces you to pre-commit to one or the other; hopefully mvnsh has improved this. I typically run clean install which are phases so I ran: mvn cli:execute-phase. This puts you in an interactive shell where you can run clean install. Once finished you continue to stay in the shell allowing you to repeatedly execute mvn phases without having to bootstrap maven each time.

Summary
Since I spend a majority of my time executing maven and this simple experiment decreased my build times, I plan on using the cli plugin for every day use when I don't want to use my alias's. Also, this plugin should work on windows as well so don't think you are left out. Let me know how your mileage varies. Do you have any other tips that improve maven execution times?

On a side note, I'm sure the django/python crowd cringe at how much time java developers waste compiling and deploying. Execution times are something I believe they don't have to worry about.

Free Maven Repository Hosting for Open Source projects by Sonatype

2010-03-01T15:48:00.002-06:00

I'm very excited to see Sonatype support maven repositories for Open Source projects that use maven. In all honesty, they didn't have to do this. Unfortunately, the java.net repo was harming the maven reputation. I've had direct experience using the java.net maven repo and can say it was an unpleasant experience. When we open sourced our 4 JBI (Java Business Integration) Components their home existed on java.net and we used the their maven repo. It was difficult to upload anything and it seemed to be constantly down.

Before this announcement, open source java projects using maven didn't really have an option as to where they could publish their artifacts. To my knowledge neither Google Code or Sourceforge offered this capability. Apache and Codehaus did and obviously you still have the Maven Central (http://repo1.maven.org/maven2/), but I never went through the process of what it took to use them. Now it doesn't matter where your project is hosted. Hopefully the next thing to come is free Continuous Integration services in the cloud using Hudson. I think this is the next step for project hosting sites like Google Code and github.

Providing free maven repos I think has a lot of benefits and not just for maven users. For example, this should benefit all dependency management tools that are built on top of maven repos. I'm not 100% sure tools like Ivy, Grape, Gradle, and Buildr use maven repo's, but my guess is they do and this will benefit those users. Another benefit is being able to standardize on maven repositories, hopefully preventing users from searching where they can find your artifacts. I've wasted a lot of time in the past trying to find valid repositories where I could find artifacts for a project I was wanting to use.

I'm also very impressed with the features Sonatype is providing. Not only will they support release artifacts but SNAPSHOT's as well which could consume a lot of space. You'll also be able to easily sync with Central. Finally, they will support a staging repo in order to test things out before officially releasing.

So go sign up and thanks Sonatype. Also, read this post to learn how to release your project using the maven release plugin.

Commit Comments: A Conversation with your Future Self

2010-02-24T22:15:00.011-06:00

I frequently find myself searching subversion commit logs trying to find hints as to why certain "features/bugs" where introduced and why. Usually this results in wasting several hours and almost every time I get frustrated with a lack of dialogue in commit comments. I can't stress enough the importance of this often over looked feature when committing changes. A lot of developers look at it like a 30 second burden that's preventing them from taking lunch earlier. What they really should be doing is pretending it's a conversation with their future self. Six months from now you're most likely going to be seeing those comments, or lack of, wondering why you made that change. It's in that moment I'd rather have a very descriptive summary of what changed and why verses diff'in every version known to man.

I'm not asking for a Kevin Costner script, just be a little more specific. On average, my comments are 1-3 sentences. For the important changes, I've used paragraphs before. Comments with fewer than 5 words drive me nuts, and I have seen a lot of them.

Ever seen a movie where campers make a trail using something like marshmallows so they can find their way back? Think of your commit comments like marshmallows helping you unravel the mystery of an issue and start developing the habit of providing more descriptive commit comments.

Other Useful Tools
One of the tools I have grown fond of is the Jira subversion plugin. Our team uses Jira as our issue tracking system. Nothing gets committed without a Jira number. This is enforced using a subversion pre-commit hook, so every developer has to start their comment with a Jira number. Then in Jira, there is a Subversion Commits link at the bottom to where anyone can view the files changes and their comments.

Using the subversion pre-commit hook has another added bonus, which is using the Hudson Jira Plugin. After a successful build, hudson will extract the Jira number(s) from the commit comments and add a comment to the Jira stating it was integrated at a specific build # and includes a link.

Rapid REST Development with maven jetty plugin

2010-02-16T23:45:00.019-06:00

Ever wonder how much time is wasted by Java developers rebuilding and redeploying web applications? For me alone I can't imagine it. In fact, I'd be embarrassed if Google Buzz made it public knowledge without my consent. Two years ago I wrote an article "No more J2EE Apps" and I received a lot of great feedback. Let me first say that I think, IMHO, Java Developers are at a disadvantage when it comes to rapidly developing web applications with a static language. Developers using python, php, rails, or grails really don't even have to spend a second trying to solve this problem. On the other hand, Java Developers have to figure out how to best accomplish this, and every situation seems to be different: JBoss, Weblogic, Eclipse, Idea, Netbeans, Jetty, JRebel. Of all the solutions I think JRebel provides the best chance for success that solves any environment no matter the web container or developers IDE of choice.

I haven't really done much hardcore java development in awhile, but in order to improve my team's productivity, I am going to be exploring best practices the next couple of months concerning this area. First up, I am going to explain how I got the maven jetty plugin to work with our REST Services WAR and the steps necessary to redeploy changes. The nice thing about jetty is it's easy to use from maven and we could use it in our CI environment to possibly reduce our build times and provide quicker feedback. The downside is each change requires jetty to hotdeploy the new WAR. End the end I think the best solution will be a combination of JBoss+JRebel. But I won't get to for awhile.

Maven Jetty Plugin
My first prototype uses the maven-jetty-plugin version 6. The application we are testing is a WAR containing REST Services built with Jersey (JAX-RS). Here is a good posting by my co-worker Jeff Black "Jersey...Jetty and Maven style!". This example didn't work for me because for some insane reason the init-param, com.sun.ws.rest.config.property.packages, does not work in WebSphere. So I had to do some slight modifications to get it to work without that being declared. My pom.xml and jetty.xml files are below. Most "normal" non-Jersey applications don't need all of this, but it was necessary to get our legacy WAR working with jetty.

Here are the steps involved to start jetty and redeploy changes:

mvn install jetty:run - this will first build the WAR and then start jetty while also deploying the WAR. Running install was necessary because I reference the exploded WAR directory under target in the jetty configuration.
Make changes to source code
Run "mvn compile war:exploded" in a separate terminal to compile your changes and copy the new class files to the location Jersey expects to find them. Which in my case is /target/myapp/WEB-INF/classes
Click back to the terminal running jetty and hit the ENTER key. This causes jetty to reload the WAR. This is because by default I set the scan interval to 0 and jetty.reload to manual, so I can batch up multiple changes before reloading.

Overall I am happy with the results so far. Previously, it took 1-2 minutes and sometimes more to rebuild the war and hotdeploy to JBoss. Using the jetty plugin this now takes around 30 seconds. Again, I think this could be further improved by using JRebel.

Tips
I did have to update my MAVEN_OPTS environment variable to increase Java's PermGenSpace since jetty reloads the WAR each time and you'll quickly run out of memory. This was something I was already doing in JBoss. Here is what it is set to:

export MAVEN_OPTS="-Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=512m"

Sample Files
Here is my pom.xml

<plugin>
  <groupId>org.mortbay.jetty</groupId>
  <artifactId>maven-jetty-plugin</artifactId>
  <version>6.1.22</version>
  <configuration>
      <jettyConfig>${project.build.testOutputDirectory}/jetty.xml</jettyConfig>
      <scanIntervalSeconds>${jetty.scan.sec}</scanIntervalSeconds>
      <useTestClasspath>true</useTestClasspath>
      <webAppConfig>
          <baseResource implementation="org.mortbay.resource.ResourceCollection">
              <resourcesAsCSV>${project.build.directory}/myapp</resourcesAsCSV>
          </baseResource>
      </webAppConfig>
      <systemProperties>
          <systemProperty>
              <name>jetty.port</name>
              <value>${jetty.port}</value>
          </systemProperty>
      </systemProperties>
      <systemProperties>
          <systemProperty>
              <name>jetty.reload</name>
              <value>${jetty.reload}</value>
          </systemProperty>
      </systemProperties>
  </configuration>
  <dependencies>
      <dependency>
          <groupId>commons-dbcp</groupId>
          <artifactId>commons-dbcp</artifactId>
          <version>1.2.1</version>
      </dependency>
      <dependency>
          <groupId>net.sourceforge.jtds</groupId>
          <artifactId>jtds</artifactId>
          <version>1.2</version>
      </dependency>
      <dependency>
          <groupId>com.oracle.jdbc</groupId>
          <artifactId>ojdbc14</artifactId>
          <version>10.2.0</version>
      </dependency>
  </dependencies>
</plugin>
.....
<properties>
 <jetty.port>8080</jetty.port>
 <jetty.scan.sec>10</jetty.scan.sec>
 <jetty.reload>manual</jetty.reload>
</properties>

Here is my jetty.xml located under /src/test/resources used to define the Datasource.

<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Mort Bay Consulting//DTD Configure//EN" "http://jetty.mortbay.org/configure.dtd">

<Configure id="Server" class="org.mortbay.jetty.Server">
 
  <New id="MYAPP-DS" class="org.mortbay.jetty.plus.naming.Resource">
      <Arg>jdbc/MYAPP-DS</Arg>
      <Arg>
          <New class="org.apache.commons.dbcp.BasicDataSource">
              <Set name="driverClassName">oracle.jdbc.driver.OracleDriver</Set>
              <Set name="url">jdbc:oracle:thin:@localhost:1521:XE</Set>
              <Set name="username">user</Set>
              <Set name="password">password</Set>
          </New>
      </Arg>
  </New>

</Configure>

Getting Started with Extjs

2010-02-04T07:53:00.021-06:00

I was recently approached by an internal co-worker within Accenture about our teams experience with Extjs. Out of 150,000+ employees in Accenture he was able to find us through Yammer, a free private twitter-like service for companies. Originally I was just going to respond to him via email after our phone conversation, but that would only benefit him. Instead I thought it might be a good idea to share that experience on this blog.

My current team has been using Extjs for about 2 years. I don't consider myself an expert nor do I actually like developing in a language that needs to work in multiple browsers. We started out with version 2.x and late last year converted our projects to version 3, which was no easy task.

I will say Extjs has impressed this Java developer. Their documentation and examples are excellent and feedback on the forums is great. Most of my team didn't have any prior heavy javascript backgrounds and all have been able to come up to speed quickly. Right now I think the biggest drawback is the GPL licensing of version 3. I understand why they did it, but doesn't mean I have to like it.

Extjs Resources

Examples - Extjs comes with a nice library of great examples on all kinds of things you can do with Extjs. All the examples also come with the download. One of the neatest examples is the Advanced Grid Filtering. One of the main reasons we upgraded to version 3 was for this feature.
API Documentation - The javadoc for extjs. This is the most important and most referenced documentation for any extjs developer. Keep in mind this only shows the most recent version. If you want the API documentation for previous versions, you'll need to download that version of Extjs. The download includes the same API documentation. For local installation or offline reference, there is also a very nice Adobe Air app (see download page).
Forums - The forums are very active and we have received a lot of help. We also have purchased the premium support and overall I'd say it was worth it. Especially since we no longer have that goto javascript guru anymore (yeah you know who you are).
Community Plugins - It's not huge but between the examples noted above and the community plugins we have been able to reuse a lot and create some neat stuff pretty fast.
Blogs - I'd recommend subscribing to the official Extjs blog as well as this search related feed from DZone. And, just because I think it is cool, here is Google Reader bundle I created that captures everything I tag with extjs. Finally, here is a search on my blog about articles I have written about Extjs.
Books - According to Amazon there are multiple books out right now about developing with Extjs. I have "Learning Ex JS" by Shea Frederick.

Miscellaneous

Javascript/CSS Consolidator - One of the best moves we made was investing in a javascript/css consolidator. Props go out to Joe Kueser for doing this for us. After a few months of development, our application started to grind to a slow crawl as several (80+) javascript and css files where being downloaded. Using the jawr framework we reduced that to around 20 GET requests which improved performance dramatically. Not only does it combine multiple files into one but it also minifies them (removes comments and spaces, etc) and supports gzipping them. I have been very impressed with jawr. It has exceeded our expectations and I would recommend it to anyone.
Development Environment - No web developer would be complete without Firefox and Firebug. But for those occansional nasty IE issues I haven't found the perfect solution. In the past I have had some luck with jsdt. Currently I am using IE8 in virutalbox in IE7 mode with it's built in Developer Tools. Although not firebug, it's the closest I have come to find. For those really bad IE6 issues I have used the Internet Explorer Developer Toolbar which beats scattered alerts in your code and is better than nothing.
Extjs version 2.x or 3.x - There has been a lot of history surrounding the licensing strategy so I won't go into all of that. Just know that version 2.x is LGPL and can be used or embedded into your applications for free. Version 3.x is GPL which means if your project isn't some internal IT app or also GPL you need to purchase developer licenses.
Public Uses - Here are just a few well known sites I have noticed using Extjs: Quicken Online and Dow Jones Industrial Index.
jQuery Integration - In case you want to combine jQuery and Extjs you can by using the extjs-jquery-adapter available in the download. I haven't really used jQuery a lot but from what I have read and heard, it's a great javascript library that supports animation and DOM querying. You can do these things in Extjs, but jQuery looks like it might do it better and cleaner. I think Extjs excels in providing out of the box prebuilt and customizable Widgets/Components like the Grid (Table).

Precompiling JSPs for WebSphere 6.1

2010-01-28T22:51:00.017-06:00

Don't envy me..........Your about to witness my efforts from the past 2-3 weeks.

For the past week I have been trying to get precompiled JSPs working for WebSphere 6.1, because our target environment does not include the Java Development Kit (JDK) for security reasons. The following is a brief explanation on how to precompile JSPs for WebSphere 6.1 using maven2. And as an added bonus, I'll also explain how to use the maven-was6-plugin to automate deployment of a WAR using Jython.

Precompiling JSPs for WebSphere
First off, let me say, this was not fun. I spent a ton of time trying to figure out why the precompiled JSPs worked on JBoss 4.2.1, but not WebSphere 6.1. Both maven jspc plugins (jspc-maven-plugin and maven-jetty-jspc-plugin) produced precompiled JSPs that worked on JBoss, but not WebSphere. From what I could tell it all boiled down to the fact that JBoss 4.2.1 includes the 2.1 version of jsp-api while WebSphere 6.1.0.27 includes the 2.0 version. Just so you believe me, the 2 conflicting jars in WebSphere were: $WAS_HOME/lib/j2ee.jar and $WAS_HOME/plugins/com.ibm.ws.webcontainer_2.0.0.jar. On the plus side, I did find a great Java decompiler for linux called jd-gui. I'd highly recommend it for any operation system. It uses jad, but provides a nice GUI interface that even lets you open a jar and explore any .class file.

In summary, based on my experience, the 2 existing maven jspc plugins do not create compatible precompiled JSPs with WebSphere 6.1.0.27. Surprisingly, the solution that ended up working was using the JspBatchCompiler.sh script that comes with WebSphere.

The JspBatchCompiler.sh was exactly what we needed. Thankfully, you can give this script a WAR or EAR file and it will explode it, precompile all the JSPs, and repackage it back up again. This script is located under $WAS_HOME/bin. Once I verified it by manually running the script, the next step was to automate it using maven. Since I wasted so much time figuring everything else out, I didn't spend a ton of time improving the maven portion. Instead I decided to go with what I knew and that was using the exec-maven-plugin to run the script. The following is the profile I used to precompile the JSPs in the WARs located in the EAR.



<profile>
<id>precompile</id>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<executions>
   <execution>
       <id>precompile-was-ear-jsps</id>
       <phase>validate</phase>
       <goals>
           <goal>exec</goal>
       </goals>
       <configuration>
           <executable>${wasHome}/bin/JspBatchCompiler.sh</executable>
           <arguments>
               <argument>-ear.path</argument>
               <argument>${pom.basedir}/target/${project.artifactId}-${project.version}.${project.packaging}</argument>
               <argument>-jdkSourceLevel</argument>
               <argument>15</argument>
           </arguments>
       </configuration>
   </execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<executions>
   <execution>
   <id>rename-replace-was-ear</id>
   <phase>validate</phase>
   <goals>
       <goal>run</goal>
   </goals>
   <configuration>
       <tasks>
           <move file="${java.io.tmpdir}/${project.artifactId}-${project.version}.${project.packaging}"
         tofile="${pom.basedir}/target/${project.artifactId}_jspc-${project.version}.${project.packaging}"/>
           <delete file="${pom.basedir}/target/${project.artifactId}-${project.version}.${project.packaging}"/>
       </tasks>
   </configuration>
   </execution>
</executions>
</plugin>
</plugins>
</build>
</profile>

The first plugin section uses the exec-maven-plugin to run the JspBatchCompiler.sh and takes the EAR as input. The second plugin section uses the maven-antrun-plugin to basically rename the EAR to include a keyword (_jspc) in the EAR filename so everyone knows when they have an EAR with precompiled JSPs. It then moves the new EAR from it's tmp location back to the modules target directory where everything expects it to be. Once that is done, it removes the old EAR to avoid confusion.

About the only improvement that could be made is making it portable to other Operating Systems like Windows. This "could" be accomplished by using the JspC ant task WebSphere provides, but I couldn't find any good examples of how to do that via maven, so I took a rain check.

Deploy WAR to WebSphere 6.1
This Jython code snippet literally saved our sprint. I am not sure how I would have automated deploying and undeploying a WAR via maven without it as the maven-was6-plugin really only works with EARs. This is because when deploying a WAR you need to provide the WARs contextroot, which the plugin currently doesn't support (MWAS-59). I was however able to call the Jython script from the maven-was6-plugin to undeploy and deploy a WAR.

The following profiles and Jython scripts show how to use maven and Jython to undeploy and deploy a WAR. The Jython scripts exist in files under the same directory as the pom.



<profile>
<id>undeploy-war</id>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>was6-maven-plugin</artifactId>
<executions>
 <execution>
   <id>undeploy</id>
   <phase>validate</phase>
   <goals>
     <goal>wsAdmin</goal>
   </goals>
 </execution>
</executions>
<configuration>
 <wasHome>${wasHome}</wasHome>
 <profileName>AppSrv01</profileName>
 <conntype>SOAP</conntype>
 <applicationName>petstore_war</applicationName>
 <earFile>${pom.basedir}/target/petstore.war</earFile>
 <updateExisting>false</updateExisting>
 <language>jython</language>
 <script>uninstallApp.py</script>
 <host>localhost</host>
</configuration>
</plugin>
</plugins>
</build>
</profile>

# File: uninstallApp.py
# Jython script to undeploy WAR
# FYI, the was6 plugin does support the ability to pass in params to the jython script
cellName = 'testbed01Node01Cell'
nodeName = 'testbed01Node01'
serverName = 'server1'

#Install the app
print "Installing App: "
AdminApp.install("../petstore.war", "-contextroot /petstore -defaultbinding.virtual.host default_host -usedefaultbindings");
AdminConfig.save();

#Start the app
apps = AdminApp.list().split("\n");
theApp = ""
for iApp in apps:
if str(iApp).find("petstore") >= 0:
theApp = iApp;
print "Starting App: ", theApp
appManager = AdminControl.queryNames('cell='+cellName+',node='+nodeName+',type=ApplicationManager,process='+serverName+',*')
AdminControl.invoke(appManager, 'startApplication', theApp)
print "Application installed and started successfuly!"

Here is the profile and Jython script I used to Deploy a WAR:



<profile>
<id>deploy-war</id>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>was6-maven-plugin</artifactId>
<executions>
 <execution>
   <id>deploy</id>
   <phase>validate</phase>
   <goals>
     <goal>wsAdmin</goal>
   </goals>
 </execution>
</executions>
<configuration>
 <wasHome>${wasHome}</wasHome>
 <profileName>AppSrv01</profileName>
 <conntype>SOAP</conntype>
 <applicationName>petstore_war</applicationName>
 <earFile>${pom.basedir}/target/petstore.war</earFile>
 <updateExisting>false</updateExisting>
 <language>jython</language>
 <script>installApp.py</script>
 <host>localhost</host>
</configuration>
</plugin>
</plugins>
</build>
</profile>

# File: installApp.py
# Jython script to deploy WAR
# FYI, the was6 plugin does support the ability to pass in params to the jython script
cellName = 'testbed01Node01Cell'
nodeName = 'testbed01Node01'
serverName = 'server1'

#Install the app
print "Installing App: "
AdminApp.install("../petstore.war", "-contextroot /petstore -defaultbinding.virtual.host default_host -usedefaultbindings");
AdminConfig.save();

#Start the app
apps = AdminApp.list().split("\n");
theApp = ""
for iApp in apps:
if str(iApp).find("petstore") >= 0:
theApp = iApp;
print "Starting App: ", theApp
appManager = AdminControl.queryNames('cell='+cellName+',node='+nodeName+',type=ApplicationManager,process='+serverName+',*')
AdminControl.invoke(appManager, 'startApplication', theApp)
print "Application installed and started successfuly!"

That's it. The Jython scripts could be improved by making the cell, node, and server names configurable instead of hardcoded and it "appears" the maven-was6-plugin supports passing in properties, but I just didn't have the time to figure it out at the moment.

By the way, I hope maven 3 has solved the XML verbosity when it comes to doing simple things like creating profiles. That's a lot of XML to do very little.

DZone Top Links Feed

2010-01-14T11:06:00.003-06:00

I have a problem: The DZone Top Links section is great but doesn't support an RSS feed. It has feeds for just about everything else. This would be like digg or tweetmeme not having a feed for their most popular links. What makes it worse is I read a majority of their Top Links, but in order to do so I have to keep a tab open in firefox. Wouldn't it be great if I could just subscribe via Google Reader? And now thanks to Yahoo Pipes's screen scrapping capability you can.

Click this link to subscribe to the DZone Top Links Feed: http://pipes.yahoo.com/jlorenzen/dzonetoplinks.

This was accomplished by cloning my RssHuskerPedia pipe and changing a few things around. These pipes depend on the Fetch Page module which essentially lets you scrap the page allowing you to create a list. This feed doesn't contain all the metadeta that would normally come from an official DZone feed, but it supports the basics and prevents me from missing great articles that I would have normally missed.

Please vote this up on DZone to get the word out and hopefully DZone will create an official one. If it gets voted up enough and makes it to the Top Links section, hopefully you won't get stuck in an infinite loop.

Installing WebSphere 6.1 on CentOS 5.3

2009-12-15T10:36:00.002-06:00

My colleagues and I have had a very difficult time trying to get IBM WebSphere 6.1 installed in CentOS 5.3. Thanks to the help of Matt White, we finally believe we have figured out a work around.

The Issue
The swing installer does not load after clicking on the Install link within the HTML Launch page. This issue is best documented in the Redhat and IBM communities.
Basically, we would perform a netinstall of CentOS 5.3, unpack was.cd.6100.wasdev.nocharge.linux.ia32.tar.gz into a subdirectory, and run ./launchpad.sh. Firefox would load the install page which contains a link to install WebSphere that brings up a Java Swing Application that performs the actual install. Problem is the Java Swing app never appears. After closing the browser, launchpad complains saying, "No supported Web browser was detected". Also we discovered an error in the following log file /tmp/niflogs/log.txt:

"Process, com.installshield.wizard.StandardWizardListener, err, could not initialize interface swing"

The Solution(s)
We actually never figured out how to solve this issue. We tried different JDKs, different browsers, 64 bit versions, etc. Nothing fixed the problem. Thanks to Matt White though, we did figure out a possible work around.

Open a new terminal and go to the websphere directory were you unpacked the installer. cd into the WAS directory and run: sudo java -jar setup.jar. The Java Swing install app appears and you are now able to install WebSphere. We have had mixed results with it automatically creating a default profile (AppSvr01), so if it does not create a default profile, you can manually create it by running the /opt/IBM/WebSphere/AppServer/bin/ProfileManagement/pmt.sh script.

The other possible solution is to install an older version of CentOS such as 5.1, install WebSphere 6.1, and then upgrade CentOS to 5.3. I tried doing this with 5.2 and the launchpad still failed, but we know of others that we think used 5.1 and it worked.

Preventing Insecure Login pages

2009-11-16T13:09:00.014-06:00

If you are a webpage author or use confidential information on the internet (credit card information, social security number, login credentials for paypal or banking), you might think twice the next time you log into your favorite website. I know I started paying more attention after listening to Steve Gibson's Security Now podcast titled "The Fundamentally Broken Browser Model". That title is a bit confusing, but the underlying issue is very serious. That is why I would like to further explain the issue by providing a simple example, then a real world example using facebook, and then discuss a simple solution.

First, what does Mr. Gibson mean by The Fundamentally Broken Browser Model? Well, at this years Black Hat conference, a hacker named Moxie Marlinspike gave a presentation where he talked about how he was able to capture sensitive information at a public WiFi hotspot using open tools he created. Specifically, during a 24-hour period, he intercepted 114 logins to yahoo, 50 logins to gmail, 42 to ticketmaster, 14 to rapidshare, 13 to hotmail, 9 to paypal, 9 to LinkedIn, and 3 to facebook. So how did Moxie do it? He took advantage of a common flaw of most login pages: the login pages themselves are not received by the client over SSL allowing man-in-the middle attacks to change the submit URL.

SSL (Secure Socket Layer) has been the common method for securing HTTP, but has been limited to only sensitive areas of a website for performance reasons. Sites have always protected a user's private information (usernames, passwords, credit card numbers, etc) using SSL, but to date no one has thought about actually securing the login page itself and this is a huge problem as most sites don't encrypt their login pages. Not encrypting login forms leaves it open to modification before it returns to the user.

This attack has 2 basic steps. First, the malicious user needs to be on the same network (LAN) and utilize ARP (Address Resolution Protocol) spoofing techniques to insert himself in-between connections (this is pretty scary considering how many times I have connected to restaurant, hotel, and airport WiFi hotspots). Second, a LAN user has to visit a login page that was received over a non-SSL connection.

Simple Example
To explain the issue better, let me use a simple example (let's leave ARP spoofing out for now). A user visits an e-commerce site http://www.eco.com. Like most sites, eco.com includes a Login link at the top. Clicking on this link takes the user to http://www.eco.com/login, which returns a simple login form in HTML.

<form method="POST" action="https://www.eco.com/login/authenticate">
    Username: <input name="username" type="text">
    Password: <input name="password" type="password">
    <input value="Submit" type="submit">
</form>

The user types in their username and password, clicks the Submit button, and their private information is sent encrypted over SSL. It is sent over SSL because the form's action value is set to an SSL URL (https://www.eco.com/login/authenticate). But there is one big problem. The Login link the user click was non-SSL (http://www.eco.com/login), meaning the response back to client was sent over the network as clear-text and could easily be modified by man-in-the middle attacks using ARP spoofing. A malicious user could change the form's submit URL from https://www.eco.com/login/authenticate to http://www.eco.com/login/authenticate and the user would never know it happened. There are few default clues to indicate this is happening. So a malicious user, changes the submit URL to a non-SSL URL, a user clicks Submit, and their credentials are sent as clear-text over the network.

facebook example
Armed with this new information I wondered how some of my favorite sites handle this situation. No matter what I tried, it looked like gmail, ebay, and paypal where safe and used SSL for their login pages. So that gave me some peace of mind. However, facebook provides us with a perfect bad example.

If your like me, I type facebook in my browser and use the CRTL+ENT keyboard shortcut to fill in the rest. So I end up at http://www.facebook.com. If you are not currently logged into facebook, you are presented with the following home page that includes a form to register or login.

Again, this home page includes a login page that was sent over a non-SSL connection. As we expect, if you look at the source, you can see the HTML form does post securely to the URL https://login.facebook.com/login.php?login_attempt=1. So what do you do if you don't want to fill in a form that was received over a non-SSL connection? Fortunately, facebook also supports SSL for its home page (https://www.facebook.com), it just takes a little awareness and an extra step.

Solution
More sites need to use SSL for their login forms, and not just for when users post their credentials. At a minimum, like facebook, sites should also support SSL and ideally all login forms would automatically be requested over SSL. If you come across a site that initially does not use SSL for the login page, try and use https. If that fails then think about using a VPN solution or even a travellers router.

Google App Engine with Grails

2009-11-07T00:31:00.005-06:00

After I recently got a grails app to work with spring-security, REST, and cache I wanted to try my luck and create a Google App Engine application using grails. Unfortunately, I think it's still pretty early as I ran into several issues early on. However, I see enormous potential once these issues get resolved.

I am using grails v1.1.1, app-engine plugin v0.8.5, with gorm-jpa v0.5. The first issue I ran into was deploying when using grails app-engine deploy. It failed to work subsequent times after I got the WAR initially deployed using /app-sdk/bin/appcfg.sh. It complained about not supplying an email address. Turns out there is a bug related to ant that should get fixed soon. The work around was simple enough; just hit Enter before typing in your email address.

Secondly, I was disappointed to find out that the spring-security (acegi) plugin wasn't compatible in GAE. Based on the compile errors I received locally, it appears spring-security depends on hibernate which is not an option in GAE (you either get JDO or JPA). I chose JPA because according to the documentation, JDO doesn't support any of the GORM dynamic finders methods where as JPA supports most of them. I guess now that I think about it, I'm not suprised acegi didn't work in GAE; just disappointed.

I found a minor issue I submitted concerning the default delete action in generated controllers. Seems the gorm-jpa plugin doesn't wrap the delete method around a [Domain].withTransaction closure which apparently is required (see the save action).

Finally, I was unable to get content negotiation working using the withFormat concept I described here. It worked locally running grails app-engine run, but threw an exception in GAE:

Caused by: java.security.AccessControlException: access denied java.lang.RuntimePermission getClassLoader

My guess is it's related to the Security issues they report on the FAQ:

"The current preview release of the Google AppEngine SDK has a bug that doesn't allow it to run Groovy code when the full permissions restrictions are used. This will be fixed in the next release, but in the meantime the development environment runs without emulating the permissions restrictions of the actual AppEngine environment."

Overall, I enjoyed the easy commands it provides to build, run, and deploy to GAE. The deploy was suprisingly pretty quick. I was able to view the exceptions in GAE easily. I was just prevented from making any real progress. If you want to see what I did get done, which was not much, check it out here http://james-lorenzen.appspot.com

Learning with Grails: Security, Extjs, REST, Spring Insight

2009-11-03T23:11:00.026-06:00

Unfortunately, security and performance are often postponed early in the development process and are surpassed for new functionality. Reasonable justification usually includes cost and time. A few of the real issues I think are difficulty and lack of experience. I have to admit I am no security or performance expert, and implementing both early on would be time consuming and difficult. I've always heard security done after the fact is never as good if it's done in the beginning. So I wanted to share my experience using the spring-security (acegi) plugin in grails. Not only did I learn a lot about security, but also how to add support for REST Services and integrate Ajax clients using Extjs, how to enable caching, and inspect the performance of my app using Spring Insight that comes with the Springsource tc server developer edition.

Table of Contents

Spring-Security (Acegi)
REST Support for Extjs Ajax clients
Spring Insight and Caching

Spring-Security (Acegi)

Building authentication and authorization into your app in the beginning can be difficult, but with the spring-security plugin and grails it's super easy. There are other security plugins for grails, but I decided on spring-security/acegi since our team has contemplated using it on our project several times and it seems to have a pretty good history. This isn't a how to guide on using the security plugin, the documentation is pretty good. However, I do want to share some unexpected things I ran into and how I arrived at some of my conclusions.

When applying the security capabilities, I applied Test-Driven Development (TDD) to test my assumptions and changes. After I installed the plugin, I got started right away creating Users, Roles, and Requestmaps in Bootstrap. Requestmap is the domain model mapping roles to URIs. I prefered this method initially because I wanted to persist this information to the database. First I wanted to lock down the ability to delete a model I called Event. Here is a sample Bootstrap to accomplish it:

In this first test I created an admin role, mapped the admin role to /event/delete, and then added a new user to the admin role. This worked as expected but had a pretty big security hole (see Unexpected observations using spring-security plugin). The issue was unauthorized users were still able to delete events using the edit form. Underneath, grails submits to the index action and the controller handles directing the request to the delete action in the controller bypassing the security created by my Requestmap. I could hide the Delete button on the edit page, but malicous users could still expliot this hole.

So instead of using Requestmap, I annotated my controller actions. This method can properly handle the use case above and deny unauthorized access to the delete action.
I also learned that users have to be associated with a Role, otherwise they are unable to log in (see post). This seemed rather annoying since I was expecting to use the predefined roles: IS_AUTHENTICATED_FULLY, IS_AUTHENTICATED_REMEMBERED, and IS_AUTHENTICATED_ANONYMOUSLY without having to associate all my users with roles. Two suggestions by Burt where to extend a Base model class that supported a default Role for all Users, or extend the GrailsDaoImpl class to remove the role requirement for users.

In summary, it seems best to annotation your controllers instead of using Requestmap and keep in mind that by default all users need to be associated with a role to login into your application.

REST Support for Extjs Ajax clients

Now that I have portions of my app locked down, I wanted to see how this effected Rich Internet Applications (RIAs) such as those that use Extjs. I wanted to answer 2 basic questions:

How can javascript clients remove admin functions like Delete buttons?
How could I modify my controller to support multiple clients that need HTML, JSON, or XML?

To test these questions, I downloaded and installed extjs into my grails app and created a simple grid based on the array-grid example in extjs. As part of this test I wanted to add a Delete button to the bottom toolbar, so that I could later enable or disable based on the users role.

I'm not super proud of how I contrived the roles and used them in javascript, but below is how I did it (If you were doing this for real, I'd make this a point of focus to come up with a better solution like creating a service that returned this information as JSON that could be called by an Ajax client).

Since there is no real easy way to get access to the users roles in a gsp, I set a bean in the controller that is used in the gsp. So in EventController I added the following grid action:
Then in a new grid.gsp I added the following:


Ext.onReady(function() {
    var roles = new Ext.util.MixedCollection();
    <% authorities?.each { %>
        roles.add("<%=it.authority%>");
    <% } %>
});

Then when I build the grid, I could eanble or disable the Delete button by doing:


roles.contains("ROLE_ADMIN")

That's pretty much how I disabled certain gui items that are restricted based on role. Again not very pretty, but effective.

Next, I wanted to populate my grid with real data from my controller. Fortunately for developers, grails excels in this area using convention over configuration using the withFormat content negiotation with URI Extensions. All I had to do was modify the controllers list action to support more than HTML responses:


def list = {
    params.max = Math.min( params.max ? params.max.toInteger() : 10,  100)
    def events = Event.list(params)
    withFormat {
        html { [eventInstanceList: events, eventInstanceTotal: Event.count()] }
        xml { render events as XML }
        json { render( [list: events] as JSON ) }
    }
}

Here I easily add support for XML and JSON clients while also continuing to support clients who want HTML. One thing to note, is I like naming arrays in JSON; that is why you see the render([list: events]) syntax. Without that, the JSON response looks like this

{[{"class":"Event".....

instead of my preferred way

{"list":[{"class":"Event".....

Now all the client needs to do is request the URI /event/list.json or /event/list.xml.

Here is my complete javascript that includes the example JsonStore and Grid definitions.

Spring Insight and Caching

Next, I really wanted to try out the Spring Insight capability that is now included in the Springsource tc server developers edition. This would let you see into how grails and hibernate are operating on your behalf per request. For many web applications, there are typically 2 bottlenecks that degrade performance: 1) number of trips from the client to server 2) number of trips to the database. Using the Spring Insight tool, developers can see what SQL is being executed and how long it took.

To get started with Spring Insight I followed their Getting Started Guide. Once I had Insight running (http://localhost:8080/insight), I next ran 'grails war' to create a WAR that I could deploy to tomcat (Note: if you do this multiple times, redeploy, you have to be careful with what you do in your apps Bootstrap class. I was doing a lot of inserts that caused my WAR to fail deployment because by default when running 'grails war', grails defaults to the production environment which uses an update file database. So my Bootstrap was trying to add roles and users that already existed and that caused deployment to fail).

The WAR deployed fine and I was up and running watching my apps performance metrics in the Insight dashboard as I moved around in my app. What I noticed first was unexpected multiple JDBC calls when I thought caching was enabled by default in grails.

What I learned was you have to enable caching on a per domain or query basis. So I added the following to my Event domain model


static mapping = {
    cache true
}

rebuilt and redeployed the war and was able to see fewer JDBC calls proving that my cached domain was working. I then added a new Event, viewed all events, and saw the extra select call since the cache was purged. Pretty impressive!

Next I wondered if spring-security was caching all the user and role information by default like it advertised. If not that could be a huge performance issue. Spring Insight was also able to prove that it was caching its results.

Overall, I learned a lot about grails, security, REST, extjs, caching, and spring insight and grails was the perfect platform to prototype these concepts in preparation for real production use.

Blank screen when installing Ubuntu

2009-08-04T11:06:00.006-05:00

Several co-workers and I run encrypted Ubuntu Desktop on our work laptops. Specifically I have an HP Compaq 6710b and they have all had monitor issues when installing Ubuntu. If you run the default "Install Ubuntu" option, the next screen will just be a blank screen. I need to get this documented because I always forget how to fix it.

While at the Ubuntu Splash screen do the following:

Press the F1 key for "Help"
Press the F6 key for "Special boot parameters for special machines"

There you will see an option in the middle described as:
Laptops with screen display problems vga=771. We need to append this to the Boot Options.

Press the ESC key to get back to the Splash page
Press F6 for "Other Options" and then Press ESC

Now the Boot Options should be visible. It should look something like this:

Boot Options file=/cdrom/preseed/ubuntu.see initrd=/install/initrd.gz quiest --

Delete the trailing -- and add in vga=771. You should have the following:

Boot Options file=/cdrom/preseed/ubuntu.see initrd=/install/initrd.gz quiest vga-771

Press the enter key and you should now be able to successfully install Ubuntu.

Linux config files available on github

2009-07-23T00:16:00.006-05:00

I pushed several of my common linux config files that I often find myself sharing with others via email to github. I'd say it's currently specific to Java developers that are using maven2. I lean heavily on alias's defined in my .bashrc file which lives under my home directory. Hopefully others will fork this project and I can merge forked changes back into my project.

Not only did I want to provide an easy way to share among co-workers, but I also wanted to be able to port to other machines or even when I need to rebuild my work laptop. Also, I just want to get more comfortable with a DVCS workflow. Thanks to Ron Alleva's post (Using Git and Subversion in 5 Easy Steps) about getting setup with using git as the client to svn, I am all set at work to start using git.

Please check it out at https://github.com/jlorenzen/linux-config-files/tree

git clone git://github.com/jlorenzen/linux-config-files.git

Currently it includes:

.bashrc - Contains my alias's
.synergy.config - Config file I stole from Joe Kueser which lets me control a windows machine with the keyboard and mouse connected to my linux laptop (see synergy).
.vimrc - Simple vim settings. Nothing special yet
idea.sh - Simple IntelliJ Idea startup script for linux
idea.vmoptions - Modified vmoptions for IntelliJ Idea
start-hudson.sh - Example of a generic linux startup script
stop-hudson.sh - Example of a generic linux stop script

For those looking to create a github account (and for my future reference) I struggled on how to sync up my local master with the master on github. Thankfully history was around to remind me.

git push origin master

This was after I did the one step (I think) to add the origin:

git remote add origin git@github.com:jlorenzen/linux-config-files.git

Minor issue with Ubuntu Desktop

2009-07-22T21:47:00.006-05:00

Linux and Ubuntu are awesome. I switched from developing on Windows XP to Ubuntu Desktop in late 2007 (7.04 or Feisty Fawn). I will never again develop on an windows machine. Hopefully I won't ever work for a company that requires a windows platform, but at that point I would rather buy my own work computer or setup Ubuntu in a virtual instance. I really appreciate Rob Madole and Brian O'Neill for showing my the right path. I remember Brian's frustration anytime he was forced to pair with me on my windows machine. Now I am like that with my co-workers.

Even though Ubuntu is great, I still have one minor compliant. Well it's really 3 things combined: Extended Monitor support, Compiz, and Gnome-Do. I am addicted to all 3, but trying to get all 3 to work together seems impossible.

Extended Monitor Support
The big problem here is going back and forth between work and home. At work I want to use my monitor, but at home I use just the laptop monitor. In the past I ran nvidia-settings when I wanted go back and forth. This has been improved with some help from Ron Alleva by using the xrandr command. He gave me some alias's to run when I wanted to switch. When I first installed 8.10, I used those alias's, but it seems lately its being auto-detected because when I plug my monitor in at work, ubuntu recognizes it and I don't have to run anything. Nor do I have to run anything now when I get home. So it would seem I have a good handle on this now, but when using an extended monitor other things don't work.

Compiz
I love compiz mainly for one thing: transparency in my terminal. I'm sure I have set a million other settings before, but enabling the Normal Visual Effects under the System > Preferences > Appearance setting is usually one of the first things I do. I believe this setting uses Compiz. With this setting enabled, my terminal window is transparent enough that I can see other applications in the background, like firefox or pidgin which I typically have behind terminal without having to switch back and forth. Unfortunately, it seems I can't have this setting enabled when I have an extended monitor.

Gnome-Do
Man I love it. I am hoping to eventually get to a point to where I have an entirely clean desktop like this. My favourite is the docky option seen here which is similar to Mac's Spotlight. Gnome-Do works just fine with an extended monitor, but docky requires compiz to be running in Normal mode and if you recall compiz doesn't work for me when using an extra monitor.

This is probably a pretty minor compliant verses all the ones I had against Windows, so I don't expect any new patches coming out of Canonical. In an ideal world, my extra monitor would just be automatically detected, compiz would work, and therefore gnome-do docky would work.

Maven Global Excludes

2009-06-19T11:30:00.013-05:00

To my knowledge, maven2 currently does not have the ability to globally exclude dependencies. Instead there is the tedius way of excluding a transitive dependency inline with the direct dependency (see Conflict Resolution) For complex multi-module projects, this can be difficult to manage and having the ability to exclude a dependency globally could be very useful. Seems like others share the same feelings (MNG-3196). Unfortunately, for maven2 users this is targeted for maven3. So until then, here is a tip on how to globally exclude dependencies in your project (provided by my co-worker Ron Alleva).

To globally exclude a dependency all you need to do is set the dependencies scope value to provided. This supports excluding transitive dependencies, which is really what you want.

So for example, let's assume I have a WAR project that depends on commons-logging-1.1, which according to "mvn dependency:tree" has a transitive dependency on avalon-framework-4.1.3.

[INFO] +- commons-logging:commons-logging:jar:1.1:compile
[INFO] |  +- logkit:logkit:jar:1.0.1:compile
[INFO] |  \- avalon-framework:avalon-framework:jar:4.1.3:compile

Assuming I want to exclude avalon-framework from my WAR, I would add the following to my projects POM with a scope of provided. This works across all transitive dependencies and allows you to specify it once.

<dependencies>
  <dependency>
      <artifactid>avalon-framework</artifactid>
      <groupid>avalon-framework</groupid>
      <version>4.1.3</version>
      <scope>provided</scope>
  </dependency>
</dependencies>

This even works when specifying it in the parent POM, which would prevent projects from having to declare this in all child POMs.

Ubuntu, Oracle XE, and SQLPLUS

2009-05-18T02:28:00.003-05:00

In the past for local development, I have used MS SQL Server running in a VMware windows instance, but that got to be too burdensome and consumed to much of my 2GB of RAM (who would have thought that 10 years ago when I was playing Star Craft on a desktop with 32MB of RAM). Anyways, on a recent business trip with a co-worker (Matt White) who also runs ubuntu, he brought to my attention Oracle XE and how easy it was to install via apt-get and how small a footprint it was considering it's a database and it's Oracle.

I have been very impressed so far and would highly recommend it for linux users wanting a local database. Again, not only is it easy to install via apt-get once you add the repos, but I really don't notice it consuming too many resources.

Two hints I would like to self document more than anything is after installation the name of the SID is XE. You don't specify it during installation, but that is what it is. So my connection string in jboss looks like this:

jdbc:oracle:thin:@localhost:1521:XE

The second hint I wanted to self document is how to get sqlplus working. Personally, I'm often times too lazy to write straight SQL to manipulate data manually. Not only that but it's not a real good use of my time. But after this weekend I got familar with it again due to a lack of a good GUI tool like Oracle SQL Developer at one of our production sites. I was basically forced to use sqlplus to change a few values. The one huge benefit it has, is you don't have to wait on some slow GUI tool to load. So locally I now have, Oracle SQL Developer for extended use, got the SQL Query Plugin in Idea to use when writing code, and now sqlplus. So when I am impatient and I don't have Idea up, I plan on using sqlplus.

It doesn't work right out of the box. You have to set ORACLE_HOME and add it's bin directory in PATH and also set the ORACLE_SID.

I added the following to my home's .bashrc file:

export ORACLE_HOME=/usr/lib/oracle/xe/app/oracle/product/10.2.0/server
export ORACLE_SID=XE
export PATH=$ORACLE_HOME/bin:$PATH

After that reload the .bashrc file by running . .bashrc and then run sqlplus.

Mock Testing with Groovy

2009-04-18T11:49:00.016-05:00

Mock classes enable developers to quickly write unit tests that would otherwise require integration tests because of the need for a database, web container, or servlet container. Using mock classes helps to test a class in isolation and enables rapid feedback. It's not ideal to have a project with only integration tests and no unit tests. Mock classes enable unit testing that otherwise would be impossible.

So how does one create a mock class? Well, there definitely is not a shortage of mock frameworks: EasyMock, jMock, Gmock, MockFor and StubFor. You can always just create your own mock class in your test suite (which I have done in the past when in a pinch). But in my opinion these solutions lack one thing: the ability to quickly create a simple mock that when called returns what I want. To many of the mock frameworks force you to jump through hoops and call methods like expect(), replay(), verify(). What I want is the ability to define a mock class in a single line and inject it myself.

I thought MockFor and StubFor would be the solution, but the documentation is lacking and I haven't figured out how to make it work for me. Ideally I would like to say something like:

def mock = new MockFor(ICarDao.class) {
   getCar: {return new Car(color: "blue")}
}

Then MockFor would mock out the remaining methods of ICarDao and now I have a mock class that implements the getCars method that when called by the Class Under Test (CUT) will return a single Car model. But MockFor doesn't work like this and neither do any of the mock frameworks to my knowledge.

There is hope however. Below you can read about 2 alternatives: groovy's metaClass and as keyword. Both require the use of groovy in your tests. If you haven't switched to using groovy to write tests yet, even for Java, then it's time to start now. There is no other framework or library that can make you more productive when writing tests. It's an instant boost.

Groovy's metaClass
As seen in this example, groovy's meta programming is very powerful. In that post I show how one can essentially mock out Thread.startDaemon() by using Thread.metaClass.static.startDaemon. Groovy's meta programming is very powerful as seen by it's heavy use in grails to make things simple. But it doesn't work in all cases.

Groovy's as keyword
Using metaClass is by far the easiest and my favorite way to create a mock class. However, this didn't work for me in my recent attempt to write some unit tests for a Java Manager class that used spring to inject a DAO that the manager used. It didn't work I believe because my Manager class never created the concrete DAO. It defines some getters and setters and expects spring to inject the concrete class. Because of this metaClass didn't work (bummer). So I did a lot of research to come up with a competitive alternative: groovy's as keyword.

Let's start by defining the Manager class:

public class CarManager {
   private ICarDao dao;

   public void startCar() {
       Car car = dao.getCar();
       .......
   }

   public CarManager setCarDao(ICarDao dao) {
       this.dao = dao;
       return this;
   }
}

Now to test this using mock classes and the as keyword all you need to do is this:

class CarManagerTest extends GroovyTestCase {
   def void test_start_car() {
       ICarDao mock = [
           getCar: {return new Car(color: "blue")}
       ] as ICarDao;

       def cut = new CarManager().setCarDao(mock);
   }
}

This uses a map and the as keyword to implement an interface. Here the key is the name of the method to mock and the value is a closure of what you want returned when called. And there is no need to define all the methods of the interface, just the ones you want to mock out.

To me, metaClass and the as keyword are much cleaner and simpler compared to the current mock frameworks. At least for this type of testing. Those frameworks might be perfectly useful for other types of testing, I just haven't ran into them yet.

Better Offline Capabilities with Maven 2.0.10

2009-04-13T20:34:00.002-05:00

This week while traveling on business, I had hoped to get a lot of work done, but was quickly disappointed when I wasn't able to build because maven couldn't download the latest SNAPSHOTs. Even though I had fresh local SNAPSHOT versions that would suffice.

Fortunately, maven 2.0.10 was recently released and it promised fixes for this exact situation (see release notes). Currently I am happily using version 2.0.9.

So now that I am at the hotel I decided to see if an upgrade would help me. I first built the submodule again to make sure I got the dreaded unable to download dependency. Downloaded and installed 2.0.10 and rebuilt again. And I am very excited to report that it worked

To stay tuned into everything Maven, subscribe to Brian's Enterprise Blog at Sonatype. Brian Fox is one of the head developers for Maven and according to Google Reader I read 100% of his posts.

grails create-app esb

2009-03-13T23:17:00.005-05:00

I know it seems like a strange app to create with grails, especially when there a several other capable opensource ESBs available (mule, servicemix, openesb), but instead of asking yourself why, ask yourself why not. While it may not offer all the same features (BPEL) it's certainly a possibility for certain circumstances. This possibility was spawned together with co-worker Kit Plummer when discussing different options for an upcoming story that required email integration. At first it seemed kind of ridiculous, especially since we were already using openesb in jboss, but it started to make some sense the more we thought about it.

So how could a MVC web framework possibly replace a feature complete ESB? Well, first let me explain my background. I don't consider myself an ESB expert, but I do have some experience with ServiceMix and OpenESB (see openesb topics). In fact, my former company, let our team develop and open source 4 JBI Binding Components for RSS, SIP, UDDI, and XMPP.

Here was a short list of complaints I had with running OpenESB v2 in jboss:

At first it was pretty simple to setup, install, and run for a single developer, but trying to duplicate that across a large distributed team and things get more complicated. This included the difficulty of setting it up in all of our CI and beta environments. It's not as easy as just running Glassfish which includes OpenESB.
OpenESB v2 basically required Netbeans, which again isn't too hard for one developer. But asking your team to run a second unfamiliar IDE is no easy task. The OSGi based OpenESB v3 does not require Netbeans, but it does make it easier.
Composite Applications are less than easy to create, test, maintain, version, deploy in CI, etc. At least not compared to a Grails WAR anyways. Being able to consistently do those 5 things over 12-24 months is really bigger than you think.
Security. It's more difficult to lock it down compared to a WAR running in jboss fronted by apache.

Here are some advantages we saw in treating Grails like an ESB:

Easy. Simple. Trivial for everything including: developing, maintaining, testing, deploying, versioning, securing, and installing.
Grails is plugin based and has a growing number of good plugins. One of the main benefits of an ESB is leveraging all the other work so you don't have to write anything. Things like HTTP, JMS, SMTP, JDBC, RSS, XMPP, FTP, BPEL, XSLT, and FILE just to name a few. Granted many Grails plugins are web focused, but there are several similar capabilities such as HTTP, JMS, JDBC, SMTP, RSS, and Workflow. Beyond that writing your own Grails plugin is easy compared to writing your own ESB component. See the Mail plugin as an example of how easy it is to send an email in Grails.
Doesn't require Netbeans. Developers can continue using their favorite IDE.

Despite all of that, I do think the case can be made better for OpenESB if your team is already using Glassfish+OpenESB (or GlassfishESB) and Netbeans. But it does make it much more difficult if your not. And I know that ServiceMix v3 was deployable as a WAR, but that was not it's default behavior. Not sure about the OSGi based v4, but I can't imagine they stopped supporting WAR deployment. Of the two I think ServiceMix reminds me more of a Grails app as far as simplicity is concerned.

There is one big disadvantage to using Grails like an ESB: fewer incoming protocols. I could be wrong on this one but with grails your probably limited HTTP and maybe JMS (outside of setting up quartz jobs and polling). But with an ESB its really unlimited (HTTP, JMS, JDBC, SMTP, SIP, XMPP).

I am sure I am missing several other key pieces, so interested in hearing from others. The nice thing is our implementation is hidden behind a REST API that could easily be supported by a bloated ESB.

Find when a branch was created in svn

2009-02-26T09:21:00.003-06:00

If you merge between branches and HEAD in subversion, you most likely need to know at what revision the branch was created at. Assuming you don't have the luxury of the new merging features in subversion 1.5, here is a trick I learned from the svn docs.

svn log --stop-on-copy http://server/svn/myapp/branches/myapp-1.0

This will stop once it hits the revision the branch was created at, verses continuing on until r1. Previously I would log the entire branch, and do a grep for the comments I inserted when I created the branch. Not ideal but it worked. Now I use the --stop-on-copy option and I know real quick the revision the branch was created at. Giving me the revision I need to use in the merge.

svn merge -r 546:767 http://server/svn/myapp/trunk

Can't wait until we upgrade to subversion 1.5 or a DVCS.

Security news for work and personal

2009-02-03T22:10:00.009-06:00

Security Now is a popular podcast that discusses important issues related to personal computer security. It's led by TWiT (This Week in Tech) producer Leo Laporte and SpinRite creater and security expert Steve Gibson. However, it's not only for geeks like myself but very understandable by non-geeks. Each week Steve does an excellent job of explaining complicated concepts in layman terms. It's really good in that each topic applies to not only my work but also home computer security. While on my trip to Korea, I was able to get caught up on the past few episodes and consequently want to share with the Lorenzen Nation the things I have learned.

For example, one of the first things I learned when first listening to SN (Security Now) was how WEP was broken. I know I know, pretty pathetic, but I actually never knew this. My guess, or hope, is someone reading this actually didn't know it either. So if you are running a WEP wireless network at home or your business, and assuming you want that network inaccessible by outsiders, like your neighbors, then consider switching to a WPA network. Apparently joining a WEP enabled network is about as easy as joining an unsecured network.

Secondly, I learned a lot about changes I could make to my home network to make it more safe and secure. Specifically, this weekend I switched my networks DNS to use OpenDNS. The main reason was for its parental controls capability. Before I made the change I was easily able to visit an adult content site. Once I switched to OpenDNS and selected the filtering ability, I was no longer able to visit these sites. Now I feel a lot more comfortable knowing that when my kids are on the computer, these sites aren't going to pop up.

Next, SN devoted several episodes to DropMyRights and Sandboxie. Both are used to help reduce windows from getting infected by malware. I would recommend any Windows user at least using DropMyRights. It's free, easy to install, and easy to use. I started using it this weekend and it's worked perfectly so far. DropMyRights was created by a Microsoft employee who wanted to login as an admin, as we all do, but still run certain applications with restricted rights. Why is that important? Well among the many things malware does, all of which require admin rights, are:

Creating files in the system32 directory.
Terminating various processes.
Disabling the Windows Firewall.
Downloading and writing files to the system32 directory.
Deletes registry values in HKLM.

All of this stuff fails if the user is not an administrator. But developers hate running as a non-admin, so the solution is install DropMyRights or not run winblowz. Then to run Firefox you run something like the following: "C:\Program Files\DropMyRights\DropMyRights.exe" "C:\Program Files\Mozilla Firefox\firefox.exe".

Sandboxie, is another neat security application that lets you run any application or drive basically in a separate sandbox. So when you run Firefox in sandboxie, if malware gets installed, it's installed in the sandbox and not your OS. It's very flexible; a caller even said they ran a thumbdrive in a sandbox. This one costs money so I have yet to install it.

Finally, I learned that if you can afford over 200 PS3's and are incredibly smart about cryptography you can crack the md5 hash and create your own valid fraudulent certificate. Over the past few years, researchers have gradually weakened md5, but have finally basically broken it to where no one should be using it now in certificates. This one applies at work and home. Not only should I not be creating certificates at work using md5, but at home I should not visit sites over https that use certificates that use md5. See the resource notes for episode #177 for further information (under Breaking SSL by Spoofing a Certificate Authority). I found several trusted certificate authorities defined on my home computer that use md5. Even a few expired certificates using md2, which I guess malware could change your system time if you weren't running DropMyRights. If you want to see this in action, set your system time to August 15th, 2004 and then visit this site https://i.broke.the.internet.and.all.i.got.was.this.t-shirt.phreedom.org and check out the certificate. This will setup a secure connection using a fraudulent cerificate.

In summary, I have learned a great deal about work and home computer security by listening to the Security Now podcast. Even if I didn't understand all of the details, it has definitely made me a more aware user security-wise. Do as I did and get rid of your WEP network, switch to OpenDNS, install DropMyRights on Windows, and subscribe to the Security Now podcast (very easy in iTunes).

Testing REST Services with Groovy

2009-01-17T02:32:00.009-06:00

For awhile now, RIAs (Rich Internet Applications) have rapidly started replacing traditional server-side web applications (JSP, JSF, etc). Typically, these flashier sites are created using Flex or javascript libraries like extjs or yui. At the heart of these sexy applications live simple REST services that return JSON or XML.

Testing these REST services should be made a high priority for several reasons:

It's the contract between the client and server. Breaking that contract should be avoided.
Your application may not be the only client. Once external parties start consuming your REST services you'll want tests in place to ensure you don't break their clients.
To validate the response is well-formed XML or properly constructed JSON
Valuable black-box test, testing the entire server-side stack starting from the REST layer going all the way down to the DAO layer to the database.

So, what's the easiest way to test REST services? For awhile now I have been combing the internet for the best tools to accomplish this goal, since I wasn't going to do it in pure Java, and I think I finally found the right combination using Groovy and HttpBuilder. Groovy because its super easy to parse XML and JSON, and HttpBuilder because it's a great wrapper for the popular Apache Commons HTTP Client library.

Now let's say you have a REST service at the URL http://localhost:8080/myapp/api/books that returns this JSON:

{"books":[
  {"name":"Being Mad","author":"Kit Plummer"},
  {"name":"Clown Life","author":"Josh Hoover"},
  {"name":"You Miss Me","author":"Ron Alleva"},
  {"name":"Talk to me Goose","author":"Jeff Black"}
]}

This is how simple it is to write a test in Groovy using HttpBuilder:

import groovyx.net.http.HTTPBuilder
import groovyx.net.http.Method
import static groovyx.net.http.ContentType.JSON

class BooksTest extends GroovyTestCase {
  def void test_get_all_books() {
      def http = new HTTPBuilder("http://localhost:8080")

      http.request(Method.valueOf("GET"), JSON) {
          url.path = '/myapp/api/books'

          response.success = {resp, json ->
              json.books.each { book ->
                  assert book.name != ""
              }
          }
      }
  }
}

To me the major advantage to this approach is being able to traverse the JSON response like I would in javascript. If the response was XML it be just as easy too.

The only two remaining items you would need would be adding the gmaven plugin to your pom and httpbuilder as a dependency.

Control versioning in new maven-release-plugin

2008-12-17T00:08:00.002-06:00

I have been waiting with much anticipation for the new version of the maven-release-plugin to be, well released. Our team uses this time saving plugin heavily but since moving to a MAJOR.MINOR.PATCH version strategy we quickly found out that it needed some new features. What we really needed was the ability to specify versions for the tag and for the new trunk. For example, when HEAD is at 2.1.0-SNAPSHOT, while in batch-mode the new default version would be 2.1.1-SNAPSHOT, but I want it to be 2.2.0-SNAPSHOT.

Fortunately, someone else was thinking the same thing and the new capabilities where already in the works. Our current version we are using is 2.0-beta-7. But I noticed today the maven team finally released 2.0-beta-8 (see repo1). Here are the release notes for the new version.

What I was specifically interested in was the new batch-mode features. Now instead of having to manually update the POMs after creating a release I can do the following automatically in CI:

mvn release:prepare -DdevelopmentVersion=2.2.0-SNAPSHOT

I haven't tested it yet, but will know in the near future if it works.

Maven Profiles: Something you need to know

2008-12-05T11:17:00.010-06:00

An issue came up today in our CI environment and I learned something new about maven profiles. Something that everyone who uses maven should know. Perhaps, like your environment, our project uses maven2 to build and we take advantage of multiple profiles (see our example here). A common use for profiles is to define new modules and properties. For example:

<profiles>
    <profile>
        <id>integration-tests</id>
        <modules>
            <module>integration</module>
        </modules>
        <properties>
           <app.port>8080</app.port>
        </properties>
    </profile>
<profiles>

This profile is activated when you run mvn -P integration-tests install. But sometimes you want a profile on all the time. Maven profiles have the ability to be actived by default. To do this just do the following:

<profiles>
    <profile>
        <id>integration-tests</id>
        <activation>
            <activeByDefault>true</activeByDefault>
        </activation>
        <modules>
            <module>integration</module>
        </modules>
        <properties>
            <app.port>8080</app.port>
        </properties>
    </profile>
<profiles>

Then you can just run mvn install and that module and property will already be active.

activeByDefault.....Sometimes
Turns out that perhaps maven should have renamed it to activeByDefaultSometimes. Reading their documentation I learned something important about maven profiles:

"This profile will automatically be active for all builds unless another profile in the same pom is activated using one of the previously described methods. All profiles that are active by default are automatically deactivated when a profile in the pom is activated on the command line or through its activation config."

So in my example, if I have more than one profile defined in my pom and I run that new profile, then my integration-tests profile is deactivated. For example, when I run mvn -P SecondProfile, the module and property defined in the integration-tests profiles are not available to my build. One trick to see which profiles are activated when running maven is to run mvn help:active-profiles.

In general I have never been a huge fan of profiles. They feel clumsy, clutter up the cmd line, and are not well known among your team. As their documentation states, "adding profiles to your build has the potential to break portability for your project". Instead of using Profiles, attach to known phases. For example, see my second example from this previous post. In this example, I use the maven-antrun-plugin to attach to the install phase. Now this will run anytime I run mvn install. No clumsy profile I have to memorize and run. Given that, I am not saying you should avoid Profiles all together. Just use them as a last resort.

Spare cycles to write some posts

2008-11-05T00:09:00.003-06:00

Starting this week I am going to spend the next couple of weeks in Panama City, Florida on business working with the 1st AF here at Tyndall Air Force Base. In my down time, like right now while I copy over hundreds of MBs over a slow VPN connection, I plan on catching up on some blog ideas I have been wanting to write.

Here is my goal:

Discuss and promote our iPhone app BorrowMe
Multi-part series on security. For the past two sprints I have written about 6 lines of code, but I have learned a ton about securing our web based application using LDAP+Active Directory and SmartCard authentication or CAC (Common Access Card) authentication as referred by in the DoD. (U.S. Department of Defense), Apache, SSL, and OpenSSO.
Talk some more about Ubuntu JeOS

More tips on using the maven2 release plugin

2008-10-22T09:47:00.005-05:00

This blog's most frequently visited post was the one I did over a year ago titled "How to create a release using the maven2 release plugin". Automating this portion of our frequent release process without a doubt has saved my team hundreds of hours over the past year. For that reason I would like to provide some new tips I discovered yesterday when I needed to change the subversion comment used by the release plugin when committing any changes.

For teams not using the maven-release-plugin yet, this handy plugin helps automate the complete process when releasing your software. It helps save time by doing the following and more:

First it will build your project running any tests you specify
If successful, it will then commit your project into a tag
Update the tag pom versions from say 1.0.0-SNAPSHOT to 1.0.0
Change the tag pom SCM URL to correctly point to the tag instead of HEAD
Most importantly, it will build the release (1.0.0) and upload the artifacts to your companies maven2 repository (archiva now in our case; was artifactory).
Finally, it will increment the pom versions in HEAD to be the next release (1.0.1-SNAPSHOT).

Tip #1 - Change Commmit Comment
By default the maven-release-plugin uses a comment like, [maven-release-plugin] prepare release project-1.0.0, when doing any type of commit such as creating a tag/branch or incrementing the pom versions. Which works wonderful until your team implements a Subversion pre-commit hook requiring all commits to start with certain keywords (Story: ZZZ, Jira: ZZ-N). Luckily the maven-release-plugin has an option to override the comment.

There are two simple ways to provide the plugin with the comment. The first sets the system property, scmCommentPrefix, to the predefined prefix. The second, provides the prefix comment in the pom.


mvn release:clean release:prepare -DscmCommentPrefix=Jira: AC-100 [maven-release-plugin]

<plugins>
 <plugin>
     <groupId>org.apache.maven.plugins</groupId>
     <artifactId>maven-release-plugin</artifactId>
     <configuration>
         <scmCommentPrefix>Jira: AC-100 [maven-release-plugin]</scmCommentPrefix>
     </configuration>
 </plugin>
<plugins>

Tip #2 - Configuration Options via -D
Disappointed that I didn't catch this earlier, especially since I have been using -DpreparationGoals for awhile now, but all of the optional parameters for the prepare goal can be defined with -D rather than in the POM (as above). So if you wanted to change the default tagBase to use /branches instead of the default /tags do the following. However, I would still put this static value in the pom.

mvn release:clean release:prepare -DtagBase=http://project.svn.org/svn/project/branches

Tip #3 - Allow SNAPSHOT dependencies
This new feature is going to be huge for legacy projects who weren't raised with the release plugin. As mentioned in my previous post, previous versions of the release plugin didn't allow you to have SNAPSHOT dependencies; rightfully so too since that wouldn't be a very good practice to have a release depend on a changing dependency. However, I think this feature prevented many projects from using the release plugin because their snapshot depenencies were too many to change and it was just easier to continue doing same old.

Even though I have not used this feature, it would seem now one could release a project using this plugin and still have SNAPSHOT dependencies. However, after creating the release it would still be a good idea to go back in the release and change any SNAPSHOT dependencies to releases.

I believe this is a new feature so you might also want to upgrade to the latest version.

Using Yahoo Pipes screen scraping to create an RSS Feed

2008-10-15T23:12:00.008-05:00

The internet amazes me. Every time I use it, or write something for it, I continue to be blown away. Take for example, my favorite website for Husker information: http://www.huskerpedia.com (for those that don't know, what I mean by husker information is Nebraska Cornhusker Football. I was born in Nebraska so I naturally follow the cornhuskers.). Someone takes the time to consolidate all husker information and updates this site. It is the de facto standard for Husker news. Unfortunately, this site does not come with any type of notifications via an RSS feed.

Now this site does not necessarily amaze me, but rather how I can take a tool like Yahoo Pipes, scrape the latest news, and in a couple of hours subscribe to that feed in Google Reader. Not only that, but I cloned Paul Arterburn's Huskerpedia Pipe to get started and now everyone can take advantage of my new feed by either subscribing to it, making it better, or cloning it for their own use.

This would make the second time I have used Yahoo Pipes to accomplish something complex very easily. Several months ago, I created a pipe called the Gestalt Shared Feed, that combined all of my co-workers shared items from their Google Reader feed. This allowed us to read the best of the best among ourselves and has really helped our communication. I think we have about 10 people who share items and many more who subscribe to it.

Check out my RssHuskerPedia Pipe and view the source if you are curious. See below for the before and after.

Me lovin Git

2008-08-28T00:20:00.002-05:00

As I stated previously, I am sold on DVCS, and today it just got better.

I came across a situation where I needed to share highly experimental code with a teammate. There was no way I could check this code into our normal repo, so what first pops into your head: email (strongbad style). You've been here and don't pretend you haven't used email. Oh yes the wonderful land of emailing around jar or zip files repeatedly.

Emailing source code around stopped today. Instead I got on the phone with Ron Alleva who helped walk me through the simple git commands to create my own local master repo. It was so simple it was ridiculous.

Here are the steps to create and share hack code in minutes

install git
cd to the folder you wish to share
run git init
run git add .
run git commit -a
Have your teammate clone it by running git clone ssh://user@host/path/to/repo.git/

Ironically after my teammate first cloned it and tested the code it failed because of a recent change I made. So I made the fix, committed it to my local repo, and he was able to git pull those changes. Also, if he makes any changes we can share them easily as well by him using git push.

Testing the new firefox plugin Ubiquity

2008-08-27T21:50:00.000-05:00

Today I found out about the new experimental firefox plugin Ubiquity and I wanted to give a try in my blog. So far I am very impressed and it seems to be working fine on linux, even though the documentation says they don't support it yet (I am running Ubuntu 8.04 with firefox 3).

It was easy to install. After installation firefox takes you to the firefox egg about:ubiquity which contains one of the best tutorials I have ever followed.

To try it out you can get a map of where I work. First install ubiquity and highlight this address:

407 Pennsylvania Ave Joplin, MO

then invoke ubiquity (CRTL+SPACEBAR) and type map. Click on the map and then click on Insert map in page.

Pretty slick. So far today I have used the language translation features, bold, tinyurl, and map. In a way it reminds me a lot of Gnome-Do for linux in that it provides a quick way to get information but obviously ubiquity is much more powerful.

Waiting for Free iPhone SDK

2008-08-18T23:37:00.002-05:00

What is this a Starbucks waiting line? Am I trying to download a microsoft product? Sure seems like it. Tonight I had a goal of getting my first Hello World app working for an iphone, but apple had other plans. While waiting for my turn in line to download the free SDK I watched the following video "Hello World Final SDK".

Getting started with Grails and Extjs

2008-08-09T23:48:00.006-05:00

This article describes how to get started using extjs with your grails app. Since the plugin is deprecated because of the GPL license fiasco, I decided to write my own simple grails script to handle installing extjs for the team for 2 main reasons:
1) Prevents me from having to commit 500+ files into SVN for every new version
2) Makes it easier to upgrade to newer versions of extjs in the future

We have 2 existing internal applications that use extjs and every upgrade I am kicking myself for not using maven to extract the extjs zip file. So with our new grails app I wanted to not make the same mistake.

Install Extjs
1) Download your choice of extjs. Note as of version 2.1 extjs is under the GPL license. Meaning if your project isn't open sourced under the GPL or an internal company app then you need to use the 2.0.x versions (2.0.2 is the latest). In our case I downloaded version 2.2.

2) Copy ext-2.2.zip into your grails plugins directory

3) cd into your grails application

4) Add the zip file to svn (or whatever): svn add plugins/ext-2.2.zip

5) run grails create-script install-extjs

6) Add the script to svn: svn add scripts/InstallExtjs.groovy

7) Modify the InstallExtjs.groovy script and add the following GANT code.

8) run grails install-extjs

9) Exclude the unziped ext directory: svn propedit svn:ignores web-app/js Exclude the folder ext.

Test it out
Now that you have extjs installed you can copy one of their examples into the grails web-app directory and update the links.

1) Open up the ext-2.2.zip file again and extract the array-grid.html and array-grid.js files from the examples/grid folder to the grails web-app directory.

2) Modify the array-grid.html file. Update the relative links for css and javascript by replacing ../../ with js/ext/. For example, href="../../resources/css/ext-all.css", should now be href="js/ext/resources/css/ext-all.css"

3) Open up the array-grid.html file in your browser.

Now you have extjs installed with a simple example on how to use it. Next I would like to create a grails controller that returns JSON to populate a simple grid.

Grails: Lessons Learned

2008-07-20T06:33:00.006-05:00

Here are some lessons learned concerning lazy vs eager fetching and how to delete a child object in a One-to-many relationship (unfortunately it's not super obvious).

First Lesson: Don't set eager fetching globally
I by no means am a grails expert, but based on my experience don't set the eager fetching in your Domain as the example shows (unless you have a very good reason and understand the consequences). Lazy vs Eager fetching is well described in the grails documentation, so I won't repeat it, but anyone using One-to-many relationships needs to know the differences.

The default behavior in grails is lazy fetching, which results in n+1 queries. In some cases this might be ideal, in others it may not. When it's not you have a couple of choices. The example in the grails documentation sets a fetchMode property on the Domain. This sets it globally and every time the Domain is accessed, grails is going to load all it's many relationships. The path I recommend is to specify the fetch mode when retrieving the data. For example, the list() method has a parameter called fetch and can be used like this: Book.list(fetch: [authors: "eager"]). This gives you the most flexibility by not specifying the fetch mode globally, but allowing you to fetch eagerly when necessary.

Second Lesson: Use Hibernate Events to help remove associations
Like myself, you might actually have a One-to-many relationship where you need to delete a child. Unfortunately this use case isn't documented very well and it actually took me a little bit to figure out.

So lets say you have the following two domains

class Parent {
  static hasMany = [kids: Kid]
  String name
}

class Kid {
 static belongsTo = [parent: Parent]
 String name
}

And you save the following

new Parent(name: "James").addToKids(name: "Ayden").save()

Now Ayden turns 18 and going off to college and you need to remove him. You might think this would work:

Kid.findByName("Ayden").delete()

But it doesn't because the parent James still has a reference to the kid Ayden in the kids list (parent.kids). So you have to do the following:

def kid = Kid.findByName("Ayden")
kid.parent.removeFromKids(kid)
kid.delete()

Why that isn't the default behavior in grails I don't know, but to prevent you from repeating code everywhere you can use Hibernate events. In the Kid Domain add the following beforeDelete property:

class Kid {
   static belongsTo = [parent: Parent]
   def beforeDelete = {
       parent.removeFromKids(this)
   }
   String name
}

And now when you want to remove a Kid, all you need to do is call kid.delete(). The hibernate events are interesting. By default grails supports 4 events: beforeInsert, beforeUpdate, beforeDelete, onLoad. However, there is a recent plugin called Hibernate Events Plugin that adds 7 more events: beforeLoad, afterLoad, beforeSave, afterSave, afterInsert, afterUpdate, afterDelete.

Grails JSON Parser

2008-07-18T06:35:00.004-05:00

Here is a quick example on parsing JSON in grails using groovy (surprisingly, google isn't returning any good hits). Also, if you needed this ability in just straight groovy, I am sure you could include the specific grails jar in your classpath.

import grails.converters.*

def jsonArray = JSON.parse("""['foo','bar', { a: 'JSONObject' }]""")
println "Class of jsonArray: ${jsonArray.class.name}"
jsonArray.each { println "Value: ${it}" }

FYI, it appears from the mailing list this was added around 1.0 RC1.

Building JSON is super easy too in grails/groovy using the render as. And don't forget to import grails.converters.

render Book.list(params) as JSON

Update: Read my recent article on testing REST Services that return JSON using groovy and httpbuilder.

Groovy Threads and MetaClass example

2008-07-11T09:15:00.006-05:00

* Update - the code has been updated. The original test was incorrect and was producing a false positive. What you see now is the correct way.

It's been awhile since I have been able to play around with grails/groovy. Now, instead of using a pretend app to learn grails/groovy, I have teamed up with Jeff Black, Chad Gallemore, and Sam Jones (fellow office co-workers here in Joplin, MO) to rewrite an existing small internal java webapp using grails. It's a perfect application for grails and so far we are loving it.

Early on, we needed to figure out 2 things:
1) Groovy way of creating Threads
2) Writing an integration test for a Groovy Service

Groovy Threads
Below is a summary of our service that shows how to start new threads in groovy.

class ProjectService {
  def discover() {
      Project.list().each {
          Thread.startDaemon {
              jobService.update(it)
          }
      }
  }
}

Service Integration test with MetaClass
Here is the integration test I wrote that tests the above Service.

void testEmptyProject() {
   def called = false

   Project.metaClass.static.list = {[]}
   Thread.metaClass.static.startDaemon = {Closure c -> c.call()}
   JobService.metaClass.update = {called = true}
   new ProjectService(jobService: new JobService()).discover()

   assertFalse('updateJobs should not have been called', called)
}

The first missing key for me was the static keyword on metaClass since in the service I am calling Project.list() and Thread.startDaemon(). The second mystery was how to mock out Thread.startDaemon() since there could be, and was, a race condition between the update closure setting called = true and my assertFalse.

Thanks Chad for the suggestion of using metaClass. I also got a lot of help from Glenn Smith's blog about testing controllers and Dustin's groovy thread example.

Maven not downloading latest snapshots or releases

2008-07-03T12:28:00.005-05:00

Ever had issues with maven not downloading the latest snapshots when you know for a fact new snapshots are available? Or your CI environment just deployed a new release (2.0), but when another Hudson job builds, maven does not download the latest 2.0 release artifact. Want an automated solution so you don't have to manually delete the artifacts from your local repository just so maven will download the latest?

Force maven to download latest snapshots
Our company uses Hudson for our automated CI environments. Our project basically has two jobs. The first job checks out and builds HEAD when modified and deploys SNAPSHOT WARs to our companies maven2 repository (artifactory). The second job, which builds nightly, uses maven to download the SNAPSHOT WARs from artifactory, creates an EAR, deploys it to JBoss, and runs integration tests. By default, maven will check once a day for changes to snapshots, so when our second job was triggered, maven inside hudson was not downloading the latest SNAPSHOT WARs.

The solution was to append the -U in the maven goals (run mvn -help). It stands for update-snapshots and tells maven to update all snapshots no matter what.

Force maven to download latest releases
Our next problem was when we created a branch and started creating release artifacts such as 2.0. Unfortunately the description given by maven for the -U option is incorrect (or at least in v 2.0.9), "Forces a check for updated releases and snapshots on remote repositories". As much as I tried, the -U option wouldn't work in our hudson job to force maven to download the latest non-snapshot releases.

The only current solution I know of is to use the maven-dependency-plugin and its goal purge-local-repository. So in your maven goals at some point execute mvn dependency:purge-local-repository and maven will physically delete your projects artifacts from the local repository (/home/user/.m2/repistory) and its transitive dependencies (I think). I tried setting the actTransitively to false and it didn't work for us so I just removed it. I also set verbose to true so I could see what maven deleted in Hudson's console output.

The pipes are used to separate out different goals to isolate its classpath or properties. That way we can skip tests in one run, and then run them in the next all in the same goals section.

DVCS? I'm sold

2008-06-26T00:01:00.002-05:00

As a casual reader of DVCS advantages like git and mercurial, I think I am ready. Motivated by a great article shared by Kit Plummer, I am sold and ready to start using it. Unfortunately, it's probably not going to happen any time soon at work until a) IT supports a mercurial server like they do svn or b) I start a new project were new decisions like language (ruby/groovy) and source control can be made and I'm not hindered by "we are already using svn, why switch" or "I just got comfortable with svn" or "every team member would have to learn mercurial". Reminds me of my favoriate Zed Shaw quote from this article:

"This folks is the classic problem with programmers today. They absolutely refuse to learn anything new unless they can see that learning the new hotness will give them an immediate 200% boost in salary or get them hot honeys at the next conference."

Now that I see that Hudson has a mercurial plugin and Jira does as well, then technically speaking, I can't think of any other reason not to switch.

How to FTP artifacts in maven2

2008-06-15T21:02:00.007-05:00

If there is one word a team should keep in mind when building a CI (Continuous Integration) environment it's Automate, Automate, and Automate (see Production-ready software, on-demand). Most teams, including mine currently, performs their releases manually; creating the branch, incrementing the POMs, uploading the artifacts to a repository, etc. This is one reason I wrote about using the maven release plugin to automate this process. I will say that having a CI server such as Hudson only helps in making automation easier.

My team also creates an official release about once every two months and supports that release anywhere between 1-2 weeks to months. It's also pretty intense during those 1-2 weeks when we have co-workers on-site installing and supporting our software in a completely different timezone and with limited access to the phone and internet.

Over the weekend I wrote a simple maven2 pom to help automate FTP'ing the release artifacts to an FTP server. Last week we created a branch and a job in hudson that builds that branch. After it successfully builds we use the assembly plugin to package our EAR, documentation, SQL files, and everything else into a single folder. In the past, a guy would then manually copy these files to the FTP server, which was used by the on-site team to download the latest artifacts containing improvements and bug fixes. This release we wanted to automate this step to increase our response time for the on-site team.

First Try using GMaven Plugin
I knew that Ant had an FTP task and I love doing Ant in Groovy because its so much easier so I decided to first try the GMaven plugin (maven groovy plugin). It was a short trip since the FTP task is an optional library in Ant and you have to include the jar in your POM, but I could never get the gmaven plugin to recognize the dependency (see MGROOVY-152). Look at the Jira issue attachments if you want to reference my POM as an example. It's too bad I couldn't get this to work because it would have been sweet:

log.info('Entering in ftp script')

def config = [server: 'localhost',
   remotedir: '/home/jlorenzen/ftptest',
   user: 'ftp',
   password: 'ftp'
];

ant.ftp(config) {
   fileset(dir: '/home/jlorenzen/Documents')
}

Second Try Maven AntRun Plugin
I was finally able to get this to work using the maven-antrun-plugin. Here is my POM

<build>
  <plugins>
      <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-antrun-plugin</artifactId>
          <executions>
              <execution>
                  <id>ftp</id>
                  <phase>install</phase>
                  <goals>
                      <goal>run</goal>
                  </goals>
                  <configuration>
                      <tasks>
                          <ftp server="localhost"
                                  remotedir="/home/jlorenzen/ftptest"
                                  userid="ftp"
                                  password="ftp">
                              <fileset dir="/home/jlorenzen/Documents"/>
                          </ftp>
                      </tasks>
                  </configuration>
              </execution>
          </executions>
          <dependencies>
              <dependency>
                  <groupId>ant</groupId>
                  <artifactId>ant-commons-net</artifactId>
                  <version>1.6.5</version>
              </dependency>
              <dependency>
                  <groupId>commons-net</groupId>
                  <artifactId>commons-net</artifactId>
                  <version>1.4.1</version>
              </dependency>
          </dependencies>
      </plugin>
  </plugins>
</build>

Accessing Hudson Variables with a Free-Style Project

2008-05-18T23:26:00.003-05:00

Here is a great article on creating a manifest file containing hudson variables so that you can know exactly what build number and svn number built your artifact. However, in my case I wanted to create a properties file under my maven2 project under src/main/resources and let maven filtering take care of the rest. But for some reason it wouldn't work. After some further research, I eventually found out that free-style projects appear to not work the same as a maven2 project in hudson when trying to access hudson variables.

In summary, hudson variables work as expected when you create a maven2 project in hudson. With a free-style project you have to perform an extra step. Why, I don't know; I am sure there is a good reason. It's just too late to figure it out.

Here are the extra steps you can follow to create a properties file containing hudson values.

Update POM
In your projects POM or parent POM add the following properties at the bottom

<project>
 ....
 <properties>
     <build.number>${BUILD_NUMBER}</build.number>
     <build.id>${BUILD_ID}</build.id>
     <job.name>${JOB_NAME}</job.name>
     <build.tag>${BUILD_TAG}</build.tag>
     <executor.number>${EXECUTOR_NUMBER}</executor.number>
     <workspace>${WORKSPACE}</workspace>
     <hudson.url>${HUDSON_URL}</hudson.url>
     <svn.revision>${SVN_REVISION}</svn.revision>
 </properties>
</project>

Create a properties file
Create a application.properties file under your maven2 projects src/main/resources directory.

Now add this to it

build.number=${build.number}
build.id=${build.id}
job.name=${job.name}
build.tag=${build.tag}
executor.number=${executor.number}
workspace=${workspace}
hudson.url=${hudson.url}
svn.revision=${svn.revision}

Groovy Sort List

2008-05-17T01:06:00.002-05:00

I am posting a simple example on how to sort a list in groovy because the examples google knows about aren't what I was looking for. With some deep digging I was able to find a clue that eventually solved my problem.

It's real easy to sort a list of numbers

assert [1,2,3,4] == [3,4,2,1].sort()

Or even strings

assert ['Chad','James','Travis'] == ['James','Travis','Chad'].sort()

But this was my example

class Person {
  String id
  String name
}

def list = [new Person(id: '1', name: 'James'),new Person(id: '2', name: 'Travis'), new Person(id: '3', name: 'Chad')]

list.sort() returns James, Travis, Chad

The solution is ridiculously simple (not that I thought the previous sort would work; I have to be realistic; groovy can't do everything for me).

list.sort{it.name} will produce an order of Chad, James, Travis.

In the previous example note the use of the sort closure sort {} verses the sort() method.

Now I am not sure, off the top of my head and without a Groovy book handy, the simplest way to sort case insensitive.

assert ['a1','A1'] == ['A1,'a1'].sort{fancy closure}

Finally OpenSolaris Installed

2008-05-10T00:19:00.004-05:00

Finally, after 4 attempts, I got a half way decent install of Solaris. My previous attempts included installing Solaris Express Developer Edition on VMware Server 1.0.4 and VirtualBox 1.5. Both had major issues but I had the best luck with VMware Server, it just took like 15 minutes to boot and I couldn't install vmware tools. So at JavaOne 08 they announced http://www.opensolaris.com with the version 2008.05. So I thought I would give it a try and their install documentation seemed very thorough (see how to install on virtualbox). Unfortunately, installing this in virtualbox did not work (would't boot) and so I decided to give it one more shot on VMware Server and I guess I got lucky because it actually worked to my surprise. And it actually boots rather fast compared to my previous experience with solaris.

Since it uses GNOME it reminded me a lot of my ubuntu system. I was even able to successfully install vmware-tools which I got really excited about because I would have bet against this previously. Compiz is installed by default but I was unable to change the settings yet (it kept rebooting for some reason). Maybe it's because I haven't enable the nvida drivers yet; which was also a surprise that the nvida drivers where already installed.

Here are some features compared to Linux. I am not a file system guru but ZFS sounds interesting.

Anyways so far so good. I do miss sudo. I am now able to start working efficiently on providing hudson as a package in solaris. I am assuming that once I get done the below search will actually return a result (me <-- crossing my fingers).

Why again am I not using JDK 1.6?

2008-05-08T11:33:00.004-05:00

Why in the world am I still using jdk 1.5? It's almost 4 years old. Yesterday it just kind of hit me and I said to myself, "James, why not start using jdk 1.6 or java 6?". Is there anything technically preventing jdk 1.5 users from using jdk 1.6? Back in the day, weblogic 6.1 only worked with jdk 1.3 and when I wanted to start using jdk 1.4 it just wouldn't work. So for some reason I still had that same mindset.

So I started to do a little research and what I found was pleasantly surprising and I immediately downloaded the latest version of java 6 and started using it (verdict still out on that but so far so good). JBoss 4.2.1 seems to start just fine with java 6.

So what exactly did I find? First, straight from the java 6 home page is the answer to the question: Q: How is Java SE 6 different from the previous version (J2SE 5.0): what are the improved and updated areas, such as functionality, security, performance?

...the release delivers dramatic out-of-the-box benefits without any coding changes or even a re-compile necessary. Simply running existing Java applications on this latest release is all that is needed.

Now my interests are peaked. So how come I haven't heard this yet. Perhaps I was just content being able to upgrade from 1.4 to 1.5.

I think this article by Rick Ross on javalobby has some great points. Here are the ones that interested me:

Java 6 incorporates over 300 bug fixes
Java 6 features general performance improvements while preserving compatibility with older versions
In many cases you may find that your applications run as fast or faster under Java 6 as they would if you spent significant effort tuning heap, thread and garbage collection parameters under Java 5.

That last point hits me right at home since we recently went through "significant effort tuning heap, thread, and gc params."

So what about performance? This article is in an depth analysis of the performance improvements. The summary is it's faster.

So like mentioned above I downloaded java 6 update 6. Jboss 4.2.1 started just fine. Today I plan on building our EAR using maven 2.0.9 and deploying it to see if there are any issues. See the official compatibility page here. Specifically it states "Java SE 6 is upwards binary-compatible with J2SE 5.0 except for the incompatibilities listed below. Except for the noted incompatibilities, class files built with version 5.0 compilers will run correctly in JDK 6."

Pimp my Linux: Maven2 Bash Completion

2008-05-01T22:00:00.003-05:00

Inspired by this bash completion for ssh, I wanted to see what it would take to do something similar for maven2. Thanks to google it's already been done.

It's not perfect, but it's a start. For example, the choices are hard coded in the script. And when I tab after typing mvn assem it outputs mvn assembly\:assembly. Shouldn't be too hard to find that extra slash.

Here are some thoughts on how one could improve it. First, maybe it could be improved for at least the core plugins by searching the local repo in /m2/repository/org/apache/maven/plugins. Or searching the local repo for any pom that contains a packing value of maven-plugin (maven-plugin). Combine that with the help plugin to get the possible goals and all their parameters and one could probably create a pretty slick and useful bash completion for maven2.

For example, you can get everything you need for any plugin if you know the groupId and artifactid by running this:
mvn help:describe -DgroupId=org.apache.maven.plugins -DartifactId=maven-war-plugin -Dfull=true

I say all this, but I use a lot of alias's for my maven commands. Here are a few of my favorites:

alias mci='mvn clean install'

alias build='mvn clean install -Dmaven.test.skip=true -Dpmd.skip=true -Dcheckstyle.skip=true'

Using Maven War Overlays to extend Hudson

2008-05-01T12:40:00.006-05:00

I just recently found out about one of the neatest features of the maven-war-plugin called WAR Overlays. Basically it provides a very simple way to merge multiple WARs together to create an Uber WAR. You simply add a WAR as a dependency in your POM verses adding a JAR, and the maven-war-plugin will take care of the rest. My team uses this ability to extend the JSPWiki WAR to add in our wiki pages. The result is an Uber WAR including the JSPWiki stuff and our wiki pages. Then when a new JSPWiki WAR is available we just update the dependency version in our POM.

So for an example, I am going to demonstrate extending my favorite CI tool Hudson since it's freaking awesome and is downloaded as a WAR. Don't try this at home since hudson already provides the ability to extend it using plugins (which also rocks by the way).

Create a Simple WAR Project
First, create a war project using maven-archetypes. Execute mvn archetype:generate and select #18. Run mvn clean install to ensure it builds correctly (I am using maven v2.0.9).

Install hudson into local repository
Next we need to be able to consume the hudson war in our pom and since I am unware of the hudson war being available on any external repository we are going to just install it manually into our local m2 repository.

Download the latest hudson war
Install hudson.war into your local repository using the mvn install plugin by running: mvn install:install-file -Dfile=hudson.war -DgroupId=hudson -DartifactId=hudson -Dversion=1.0 -Dpackaging=war

Consume the hudson.war in your pom
Open up your war's parent pom and add the hudson war as a dependency.

<dependency>
  <groupId>hudson</groupId>
  <artifactId>hudson</artifactId>
  <version>1.0</version>
  <type>war</type>
  <scope>runtime</scope>
</dependency>

Next, since we want to run hudson in embedded mode without a container we need to add to the generated manifest file the Main class since I am too lazy to figure out how to include the original hudson manifest file (even though I am sure its possible). Include the following in your build section:

<build>
<finalName>mywar</finalName>
<plugins>
<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-war-plugin</artifactId>
  <configuration>
    <archive>
      <manifestEntries>
        <Main-Class>Main</Main-Class>
      </manifestEntries>
    </archive>
  </configuration>
 </plugin>
</plugins>
</build>

Build and Run it
Now build the war again: mvn clean install. And your WAR should be merged with the hudson war. In fact for verification, your simple war contained an index.jsp under src/main/webapp and if you extract the war under target you will see your index.jsp.

Run: java -jar target/mywar.war
and go to: http://localhost:8080/index.jsp

If you are doing this on a serious level, you might want to look into the maven-cargo-plugins ability to create Uber wars.

Reply: JBI Misses the Mark by Ross Mason

2008-04-18T15:33:00.004-05:00

I would like to reply to Ross Mason's article titled JBI Misses the Mark. It was too hard to include this in a comment, so I decided to post my reply here.

Ross,
As a JBI Component Developer and user of ServiceMix and OpenESB I am going to try my best to give some answers to your points with the goal of discussion and not persuasion. I am not a "die-hard" fan trying to push JBI adoption. So hopefully I will come off as unbiased and honest as possible.

"It’s been a few years since I read the JBI specification, but here goes."

As evidence by the different ways JBI was implemented by Apache and Sun, just reading the spec alone I think is not sufficient to disregard JBI. For example, I remember when I first read the spec and looked at ServiceMix, I was totally confused. I was asking myself, where are all the WSDLs? Most of the comments I have read from JBI opponents have usually just read the spec or heard from co-workers and have actually never used or seen JBI used. Now I am not saying this person is you Ross; you have obviously done your homework. I believe the biggest failure of JBI has been communicating clearly what JBI is and how to use it. It took me many months until I felt like I actually "understood" JBI. And before that moment happened I thought I knew what JBI was only to find out something new and that I never actually did.

"Adaptive Integration"

Unless I misunderstood it, your definition of Adaptive Integration fits well with my experience with JBI. Using JBI I can build "best-of-breed" integration solutions using any number of the JBI components available with ServiceMix (SM) or OpenESB (OESB). If I need to integrate with JDBC I can do that; or RSS, XMPP, SIP, Corba, and CICS. Perhaps my lack of knowledge of Mule capabilities is making me naive here, but to me I can use JBI to bridge multiple systems together.

"JBI attempts to standardize the container that hosts services, the binding between services, the interactions between services, and how services and bindings are loaded. The problem occurs when the APIs around all these things makes assumptions about how data is passed around, how a service will expose its interface contract and how people will write services."

In a nutshell the only thing I think JBI standardizes on is the interactions between components or put another way the format of a message that goes over the NMR. Beyond that its up to the component developers to do what they wish. For example, with the RSS BC, there are no assumptions that need to be made about incoming requests from the NMR; it's a standard JBI message. Doesn't matter if it came from a SIP, HTTP, Corba. To me that is the beauty of JBI. Once I create my JBI layer in my component anybody in the container can call me.

"Xml messages will be used for moving data around"

As stated above, JBI doesn't say if your legacy system doesn't know XML, then you can't use JBI. The exact opposite is true. To me this is the perfect fit for JBI and Binding Components. My definition of a Binding Component is a translator. It's sole job is to translate protocol X to JBI and translate JBI to protocol X. Perhaps what you are suggesting is that some message types don't "convert" easily to XML since that is what essentially messages over the NMR are. So in your example, a Cobol CopyBook message can't be or shouldn't be attempted to be converted to XML. Not sure I would agree since I would think, given a schema, anything could be represented in predictable XML.

"Data Transformation is always XML-based"

Since everything is converted to XML to be passed over the NMR, the only transformation needed for that is XML. So what you is true, but to a JBI user I guess it doesn't matter. I guess on the other side, if the NMR allowed for other message types, then yes I guess you would need more transformers, but I guess with JBI the transformers are the Binding Components.

"Service contracts will be WSDL"

This is a very true statement and I am not exactly a WSDL/XML lover; unless I don't have to deal with it. I think the original intent was there would be a lot of tooling support for JBI to alleviate that burden from the JBI user. To me Sun has done a better job of this than SM, but this is definitely a drawback to JBI. However SM has all but eliminated WSDL's in JBI so users of SM don't need tooling support since they don't actually require WSDLs.

"No need for message streaming"

Another very true statement of there not being an "easy way to stream messages in JBI". With XML being so verbose I believe SM or OESB would probably dislike a 3 MB XML file getting pumped through it. Especially if that gets copied over and over again from one component to the next. This is where JBI might have difficulties scaling. As a colleague pointed out, the content of a JBI message is XML. And that XML has to be marshalled and unmarshalled. And that process is very heavy and even more so if there are reliability/durability constraints on the Message.

However, I think its a current tradeoff in order for all components on the NMR to communicate in a predictable fashion. Since that is the main concept of JBI: the ability for all the JBI components communicate. They have to communicate some how. One component can't decide that he expects format A and another decides he requires format B. Then nothing would work. So it sounds like to me XML was that language to guarantee interoperability between components.

On a side note, JBI components have the ability to set the content of a JBI message that goes on the NMR as anything implementing the interface javax.xml.transform.Source. Meaning that they can use streaming XML with the SAX and Streaming implementations. However, once consumed, that stream is gone and is probably represented as DOM (ouch) somewhere in the process. Our components use the Streaming impl when creating JBI Messages, but that doesn't mean the others do.

Either way, I have to think that smart people like Peter Walker, yourself, and other spec leaders can't come up with something to resolve this and get the best of both worlds.

"You need to implement a pretty heavy API to implement a service"

I disagree that JBI users have to know more about JBI than is necessary, but then again its hard for me to play that person since I am also a component developer. But most of the people that I have taught how to use JBI have been successful in using it without having to know as much as me. I know in SM you can expose a POJO pretty easily. In OESB you can use the Scripting Engine or create an EJB Service to be used in JBI. From what it sounds like though, its much easier with Mule.

Again I have not had a hard time teaching non JBI users about JBI. However I skip the spec and jump straight to a demonstration of how you can use JBI. This seems to help. I remember the first video I watched when the light bulb finally went off. It was a video on how to use the Sun SMTP BC and creating a Service Assembly with Netbeans. After I watched that video everything just came together for me and my co-workers. This is basically how I start off explaining JBI: It's best explained with an example rather than a bulleted powerpoint.

"It’s not actually that clear what a service engine is in JBI."

Touche. I hear this a lot, even from me. Again, as I stated earlier, the JBI community has done a poor job of training and communicating to its users and developers. My first definition of a Service Engine was basically a Service. I still don't have a good definition of an SE; usually I will just talk about what existing SEs are out there like the BPEL and XSLT SE. To me Binding Components are easy to define so an SE is just everything else (granted not exactly the best definition).

"The JBI specification is obfuscated with lots of new jargon"

Yes, yes, and yes. Again poor job of communicating. However, the JBI user should not be concerned with most of this jargon; only the JBI developer. For example, the JBI user shouldn't need to know about what the NMR is or what it does; but it's important for JBI developers to know.

"This breaks the re-use story since, if I use a JBI Binding Component in one container doesn’t mean it will behave the same way in another container."

Ideally JBI was suppose to be this loving community of predictable containers and components. Unfortunately that hasn't exactly happened. I will say this though, from my personal experience of using different components on different containers, for the most part most components and Service Assemblies work in a predictable way on other containers. For example, I have successfully used many SM components in OESB and and I have heard of Sun components working in SM. However the failure is you can't intermix Sun and SM components (or at least from what I know). For example, you can't use the SM HTTP BC with the Sun BPEL SE. To me this wasn't a failure of JBI; only how the JBI container was implemented. In a perfect world this would have happened but it hasn't.

Ironically it actually doesn't take that much to get your components to work on both SM and OESB. Of the four components we have written (RSS, SIP, XMPP, and UDDI), XMPP has been modified to work on both. We haven't had a chance to ensure the other 3 can, but hopefully we can get to that or the open source community could help. At the time of development we didn't actually understand JBI that well and consequently when we had our components originally working in SM and tried to port them over to OESB, they worked but only with other SM components. In order for our components to work with Sun components we had to basically rewrite the JBI layer. Unfortunately, due to time constraints it was not our current goal to keep backwards compatibility with SM. It wasn't until after we finished all this that we actually understood JBI and what it meant to create a portable component.

"If you look at every JBI implementation each has written their own JMS, FILE, HTTP, FTP etc Binding components… not exactly what I’d call re-use."

You are exactly right. Each container has implemented their own components for File, JMS, HTTP. Most likely because of the issues I mentioned above. However I don't think this doesn't mean JBI can get there. If these projects wanted to make them portable than we wouldn't have this problem. Instead they focus their efforts on other things.

As a side note, since my group was using OESB, when I joined a new group, we needed to get OESB working on JBoss. This currently was not supported and my previous group was able to work with Sun to get this working and its currently being incorporated in OESB.

Summary
IMHO I think a lot of assumptions where made about JBI, but you also addressed some of the issues with JBI. One reason I think JBI has not been adopted is the fact that it didn't have the backing early on of IBM and Bea. From what I recall they dropped out pretty early once they realized it was going to be a competing product. Another reason for a lack of adoption is I think again the lack of communicating what JBI really is.

I hope I didn't reiterate responses you have heard in the past. I wasn't around when JBI was originally approved, so I don't know a lot of the history. These are just my opinions I have collected through developing components and using SM and OESB.

I hope it helps and I look forward to your response.

Sharing your folders in Ubuntu

2008-04-16T09:23:00.002-05:00

Today I needed the ability to modify files on my Linux host machine from my Windows VM instance running in VMWare Server. This was actually pretty easy and once I found this article I was able to get it working in about 5 minutes.

Enjoy

Using Grails and Glassfishv3

2008-04-15T23:28:00.002-05:00

In case you missed it, an important feature is being worked on in order to support direct deployment of a Grails application in Glassfish v3 (see Getting Started with Grails on Glassfish). According to Mr. Gupta, they are working towards being able to deploy a Grails app to GFv3 using the grails commands (currently Grails uses Jetty). Also they are working on deploying a Grails App for production purposes without having to create a WAR; which is the current method. Both of which will definitely improve developer productivity when using Grails and GFv3.

And since it was recently announced that GFv3 can run on OSGi, I think GFv3 continues to look very interesting.

Update: Always specify a version for Maven2 plugins

2008-04-10T16:45:00.002-05:00

I previously wrote about specifying a release version for maven plugins in your POM files to have a more stable build and stop depending on SNAPSHOT plugin dependencies. It now seems with the latest version of maven2, version 2.0.9, that maven now comes with release versions for all the commons plugins (see the release notes).

"MNG-3395 - Starting in 2.0.9, we have provided defaults in the super pom for the plugins bound by default to the lifecycle and a few other often used plugins. This will introduce a bit of stability to your builds because core plugins will not change magically on you when they are released. We still recommend taking control of your plugin versions via pluginManagement declarations as this is the most robust way to future proof your builds. Defaulting the plugins in the superpom was a step towards introducing stability for small builds and new users. A full table of the versions used is shown in the next section."

Note how they still recommend specifying release versions in the pluginManagement section of your POM.

Update: Using the JBI JavaEE Service Engine

2008-04-10T15:17:00.008-05:00

Previously I wrote about how to use the JBI JavaEE Service Engine to increase performance when orchestrating EJB Services created in Glassfish with the BPEL Service Engine in OpenESB. That was almost 8 months ago and Netbeans and OpenESB have changed a lot and consequently I needed to update it.

So I would like to provide an update using the latest Netbeans (20080402).

Again, if you haven't already, go ahead and download and install OpenESB which includes Netbeans v6 and Glassfish v2.

Create an EJB Module

File > New Project > Category Enterprise > Projects EJB Module

Create a new Web Service

Right click on your EJB Module project
Choose New > Web Service
Add a new Operation

Create a new BPEL Module

File > New Project > Category SOA > Projects BPEL Module

Create a new BPEL Process

Right click on your BPEL Module project

Choose New > BPEL Process

Create a new SOAP WSDL

Right click on your BPEL Module project
Choose New > WSDL Document
Go through the wizard and for the Binding Type make sure its SOAP

Configure your BPEL Process by dragging the SOAP WSDL onto the left side of the BPEL Process.

Add the standard "buttons" to the BPEL Process Sequence

Receive
Assign
Invoke
Assign
Reply

Utilize the EJB Web Service in the BPEL Process (this is where things are different than in past versions)

Right click on the Web Service under the EJB Module and select Generate and Copy WSDL

Uncheck Do no copy if it is checked.

Expand your BPEL Module and copy the WSDL into the BPEL Module's src directory.

Now you should have a copy of the Web Service's WSDL in your BPEL Module.

Drag this new WSDL onto the right side of the BPEL Process so that we can tie the Invoke call to it.

Create a Composite Application

File > New Project > Category SOA > Projects Composite Application
Right click on the new Composite Application and select Add JBI Module
Add both your BPEL Module and EJB Module
Clean and build the Composite Application (Right click > Clean and Build)
View your CASA file by double clicking on the Service Assembly file under your Composite Application.

Your CASA file should look something like this
Now you should be able to start Glassfish and deploy your Composite Application. Now your BPEL Process will utilize the Java EE Service Engine. FYI, do not deploy the EJB Module separately since its included in the Composite Application.

Another Netbeans 6 tip

2008-04-09T00:12:00.004-05:00

I previously wrote about not installing multiple versions of Netbeans 6 and now I would like to include another tip.

When installing a new version of Netbeans 6, while uninstalling your previous version make sure you remove the .netbeans directories. Otherwise the support for creating Tests in Composite Applications doesn't work.

In fact just delete the following before each new install and you will have better luck:

rm $USER_HOME/.asadminpass
rm $USER_HOME/.asadmintruststore
rm -rf $USER_HOME/.openesbinstaller_[machine_name]
rm -rf $NETBEANS_HOME/.netbeans
rm -rf $NETBEANS_HOME/.netbeans-derby

Note this is on Linux using the most recent version 20080402. Not sure where the directories are placed or called on windows. Also I believe older versions of Netbeans 6 placed the .netbeans* directories under $USER_HOME, so you might check there as well.

IntelliJ Idea able to open Maven2 POMs directly

2008-04-08T13:34:00.003-05:00

Not sure all my Idea co-workers are aware, but if they are its worth repeating. Now in Idea 7 (I am specifically using 7.0.3) you can quickly open your project in Idea by pointing it to your projects maven2 POM. It works pretty slick and now that saves me from having to run mvn idea:idea. It also supports the ability to synch with the maven structure on startup in case you add new dependencies to your POM, you shouldn't have to perform any magic to get Idea to recognize them.

Jeff Black reports the same ability with Netbeans 6. And here is the original Idea post describing this new feature.

Using WebEx to demo from your Linux machine

2008-04-08T13:02:00.002-05:00

In my unending quest to figure out how to efficiently do a demonstration from my Linux machine, I finally was fortunate enough to come across a solution. Today I did a demonstration for a friends company and they use WebEx. Luckily enough for me I reluctantly tried it first on my Linux machine, but not expecting it to work, and it worked! I was able to do a demo from my Linux machine and it actually worked seamlessly.

So if your company consists of different types of operating systems (windows and linux) and you need software that works on both for employees to share and view demonstrations then take a look at WebEx. It is not free so that is a bummer, but neither is Goto Meeting (which is what we currently use).

New simple Linux tip for terminal: CRTL+L

2008-04-04T12:35:00.001-05:00

I just found this out (basically because I saw a coworker clear his terminal window with a keyboard shortcut on his Mac, as opposed to running the clear command).

If you want to clear your terminal screen hit: CTRL+L.

Ubuntu Remote Desktop

2008-04-04T09:21:00.002-05:00

As an update to my premature post yesterday, I was able to finally use Ubuntu's Remote Desktop ability to share my desktop to a windows machine in order to perform a demonstration. VNC Server did not work and neither did No Machine (for me that is).

Luckily someone mentioned Ubuntu's Remote Desktop and I was able to VNC from windows using RealVNC's VNC Viewer and presto. Now I will just join a session in Yugma, share out my windows desktop, which is displaying a VNC session into my linux laptop. Wew!

I can't wait for the day when I can get a Mac or Yugma can share a linux desktop.

Using Yugma to demo from your Linux machine

2008-04-03T15:04:00.005-05:00

Currently our company pays to use Goto Meeting to coordinate meetings. Unfortunately, Goto Meeting does not work in Linux and consequently I usually connect to meetings from my Windows VMWare instance.

Today I had the need to do a demonstration with offsite coworkers and I needed to do it within Linux. Since I have used the free service called Yugma in the past and it supposedly works in Linux I decided to spend some time trying to get it to work.

Well see for yourself.
It took some time, but with their free online chat Help Desk (which unexpectedly was amazing) we eventually got it to work.

I image others wouldn't have as many issues as myself, but just in case here are some of the steps I had to perform to get this to work.

First, if you haven't already created an account, go ahead and create one (http://www.yugma.com). Then try visiting the Start a Session page. If you haven't already installed the software it should provide a link to download yugma. If not then you have issues like I did. Basically I had an old java version that FireFox was using and I had to update it to JRE 6. I followed their instructions here, but that wasn't enough.

With the help of the Yugma Help Desk I still had to do the following:

sudo apt-get install sun-java6-jre sun-java6-plugin sun-java6-bin
update-alternatives --list java
this should print out
/usr/bin/gij-4.1
/usr/bin/gij-4.2
/usr/bin/cacao
/usr/lib/jvm/java-6-sun/jre/bin/java
sudo update-alternatives --config java
[sudo] password for jlorenzen:

There are 4 alternatives which provide `java'.

Selection Alternative
-----------------------------------------------
1 /usr/bin/gij-4.1
2 /usr/bin/gij-4.2
3 /usr/bin/cacao
*+ 4 /usr/lib/jvm/java-6-sun/jre/bin/java

Press enter to keep the default[*], or type selection number: 4
Using `/usr/lib/jvm/java-6-sun/jre/bin/java' to provide `java'.
Then restart FireFox. Revist the Start a Session page and you should be golden.

As a developer I really like Yugma because it has a free version that allows you to invite at least 10 people. Which is perfect for developers. And since our Goto Meeting licenses are controlled its difficult to get a Goto Meeting scheduled on short notice (Although Josh Hoover is the King of Goto Meetings). The other nice thing about Yugma is its easy to start and share a session. I remember in Windows Outlook you could even install a package that put buttons in Outlook and set it all up; very handy.

Automated Performance Tests using JMeter and Maven

2008-03-21T00:32:00.008-05:00

Learning how to write and automate performance tests isn't easy. Perhaps that is why I have never actually wrote and automated performance tests; until now. For my latest open source side project, sass4j, I wanted to start the project with some automated performance tests. Since we were already using maven2, I wanted to find a maven2 plugin that allowed me to write complex performance tests and automate them easily in a continuous integration environment such as Hudson. Since I had heard good things about JMeter and there was an existing JMeter plugin, I decided on those technologies. Unfortunately the path this took me down was rather long and annoying, but I eventually figured everything out and that is the reason I would like to document the steps for others to reproduce in less time.

But first, why would anyone want to spend unnecessary time setting this up? The big advantage I think, is having the ability to compare nightly test results with a baseline and compare them with expectations. If my latest changes caused a major decrease in performance when I only added a few lines of code, then perhaps something is wrong. The second advantage is the sooner a performance issue is found the cheaper it costs to fix it. Think about the change set a developer has to look at if nightly performance tests were ran, compared to finding a performance issue 6 months from when the bug was introduced. With the former scenario, I only have to look at what has changed in the last 24 hours.

Now onto how to actually do this. I had two references in getting this done: The Official Apache JMeter Maven Plugin Site and a blog on AMIS by Robbrecht van Amerongen. Both of which are incomplete, but combined provide enough information.

Here is an outline of what you need to do. End to end this should take you about 15 minutes to setup; compared to the hours I spent I think you are getting a deal. Also you need to think about doing this in your artifactory server or company maven repository and not locally (doing it locally only helps you and not your entire company).

1) Download the JMeter maven bundle I created containing all the necessary artifacts
2) Install JMeter Plugin dependencies
3) Install the JMeter Plugin
4) Install JMeter
5) Update your maven project
6) Create jmx files and run mvn verify

The first 3 steps are necessary because there are specific dependencies the JMeter plugin requires that are not available on any public maven repo I can find, nor is the JMeter plugin.

1) Download the JMeter plugin bundle

Click here to download the JMeter plugin zip file
Unzip it

2) Install JMeter Plugin dependencies

cd to the extracted folder jmeter
run the following mvn commands to deploy the jar files locally. Obviously update the location of your local maven2 repository and keep in mind the file paths are specific to linux. Again do this once in your company's maven repository.

mvn deploy:deploy-file -DgroupId=org.apache.jmeter -DartifactId=jmeter -Dversion=2.2 -Dpackaging=jar -Dfile=jmeter-2.2.jar -DpomFile=jmeter-2.2.pom -Durl=file:///home/jlorenzen/.m2/repository/
mvn deploy:deploy-file -DgroupId=jcharts -DartifactId=jcharts -Dversion=0.7.5 -Dpackaging=jar -Dfile=jcharts-0.7.5.jar -Durl=file:///home/jlorenzen/.m2/repository/
mvn deploy:deploy-file -DgroupId=org.apache.jorphan -DartifactId=jorphan -Dversion=2.2 -Dpackaging=jar -Dfile=jorphan-2.2.jar -Durl=file:///home/jlorenzen/.m2/repository/
mvn deploy:deploy-file -DgroupId=org.mozilla.javascript -DartifactId=javascript -Dversion=1.0 -Dpackaging=jar -Dfile=javascript-1.0.jar -Durl=file:///home/jlorenzen/.m2/repository/

3) Install the JMeter Plugin

Unzip the maven-jmeter-plugin.zip file that was included in the JMeter plugin bundle.
cd to the maven-jmeter-plugin folder
run: mvn install
This will install version 1.0 of the maven-jmeter-plugin. It's important we install a release verses a snapshot because you don't want your project to depend on snapshot plugins because this could have nasty side effects for building and releasing.

4) Install JMeter
Now that you have the plugin installed you can actually start modifying your projects pom to use it. You first need to install JMeter because we will need a jmeter.properties file, some XSL files, and you are going to need it to create the .jmx files.

Download and install JMeter

5) Update your maven project

Under your project create the directory: src/test/jmeter and src/test/resources
Copy the jmeter.properties file from the JMeter bin folder to src/test/jmeter.
Update the property jmeter.save.saveservice.output_format in the jmeter.properties file from csv to xml.
Copy the files jmeter-results-detail-report_21.xsl and jmeter-results-report_21.xsl from the JMeter extras folder to src/test/resources
Add the following to your POMs build/plugins section

<build>
   <plugins>
       <plugin>
           <groupId>org.apache.jmeter</groupId>
           <artifactId>maven-jmeter-plugin</artifactId>
           <version>1.0</version>
           <executions>
               <execution>
                   <id>jmeter-tests</id>
                   <phase>verify</phase>
                   <goals>
                       <goal>jmeter</goal>
                   </goals>
                   <configuration>
                       <reportDir>${project.build.directory}/jmeter-reports</reportDir>
                   </configuration>
               </execution>
           </executions>
       </plugin>
       <plugin>
           <groupId>org.codehaus.mojo</groupId>
           <artifactId>xml-maven-plugin</artifactId>
           <version>1.0-beta-2</version>
           <executions>
               <execution>
               <phase>pre-site</phase>
               <goals>
                   <goal>transform</goal>
               </goals>
               </execution>
           </executions>
           <configuration>
               <transformationSets>
                   <transformationSet>
                       <dir>${project.build.directory}/jmeter-reports</dir>
                       <stylesheet>src/test/resources/jmeter-results-detail-report_21.xsl</stylesheet>
                       <outputDir>${project.build.directory}/site/jmeter-results</outputDir>
                       <fileMappers>
                           <fileMapper implementation="org.codehaus.plexus.components.io.filemappers.FileExtensionMapper">
                               <targetExtension>html</targetExtension>
                           </fileMapper>
                       </fileMappers>
                   </transformationSet>
               </transformationSets>
           </configuration>
       </plugin>
   </plugins>
</build>

6) Create jmx files and run mvn verify

Now use JMeter to create your .jmx files and place them under the src/test/jmeter directory.
run: mvn verify to execute your performance tests
run: mvn verify pre-site to execute your performance tests and produce the test results in an HTML report.

If you want to see a real example of this click here. Now the only step I am leaving out is actually automating it which isn't the hard part luckily. All you need to do in Hudson is create a new job and execute the mvn goals mvn verify to get them automated in a CI environment.

Always specify a version for Maven2 plugins

2008-03-10T14:53:00.010-05:00

For the past several months my team has been suffering a lot of downtime because our local builds fail and our Continuous Integration environment fails for what appears to be no reason. What makes these issues difficult to debug is what might fail on your machine, won't fail on mine ("works on my machine"). We eventually found the cause and that was we were not specifying a version for plugins in our maven2 poms. Pretty dumb, eh?

Wrong way

<plugin>
 <groupid>org.apache.maven.plugins</groupid>
 <artifactid>maven-surefire-plugin</artifactid>
</plugin>

Correct way

<plugin>
 <groupid>org.apache.maven.plugins</groupid>
 <artifactid>maven-surefire-plugin</artifactid>
 <version>2.3</version>
</plugin>

This matters because if you leave out the version, maven2 defaults to SNAPSHOT. So what was happening is my project unknowingly became automatic beta testers for third party plugins.

So the moral of the story is always include a version (preferably a stable release version like 2.3 and not 2.3-SNAPSHOT) when defining your plugins. This will go a long ways in making your build more stable. And better yet, define those plugins in your parent pom using the <pluginManagement> section. This keeps your versions in one place so that when you do upgrade intentionally to a newer version you only have to modify it in one place for your entire project. For example:

<pluginmanagement>
  <plugins>
      <plugin>
            <groupid>org.apache.maven.plugins</groupid>
            <artifactid>maven-surefire-plugin</artifactid>
            <version>2.3</version>
        </plugin>
  <plugins>
</plugins>

Specifically the maven-surefire-plugin and maven-war-plugin have cost us a lot already, so recently I updated all our poms to define a version for all the plugins we were using. And one reason I know it would fail for some, but not others, is some prefer to build in offline mode (-o) always which prevented them from seeing the problems.

Avoid installing 2 versions of Netbeans 6

2008-03-10T00:11:00.003-05:00

Tonight I installed the latest OpenESB software (Build 20080303) and was rather confused as a lot has changed since I last used it back in October 2007. I found myself being rather unproductive so I decided to revert to an earlier version. So I downloaded Build 20080214 and installed it, but I did not uninstall the previous version. With the older version I repeatedly received exceptions on just about everything I clicked on. Many of my co-workers battle these issues everyday, but now I might have figured out why. For some reason, Netbeans just doesn't like being installed twice on the same system under the same folder (for me was /workspace/java/openesb). Under there I had 2 netbeans and 2 glassfish folders. Once I uninstalled everything and reinstalled the older version (Build 20080214) it worked perfectly.

So when you are receiving all those nasty exceptions every 5 seconds in Netbeans, make sure you only have one installed.

Challenges using Icefaces

2008-03-06T15:25:00.005-06:00

I would just like to take a quick moment and inform anyone about Icefaces, and the challenges I have faced using it. Take it from my current experience (version 1.6.1 and 1.6.2) and the words of an Icefaces developer that since Icefaces "is a stateful technology......providing enhanced features for the user....but require server-side resources to do so." That last statement has been our recent struggle.

Unfortunately, on my current project we don't have the luxury of lots of server-side resources. It's disappointing to say we have to share a single server that is already running IIS; then we have to run Jboss and MS SQL on the same machine. Obviously this is not by choice and doesn't look to change in the near future.

It might be Icefaces never intended their framework to run in such a limited environment. But that really doesn't help me now. It would be great if they published information about minimum hardware requirements, especially if they are aware of performance issues. Equally important I am sure we have strayed away from best practices in using Icefaces.

Please don't ask me why we choose to use Icefaces in the first place (I wasn't with the group at the time). But I assume it was a lack of knowledge of how Icefaces works, and we weren't aware of the environment limitations. Either way, we are facing difficult challenges that are very frustrating.

For example, we recently had severe problems with an editable table which was sortable (much like an excel spreadsheet). When users clicked twice on the header to sort, the data appeared to be sent twice and some how our data got corrupted. Consequently we had to disable the ability to sort.

Also, based on my understanding of Icefaces, it appears that all clients maintain a constant connection with the server. I am assuming this is what the Icefaces developer above was referring to when he says Icefaces is a stateful technology. For some reason, I feel this constant connection would cause challenges for 50-100+ concurrent clients with limited server-side resources.

Hope it helps someone.

How to simulate onblur event for a Panel in Extjs

2008-03-04T08:43:00.006-06:00

Here is a simple trick in Extjs I borrowed from Ext.layout.BorderLayout that simulates listening for the onblur event of an Ext.Panel. This trick is necessary because unfortunately Ext.Panel does not natively support registering for the onblur event (or at least I don't know how). We typically use this trick when we show a Panel and want to remove it when the user clicks off, or to state it differently, when the Panel looses focus. This is very similar to how menus work.

Step 1: Register for mousedown event

this.panel.show();
Ext.getDoc().on("mousedown", this.handleDocMouseDown, this);

First, we show the panel. Here it is assumed this was created previously. Second, we use Ext.getDoc() to return the current HTML document object, and then we register for the mousedown event. The second parameter, this.handleDocMouseDown, is our function we want called when this event is fired. Notice that we register for the mousedown event as opposed to the click event. This is intentional because listening for the click event will actually call this.handleDocMouseDown before your ready because a click involves both mousedown and mouseup.

Step 2: Handle the event

handleDocMouseDown : function(e) {
if (!e.within(this.panel.getEl())) {
   this.panel.destroy();
   Ext.getDoc().un("mousedown", this.handleDocMouseDown, this);
}
}

In our second step we first check to make sure the user didn't click anywhere in the Panel that was just displayed. If it wasn't then we destroy the panel and unregister the mousedown event.

Groovy's MarkupBuilder and Namespaces

2008-02-28T08:42:00.014-06:00

Yesterday Kit Plummer posted an XML example using Ruby to demonstrate how to include attributes and namespaces. So I went on a quest to duplicate his output using Groovy to compare the difference and evangelize.

Here is the desired output:

<person:person text='test'>
   <name>Jim</name>
   <phone>555-1234</phone>
</person:person>

Using the Groovy Console, and Groovy's ninja-like MarkupBuilder, you can execute the following script to get attributes and namespace prefixes:

import groovy.xml.MarkupBuilder

def writer = new StringWriter()
def xml = new MarkupBuilder(writer)

xml.'person:person'(text: 'test') {
   name('Jim')
   phone('555-1234')
}

println writer.toString()

If you prefer to define a default namespace instead do this:

xml.person(xmlns: 'http://people.org', text: 'test') {
   name('Jim')
   phone('555-1234')
}

Finally, here is an example that uses them all:

xml.'person:person'('xmlns:person': 'http://people.org', xmlns: 'http://people.org', text: 'test') {
   name('Jim')
   phone('555-1234')
}

This will output:

<person:person xmlns:person='http://people.org' xmlns='http://people.org' text='test'>
   <name>Jim</name>
   <phone>555-1234</phone>
<person:person>

I think it's more readable than the Ruby example and definitely much better than the equivalent Java code.

By the way, I attempted to use this online syntax highlighter and did not like it. Does anyone know of a good online syntax highlighter?

Volunteering as Hudson OpenSolaris IPS maintainer

2008-02-08T13:02:00.000-06:00

Starting with an open request by Hudson project lead Kohsuke Kawaguchi, I volunteered to be an OpenSolaris IPS package maintainer for Hudson. Not sure I am qualified but I am a huge Hudson advocate and am looking forward to the challenge.

My main goals are to:

Learn more about how open source communities work
Go in the opposite direction of my comfort zone and learn something new

Stay tuned on this Hudson wiki page I have created to document this effort.

Grails 1.x to add Maven 2 plugin support

2008-02-07T09:36:00.001-06:00

From an InfoQ interview with Grails project lead Graeme Rocher discussing the recent release of Grails 1.0 and its future:

In addition, we plan to include Maven 2 support out of the box with Grails through a Maven 2 plug-in that hooks Grails into the Maven life cycle. Remember Grails is not just a web framework, but a software stack which aims to solve every part of the software life cycle from the build system to the ORM layer.

For those of use already using maven2 this is pretty good news. Not that there weren't ways to get around it, but it will be nice to have maven2 support out of the box.

Maven Groovy Plugin Example

2008-01-24T07:59:00.000-06:00

Here is a powerful example by a co-worker on how to use the maven-groovy-plugin in your maven2 POMs. In this example Ron shows how you can embed a Groovy script in your POM to enforce a specific size of the resulting artifact (in this case an EAR).

This is very valuable because now you can do just about anything in your POM, including adding Java code as in this example (look under Using Java Classes).

However, with this power comes responsibility and you don't want to get in the habit of using this everywhere. Usually this is a red flag meaning you should create your own plugin to be reused by everyone. We should all be able to recognize the drawbacks of Copy+Paste.

How to connect to a Grails HSQL Database

2008-01-03T23:00:00.002-06:00

The following is how you can use DBVisualizer to connect to a Grails HSQL Database. Grails uses Spring and Hibernate and its default database uses HSQLDB. So why would you want to view the Grails database? Maybe you are curious of how GORM works; debug your Domains; perhaps you don't trust Grails, or finally you might be a control freak. Or like me you are all of the above.

Update Datasource.groovy
First, if you want to view the development database you need to change the url to not use an in-memory database because DBVisualizer will not have access to the Grails JVM memory. Open up the DataSource.groovy file and locate the development environment section and replace the existing url "jdbc:hsqldb:mem:devDB" with "jdbc:hsqldb:file:devDB;shutdown=true".

Start Grails
>grails run-app

Create New Database Connection in DBVisualizer
Open up DBVisualizer (my version is 6.0.7) and create a new Connection (if you are unable to use DBVisualizer you can also use the HSQL Database Manager - see this post).

Type in a Connection Name: Grails Development
Choose HSQLDB as the DatabaseType
Choose HSQLDB embedded as your Driver using the hsqldb.jar located under your Grails home directory ($GRAILS_HOME/lib)
Update the Database URL to include your grails app base directory (jdbc:hsqldb:file:{your-grails-app-base-dir}/devDB). For me this was jdbc:hsqldb:file:/workspace/checkout/netcds-admin/devDB. Note that according to the HSQL document, Window users don't have to specify the C: drive.
Make sure your Userid is sa
Click Connect

View Tables
Then on the left you should be able to see your tables under PUBLIC --> TABLE.

Special thanks to the Nabble users who responded to my question: Christian Laakmann and Helmut Denk.

Justify using Groovy

2007-12-28T21:50:00.000-06:00

How does one justify using Groovy on a project? To me it's no different than including any other dependency in my project and in the end it has the same result but with a lot more benefits. There are a lot of skeptics out there, specifically among my team members, and I just can't imagine why they wouldn't want to use Groovy in their every day development. I guess in this instance "knowledge is a curse." It's kind of difficult to explain why I believe in what Groovy is doing, but I will at least give it a try.

First, the biggest initial hurdle people can't get over is they think to start writing Groovy they have to stop writing Java. That couldn't be further from the truth. Groovy was never meant to replace Java. I have used Groovy to write unit tests for my Java code and I have written Groovy code that has used Java code and vic versa. My experience has shown that Groovy and Java just work seamlessly together, especially when using the maven-groovy-plugin and the IntelliJ Idea JetGroovy plugin (for more information see More Groovy Goodies by Joe Kueser).

Secondly, to me the biggest benefit of Groovy, with no cost to the developer, is the hundreds of extra methods added to existing Java classes (see the Groovy JDK). Just by including a new dependency, I have at my disposal hundreds of new time saving methods that I can start using immediately. For example, there are approximately 50 new methods added to java.lang.String (not sure I will ever use all of them but you get the picture). Now lets say for example you were using some open source project like JFreeChart. And you were using version 1.0.8, and 2.0 was out and it included hundreds of new methods. How long would it take you to change the version in your maven2 pom? To me this is not much different than using Groovy to "extend" Java.

Finally, I want to give a quick example of what the end result of a Groovy class is since Groovy is just complied into Java bytecode. Since I have repeatedly stated that Groovy is no different than including a new dependency in your project let me prove it.

Borrowing an example from my previous post, lets say my Groovy class looks something like this:

class GroovyFileTest {
    def void test() {
        def file = "/workspace/sandbox/grvyfile/src/test.txt"
        println new File(file).text()
    }
}

My next step would be to use the Groovy Complier (groovyc) to compile my Groovy source into Java bytecode (or just use the maven-groovy-plugin). That would produce GroovyFileTest.class. Using a Java decompiler, such as Jad, here is the outcome:

public void test()
{
    Class class1 = GroovyFileTest.class;
    Class class2 = groovy.lang.MetaClass.class;
    Object file = "/workspace/sandbox/grvyfile/src/test.txt";
    ScriptBytecodeAdapter.invokeMethodOnCurrentN(class1,
        (GroovyObject)ScriptBytecodeAdapter.castToType(this, groovy.lang.GroovyObject.class),
        "println", new Object[] {
            ScriptBytecodeAdapter.invokeMethod0(class1,
                ScriptBytecodeAdapter.invokeNewN(class1, 
                java.io.File.class,
                ((Object) (new Object[] {file}))), "text")});
}

For the sake of clarity I am not including all the other information that was decompiled (well.... because it's rather long). But to summarize, my two line Groovy script is convereted to 214 lines of decompiled Java code. Initially you might think that boosts well for Groovy and really shows the benefits of Groovy, but there is a lot of Groovy going on behind the scenes.

From my perspective this is the only current argument a Groovy skeptic can make: "well look at all that extra junk they are throwing in. Isn't that going to hurt performance?". And at the end it just might. I think it's obvious that Groovy bytecode probably won't run as fast as Java bytecode, but for me the benefits out way that disadvantage.

I by no means intended this to be an exhaustive list of Groovy benefits (for that I would recommend reading the excellent but lengthy article by Guillaume Laforge on the recent Groovy 1.5 features) . I just wanted to prove that the end result is no different. Hopefully people will respond with questions and criticisms and I can respond intelligently.

To me Groovy is sort of the legal steroids of programming. Those that use it have an unfair advantage, enabling them to deliver software quicker then their competitors and co-workers.

A bit of history. Americans original motivation

2007-12-25T22:42:00.000-06:00

I absolutely love history. In fact outside of programming and basketball it has to be my favorite hobby (and yes programming to me is a hobby since I enjoy it so much). For Christmas my wife gave me the Illustrated Edition of "1776" by David McCullough. I enjoyed the original very much and am looking forward to reading it again along with the 140 images and 37 replicated source documents.

Learning new things I think is why I enjoy history so much. For example, in his new book, Author McCullough starts by including an image of what it regarded as our countries first flag: The Grand Union. Also fascinating was the quote by Alison Huber, "Independence from Britain was not as yet what Americans were fighting for, but rather what they felt their rights as freeborn Englishmen." It's real easy to forget something like this and just naturally think our fore fathers started fighting for independence immediately. Instead they were just fighting for the same rights their brothers had back in England.

Stayed tuned as I plan on posting as I read more of the book; or just hit the space bar in Google Reader if you don't care.

Google Readers Discover

2007-12-05T22:15:00.001-06:00

As if I didn't have enough RSS feeds to read, Google Reader went ahead and recently added a neat little feature called Discover (or Top Recommendations). This new feature provides a short list of new feeds generated by comparing my interests with the feeds of users similar to me. In fact, according to the Help Center, it takes into account my current subscriptions as well as information from my Web History including location. It's actually pretty fascinating and I discovered several interesting feeds that I then subscribed to easily.

If you are already using Google Reader take a look at your list. If not, then consider using or switching to Google Reader; it's discovertastic!

Using Groovy to work with Files

2007-11-29T21:49:00.000-06:00

One of the best applications of Groovy I have come across so far is working with Files. Using Groovy to read data from a file can not get any easier. If you are a Groovy skeptic this is the post for you. Using the maven-groovy-plugin, which allows you to mix Java and Groovy code seamlessly, you no longer have any more excuses (go here for a maven-groovy-plugin how-to).

Recently I was writing unit tests in Groovy for my Java code and I needed to read in the contents of a file for comparison. In Java this would look something like this:

import java.io.BufferedReader;
import java.io.FileReader;
import java.io.FileNotFoundException;
import java.io.IOException;

public class FileTest {
public static void main(String[] args) {
 try {
     String file = "/workspace/sandbox/grvyfile/src/test.txt";
     String line = "";
     StringBuilder sb = new StringBuilder();
     BufferedReader br = new BufferedReader(new FileReader(file));
     while ((line = br.readLine()) !=null) {
         sb.append(line);
     }
     System.out.println("sb.toString() = " + sb.toString());
 } catch (FileNotFoundException e) {
     System.out.println("e = " + e);
 } catch (IOException e) {
     System.out.println("e = " + e);
 }
}
}

If you are still doing this STOP RIGHT NOW! This boiler plate code took me about 5 minutes to write. Since it's not something I do super frequently I don't commit this syntax to memory, and therefore almost always have to reference the API to remember to use things like java.io.BufferedReader.

Would you like to see the Groovy equivalent? Make sure you don't blink; you might miss it.

def file = "/workspace/sandbox/grvyfile/src/test.txt"
println new File(file).text

That is all folks. In fact if you want to test this real quick, use the groovy command line like the following:

>cd /workspace/sandbox/grvyfile/src
>groovy -e "println new File('test.txt').text"

For those new to Groovy, .text is essentially shorthand for calling the .getText() method on the File object. The method getText() is one of the dozens of new methods added to the JDK (see the GDK for more).

Read my comments in this post (Java: Laugh or Cry, Your Choice...) by Kit Plummer for another good application of Groovy. Here I use the List GDK methods minus() and unique() to decrease the lines of code from ~27 to ~2.

Impressive Grails Plugin Architecture

2007-11-28T00:34:00.000-06:00

I just finished watching the video of Graeme Rocher explain the Grails Plugin Architecture while at the Grails Exchange in London recently (Graeme Rocher on the Grails Plugin Architecture). I got to say, while long at 53 minutes, it was well worth the time and was pretty impressive.

Some things I didn't realize was all the default core functionality in grails is plugin based; everything in grails is based on plugins. For example: spring, hibernate, internalionalization, domains, controllers, tab libraries, etc are all plugins.

On top of that it is really easy to create your own plugin (watch the video above or go here. Here are some of the things you can do with plugins that make them so powerful:

Add new methods, constructors, properties to any class at runtime
Perform runtime Spring configuration instead of depending on some static spring XML file that you wish you could change at runtime
Modify the web.xml on the fly
Add new controllers, domains, tag libraries, etc

Here are just a few plugins that already exist:

XFire - exposes a grails service as a SOAP service
Searchable - integrates compass and lucene search
Remoting - exposes a grails service over RMI, HTTP, or burlap
GWT - integrates grails with Google Web Toolkit
Acegi/JSecurity - adds security support for grails
JMS - exposes grails services as Message Driven Beans
Wicket/JSF/Tapestry - bring in different view frameworks
Dbmigrate - support sql migration
Static resources - helps offload the delivery of static resources to another web server
Jasper Reports - create jasper reports in grails

Go here for a complete list: http://grails.codehaus.org/Plugins.

Best Groovy quote ever

2007-11-21T09:52:00.000-06:00

I think this pretty much sums up Groovy:

"Groovy is executable psuedo-code"
Scott Davis

source http://www.javaworld.com/podcasts/jtech/2007/110107jtech004.html

Javadoc equivalent for Grails (a necessity)

2007-11-16T09:55:00.000-06:00

Oh yeah! Graeme Rocher and Marc Palmer have done an excellent job creating the Reference Documentation for the Grails Framework for version 1.0-RC1 (check it out http://grails.org/doc/RC1/). This will be familiar to anyone that has referenced Java's API javadocs like I have for so many years.

This is an absolute necessity for Grails development IMO due to it's dynamicity (I think I just made that up). Outside of checking out the source, yuck, how else would you know which options are available? This is another piece of the puzzle before the official release of Grails 1.0.

Here is what the reference documentation covers:

Grails Commands - detailed usage information for the grails commands like generate-all, test-app, etc.
Constraints - explains all the current constraints available to Domains like blank, maxSize, etc.
Controllers - shows all the information available for controllers
Domain Classes - shows all methods available to consumers of domain classes like save, validate, find*, etc.
Plugins - information available for plugins
Services - explains things like scope and transactional
Servlet API - typicaly servlet stuff like request and response
Tag Libraries - information available for tag libraries
Tags - tags you can use in your GSP and I am sure, but not certain, in your JSPs.

How to remote into Windows from Linux

2007-11-15T08:56:00.000-06:00

You can easily use rdesktop to remote into a Windows machine from Linux. I am using Ubuntu 7.10 and rdesktop works fast and its easy. The default resolution is really small so here is my command to start it up specifying a different resolution.

rdesktop -g 1024x800 server

You can also specify the -f option for full screen mode, but beware it's not easy getting out of rdesktop full mode. Use CRTL+ALT+ENT to exit full screen mode.

New Amazon S3 plugin for Grails

2007-11-05T22:48:00.000-06:00

Today, Catina Consulting announced the release of an open source Amazon S3 (Simple Storage Service) plugin for Grails with the following goals:

Host and manage file assets on Amazon S3 for storage and performance advantages
Provide easy mechanisms to reference S3-hosted assets in Grails applications
Make most efficient and cost-effective use of S3 hosting resources

Their first version appears to be rather useful with some excellent capabilities coming in the future. This first version focuses on managing user uploaded static content. Planned features include supporting different media types, custom bucket schemas, security integration, support for both REST and SOAP (currently only uses REST), and reporting on S3 usage.

Installation and configuration seem rather simple so it should be rather easy for Grails users to start immediately using this plugin.

What I think is the most interesting was their ability to reuse the JetS3t Toolkit which is written in Java. This just goes to show how groovy can really be the glue between Java and how developers can reuse all that existing Java code.

Go here to learn more about Amazon S3, which is an online storage web service providing unlimited storage through a simple web service interface (REST or SOAP). S3 aims to provide scalability, high availability, and low latency at cheap costs. It accomplishes this by using the same scalable storage infrastructure that amazon.com uses to run its own global e-commerce network.

For example, Smugmug, a popular photo sharing web site, is using it to store and host pictures. For a simple cost analysis view this post by Jeremey Zawodny who contemplates using S3 to replace his backup server.

And finally, I can't leave the Rails community out of this either so here is an article on using the rails AWS::S3 library (this plugin is more feature complete than the grails plugin).

Using Groovy to easily avoid nasty NullPointerException

2007-10-31T00:02:00.000-05:00

Groovy supports a pretty neat syntactic sugar operator to easily avoid the ugly NullPointerException: the question mark '?'.

Writing this in Java

if (name != null && param1 != null && param1.getUser() != null && name.equals(param1.getUser().getName()))

is now replaced by the much more readable Groovy

if (name && param1 && (name == param1.user?.name))

Amazing how a single character can improve readability that much and avoid potential NPEs. Also if you are curious about my use of == rather than the Java required .equals() see my post on More Groovy Sugar.

Most importantly I have inserted a hidden clue in this post and the first to solve the mystery will receive a free pop of their choice not to exceed 0.77 cents. The winner will be the first commenter with the correct answer.

Using maven with Grails

2007-10-28T23:47:00.000-05:00

By default Grails comes with an Ant build file to control your web application. For some of us this seems like a step backwards in technology (although apparently not for Graeme Rocher the Grails mastermind). Even though Grails doesn't support maven 1 or 2, you can get support for dependency resolution via Ivy (see Using Ivy for Dependency Resolution). Another option available is called Maven Tools for Grails (MTG). It sounds pretty promising and an improvement over Ivy, but their most recent release (v0.2) appears to work only with Grails 0.5.6 and 0.6; the current release of Grails is 1.0-RC1.

Grails vs Rails comparison

2007-10-28T23:31:00.000-05:00

While following some of the Grails and Rails tutorials (Grails: The Definitive Guide to Grails. RoR: Creating a weblog in 15 minutes) I noticed a big difference on how they approach development. Consequently, I started a thread (Dynamic Domain classes) on the Grails user mailing list to confirm. What I found, thanks to Tim Pidgen, was that Grails is intentionally domain-centric (ie model-centric) whereas Rails is more database-centric. What's the difference? With Grails you modify the domain classes (models) and the database is updated automatically. So properties are explicitly defined in the Grails domain classes; no database modification. Conversely, with Rails you create your "empty" model class and after you modify the database (add new fields to a table) you can just refresh the page; no need to add variables to your models.

Not sure either one is right or wrong; just wanted to point out the differences because it is a pretty significant difference.

No more J2EE webapps

2007-10-28T22:11:00.000-05:00

I got my start in my programming career over 7 years ago developing web applications as part of the intranet group for a Fortune 500 company. The group was switching from developing in Haht (I believe back then it was some type of vbscript) to Java (JSPs and Servlets). I joined the group at the moment they switched to Weblogic 5.1, JDK 1.3, and Oracle 8i (I think). It was a fast paced environment as we cranked out one internal application after another. Of course back then I wasn't the best web developer, but by the time I left I felt pretty confident in being able to hold my own (but humble at the same time).

One thing that always stayed the same throughout creating each one of those applications was the ability to rapidly develop them. What do I mean by rapidly develop? To me this means making a change and being able to switch to my browser and verify that change and repeating that hundreds of times a day.

That is how things were when I moved on 5 years later: (change + save + refresh) x 100s. Now it has been around 2 years since I have developed a real web application and for some reason that process has disappeared. For some reason my fellow J2EE developers have come to accept the process of rebuilding and hot deploying or even restarting the application server FOR EVERY CHANGE! Thousands or even millions of minutes are lost everyday by J2EE developers out there producing software like this. And to that I say, not good enough.

Maybe I am missing something; if so please enlighten me by commenting to this post. Take the JBoss Application Server (AS) 4.2.1 for example. I have tried finding better ways to deploy a WAR (exploded or bundled) on JBoss with no luck. The best I can come up with is increasing the JVMs PermSpace for better hot deployment (article), but I am still left with having to rebuild the WAR and copying it over. And sooner or later I have to restart the server. What a waste.

At one time there use to be a really slick maven 1 tomcat plugin made by codeczar.com that supported the ability to tell tomcat where your webapp was, preventing you from having to rebuild and redeploy. It was a huge time saver, but unfortunately this plugin is no longer supported (see here). If you are curious you can still get this plugin here thanks to Google Cache. The documentation has also been copied here.

Where does that leave me? Any language that lets me develop at a fast rate. Even if I lose tons of reusable components. I would imagine the time I save not rebuilding/redeploying more than makes up for reusable components and other goodies J2EE offers. In fact that has been the main reason for my personal quest to learn Grails and Rails. To me these languages offer the ability to deliver faster and cheaper and I believe this article supports that. If I had to start a new web project I would definitely look long and hard at either Grails or Rails.

Effectively using Groovy Console in Grails App

2007-10-25T23:42:00.000-05:00

Each chapter in The Definitive Guide to Grails continues to impress me. Currently I am on Chapter 4 which discusses The Application Domain. In this chapter the author uses the Groovy console within the grails app to test the domains to get some quick feedback. I got to say this ability is awesome! Not only could you do some quick testing, but I am sure it would be great for debugging purposes as well. Now obviously we want to keep testing in the console at a minimum and continue writing those unit tests, but that isn't until chapter 6. So until then here is how you can use the console in your grails app.

Prerequisites
1) Already created a grails app: grails create-app bookmarks
2) Created at least one domain: grails create-domain-class Bookmark
3) Added some properties to the Bookmark domain (URL url)

Now cd into the bookmarks directory and run grails console. This will compile your domain class and make it available to the console. Then execute the following (CRTL + R):

def bookmark = new Bookmark(url: new URL("http://jlorenzen.blogspot.com"))
println 'Bookmark url is ' + bookmark.url

You should see the following output:
Bookmark url is http://jlorenzen.blogspot.com

Grails Tip: Enable SQL Logging

2007-10-24T23:22:00.000-05:00

Here is a great tip while developing a Grails application to enable SQL logging (I would only recommend this in development mode).
In your grails project add the loggingSql = true property to the development closure and that is it. This was using Grails 1.0-RC1.

development {
   dataSource {
       dbCreate = "create-drop" // one of 'create', 'create-drop','update'
       url = "jdbc:hsqldb:mem:devDB"
       loggingSql = true              
   }
}

Then after starting your app (grails run-app) the SQL will be displayed. I was actually surprised for some reason when the list action was doing a specific select statement like "select title, author from book" rather than "select * from book".

Hotswap ability with JavaRebel

2007-10-24T16:13:00.000-05:00

Here is a good article on using JavaRebel to reload your webapp changes without a hotdeploy like Rails and Grails do.

http://stephan.reposita.org/archives/2007/10/11/javarebel-impressions-java-reload-just-like-in-rails/

A Groovy switch/case example

2007-10-24T12:58:00.000-05:00

Below is an example of Groovy's switch/case feature taken from The Definitive Guide to Grails by Graeme Rocher. I am not a regular user of Java's switch/case feature, but if it had been this way perhaps I might have.


switch (x) {
   case 'James':
       println "yes it is me"
       break
   case 18..65:
       println "ok you are old"
       break
   case ~/Gw?+e/:
       println "your name starts with G and ends in e!"
       break
   case Date:
       println 'got a Date instance'
       break
   case ['John', 'Ringo', 'Paul', 'George']:
       println "Got one of the Beatles"
       break
   default:
       println "Don't know"
}

More Groovy Sugar

2007-10-24T12:21:00.001-05:00

Joe Kueser has some excellent posts on Groovy features (What Makes Groovy...So Groovy and More Groovy Goodies) and I would like to attempt to build on them. He has been nice enough to let me borrow The Definitive Guide to Grails, which so far has been an excellent resource for Groovy and Grails (probably one of the best technical books I have read in awhile).

Expando Objects
Joe does a nice job of explaining this handy new class. Here is another example taken from the Grails book by Rocher:

fred = new Expando()

fred.firstName = "Fred"
fred.lastName = "Smith"

fred.age = 40
fred.happyBirthday = {
fred.age++
}

fred.happyBirthday()
assert fred.age == 41

You can easily play around with this code with groovy's console (/GROOVY_HOME/bin/groovyConsole). As you can see Expando allows you to define an object, its properties, and its methods at run time. Keep in mind that a method is defined by setting a closure to a property that can be later called like a regular method. One immediate use I can see for Expando is in unit tests.

Groovy's == operator
This is very important for existing Java developers. Groovy's == operator is different than Java's in that it does not evaluate object identity, rather it delegates to the object's equals method. To accomplish object comparison, Groovy introduces a new is method: left.is(right).

The Groovy Truth
Term used by Rocher in the Grails book and coined by Dierk Koenig, that explains the Groovy concept of what is true and what is not. Here is an example of what can be passed to if statements in Groovy and will evaluate to false:

A null reference
An empty or null string
The number zero
A regex Matcher that doesn't match

This makes those nasty Java if statements much cleaner.

if (str != null && str.length() != 0 )

now becomes

if (str)

More Mt. Dew sugar to come.

Enabling remote access to JBoss 4.2

2007-10-23T12:07:00.000-05:00

Today I had an issue remotely accessing my JBoss instance running on another machine. Unknowingly, JBoss 4.2 binds specifically to 127.0.0.1 by default (see below for me details). Thankfully they had a simple solution; just append -b 0.0.0.0 to your startup script to bind to all interfaces. So to start JBoss on linux when I know I want to access it remotely I do:

./jboss/bin/run.sh -b 0.0.0.0

ReadMe.html content on this subject:
"JBossAS now binds its services to localhost (127.0.0.1) *by default*, instead of binding to all available interfaces (0.0.0.0). This was primarily done for security reasons because of concerns of users going to production without having secured their servers properly. To enable remote access by binding JBoss services to a particular interface, simply run jboss with the -b option. To bind to all available interfaces and re-enable the legacy behavior use -b 0.0.0.0. In any case, be aware you still need to secure you server properly."

Great JetGroovy plugin feature for Idea+Grails

2007-10-21T23:33:00.000-05:00

Just started using the JetGroovy plugin in Idea and I have to say initially it Rocks! Included is a screenshot of the plugin in Idea version 7 after I created a domain (aka model). Notice how there are nifty buttons for that domains Controller, Views, Domain Tests, and Controller Tests. I was constantly using this ability even in the 5 minutes it took me to create my Hello World grails app. Nice work guys.

Tip for using Grails in Idea

2007-10-21T22:46:00.000-05:00

Recently I tried creating my first Grails application using the JetGroovy plugin available in Idea (version 7.0). However, I have Linux (Ubuntu 7.10) and you don't get those nice little icons to start Idea when you install it. Consequently when I tried to create a new Grails project I received the following error:

grails: JAVA_HOME is not defined correctly; can not execute: java

How in the world is that possible I thought to myself? What half decent java programmer doesn't have JAVA_HOME set? Oh but then I remembered my special start script responsible for starting Idea. Idea requires the variable JDK_HOME to be set, but Grails within Idea needs JAVA_HOME. Therefore, here is my new script I use to start Idea 7 which allows me to create new Grails projects. And don't forget Idea 7 on Linux requires JDK 1.6.

sh -c 'export JDK_HOME=/workspace/java/jdk1.6.0_03;export JAVA_HOME=/workspace/java/jdk1.5.0_12;/workspace/java/idea-7361/bin/idea.sh'