Showing posts with label rest. Show all posts
Showing posts with label rest. Show all posts

Tuesday, April 5, 2011

What's missing from our REST Services?

While reading the excellent book RESTful Web Services I discovered something. A technique used on the web since its beginning. Twitter and Facebook seem to use it. Yet it was absent in the REST Services I had been developing and if I had to guess most REST developers aren't including it either. What is so valuable the Internet would be useless without it? The answer is hypermedia.

Hypermedia is the technical term used to describe how web pages (resources) are linked or connected. The author of RESTful Web Services calls it connectedness: how the server guides the client from one application state to another by providing links. A good example of a well-connected application is Wikipedia. It's very powerful when you could pick a random page and click through every entry on Wikipedia without ever editing your browsers URL. Performing a search on Google is another great example, as it wouldn't be very useful if the search result didn't include any links. Without these links, the client must know and create predefined rules to build every URL it wants to visit.

So what do links on a website have to do with REST Services? REST was built on the foundation of the web and just because your not returning HTML doesn't mean you shouldn't create relationships between your resources. In fact, it's amazing how powerful embedding links in your responses can be. For instance, it prevents tight coupling between your clients and services as the clients don't have to construct or even know the exact URLs because the services are providing them.

Let's use Twitter as an example. Assume the following is the URL to get the 20 most recent tweets for my account:

http://api.twitter.com/1/statuses/user_timeline.json?screen_name=jlorenzen

And here is a shortened fictitious response:

[
    {
        "text": "Finished watching Battlestar Galatica"
        "id": "55947977415667712"
        "url": "http://api.twitter.com/1/statuses/show/55947977415667712.json"
    },
    {
        "text": "Started watching Battlestar Galatica"
        "id": "45947977415667712"
        "url": "http://api.twitter.com/1/statuses/show/45947977415667712.json"
    }
]

Notice each tweet includes a direct URL. Now clients can use that URL value verses constructing it, and if it changes in the future, clients don't have to make any changes.

Example using Jersey
So what is the best way to include links in your REST Services? Since I use Jersey on a daily basis, I'll go ahead and show an example of how to embed links in your responses using Jersey. Most likely, any REST framework is going to provide the same kind of features.

Using the twitter example above, the User Status Service with links may look something like this (using groovy):

import javax.ws.rs.GET
import javax.ws.rs.Path
import javax.ws.rs.Produces
import javax.ws.rs.QueryParam
import javax.ws.rs.core.Context
import javax.ws.rs.core.UriInfo

import com.test.UserStatuses

@Path("/statuses")
class StatusesResource {
 
    @Context 
    UriInfo uriInfo

    @GET
    @Path("/user_timeline")
    @Produces(["application/json", "application/xml"])
    def UserStatuses getUserTimeline(@QueryParam("screen_name")String screen_name) {
        def statuses = getStatuses(screen_name)
        
        statuses.each {
            it.url = "${uriInfo.baseUri}statuses/show/$it.id"
        }
        
        return new UserStatuses(statuses: statuses)
    }
}

In this simple example, Jersey injects the UriInfo object which we use to get the baseUri of the request. It's really that simple.


Potential Issues
Well for some it may not be that simple. For example, we discovered an issue when using a reverse proxy (apache httpd). In our production environments, we typically setup apache on port 80 to proxy a localhost JBoss on port 8080. Unfortunately, in this setup the UriInfo.getBaseUri() returns localhost:8080 and not the actual original URL the client used; which is obviously not good. Now if you don't use a reverse proxy then no worries. However, if you do or might potentially in the future, a easy solution seems to be to set the Apache Proxy module option ProxyPreserveHost to On. Setting this to On and restarting Apache seems to fix the issue.

JSONView Firefox Plugin
Once you've started embedding URLs in your REST responses, you might find it useful to install the JSONView firefox plugin. It's got some really slick features like formatting the JSON and creating clickable links for URLs.

Thursday, July 1, 2010

RSS, Lucene, and REST

Sorry for the horrible title. I struggled trying to come up with a worthy title, but after a few minutes I decided to not let perfection get in the way of good.

My team has recently worked on a new feature I am pretty excited about: adding support for RSS/Atom in our application. I know your thinking so what. It's not really the what I am excited about but the how. What I'm really excited about was how the story was defined and implemented.

Approach
We had the simple requirement from a newer customer to provide an RSS feed for newly created items. This actually wasn't the first time for this requirement. We prototyped a similar capability a long time ago using OpenESB and the RSS BC, but for multiple reasons it just didn't work out.

So our first decision had to answer: how we were going to implement it......again, but better. Before the sprint began, a few of us got together and hashed out a potential solution: how about we use the Search REST Service, which is backed by Lucene, to support Advanced searches and return RSS?

Why does this excite me so much? To understand that I need to explain our application at a high level. It's a completely javascript-based application using ExtJS (now sencha), backed by REST Services using Jersey. Consequently, we have a lot of REST Services. Right now those REST Services support returning XML or JSON using a custom Response Builder we have created internally.

I'm excited because this single user story could have a huge improvement on the entire system:

  1. If we modified the Search Service to return RSS, then all our REST Services could support RSS.
  2. The REST Service would now support Advanced searches. Previously, it only really supported basic keyword searches.
  3. Any search they perform could now be subscribe to via RSS.
Implementation
I'm not going to go into every detail on how it was done. I wasn't even actually the one who implemented it (see Matt White. He did a fantastic job.). We did have one major hurdle we had to overcome, and that was how to index items to enable advanced searches like Status=New.

Previously this wasn't possible given how we were indexing our items. We were basically indexing the item by building up a large String containing all the item information like the following:
import org.apache.lucene.document.Document
import org.apache.lucene.document.Field

def Document createDocument(item) {
    Document d = new Document()
    
    doc.add(new Field("content",
        getContent(item),
        Field.Store.NO,
        Field.Index.ANALYZED))
        
    return d
}

def String getContent(item) {
    def s = new StringBuilder()
    
    s.append(item.getTitle()).append(" ")
    s.append(item.getStatus()).append(" ")
    s.append(item.getPriority()).append(" ")
    s.append(item.getDescription()).append(" ")
    
    return s.toString()
}

The problem with this is performing a search for "New" would have returned any item with a status of New as well as any items that contained the word New. The solution was to just add another Field to the Document.
doc.add(new Field("Status",
    item.getStatus(),
    Field.Store.NO,
    Field.Index.NOT_ANALYZED));

Now the Search Service could support advanced searches like: Status:"New". You should put the value in quotes in case the value contains spaces (ie Status:"In Progress"). And since Lucene is so powerful, it also means the follow search would work: Status:"New" AND Priority:"High" AND "Hurricane". Now users have the freedom to subscribe to a near limitless amount of RSS feeds based on Advanced Searches.

Start to Finish
I think there were several reasons why this story was a success in my eyes. Most importantly where the two really smart co-workers who worked on it: Matt White and Chuck Hinson. All three of us knew of this user story ahead of time and we were able to discuss it technically days before backlog selection. This allowed us to brainstorm some ideas. Once we narrowed it down, we spent some more time separately looking into the code to find out the level of difficulty and if Advanced Searches like Status:New would be possible. Overall, together I'd say we spent 3-4 hours doing the preliminary work. Doing that preliminary work I think really enabled us to give a proper WAG for the story.

I really can't speak for how the development went (I was at Disney World for 10 days with the family), but I was really impressed with the tests Matt wrote. He wrote a number of unit tests making sure advanced searches worked and basic searches still worked. On top of that, he wrote an overall functional test using HttpBuilder executing the REST Service just as our javascript client would.

Finally, once the main work was finished, we uploaded a diff file to our internal instance of Review Board. From there I was able to perform a peer review where we found a minor bug in the changes.

Summary
I am sure it's not an original idea, but I thought it was a fun User Story that hopefully will provide a lot of value beyond what was originally estimated. Ideally, this might help others who are in similar situations.

Tuesday, February 16, 2010

Rapid REST Development with maven jetty plugin

Ever wonder how much time is wasted by Java developers rebuilding and redeploying web applications? For me alone I can't imagine it. In fact, I'd be embarrassed if Google Buzz made it public knowledge without my consent. Two years ago I wrote an article "No more J2EE Apps" and I received a lot of great feedback. Let me first say that I think, IMHO, Java Developers are at a disadvantage when it comes to rapidly developing web applications with a static language. Developers using python, php, rails, or grails really don't even have to spend a second trying to solve this problem. On the other hand, Java Developers have to figure out how to best accomplish this, and every situation seems to be different: JBoss, Weblogic, Eclipse, Idea, Netbeans, Jetty, JRebel. Of all the solutions I think JRebel provides the best chance for success that solves any environment no matter the web container or developers IDE of choice.

I haven't really done much hardcore java development in awhile, but in order to improve my team's productivity, I am going to be exploring best practices the next couple of months concerning this area. First up, I am going to explain how I got the maven jetty plugin to work with our REST Services WAR and the steps necessary to redeploy changes. The nice thing about jetty is it's easy to use from maven and we could use it in our CI environment to possibly reduce our build times and provide quicker feedback. The downside is each change requires jetty to hotdeploy the new WAR. End the end I think the best solution will be a combination of JBoss+JRebel. But I won't get to for awhile.

Maven Jetty Plugin
My first prototype uses the maven-jetty-plugin version 6. The application we are testing is a WAR containing REST Services built with Jersey (JAX-RS). Here is a good posting by my co-worker Jeff Black "Jersey...Jetty and Maven style!". This example didn't work for me because for some insane reason the init-param, com.sun.ws.rest.config.property.packages, does not work in WebSphere. So I had to do some slight modifications to get it to work without that being declared. My pom.xml and jetty.xml files are below. Most "normal" non-Jersey applications don't need all of this, but it was necessary to get our legacy WAR working with jetty.

Here are the steps involved to start jetty and redeploy changes:

  1. mvn install jetty:run - this will first build the WAR and then start jetty while also deploying the WAR. Running install was necessary because I reference the exploded WAR directory under target in the jetty configuration.
  2. Make changes to source code
  3. Run "mvn compile war:exploded" in a separate terminal to compile your changes and copy the new class files to the location Jersey expects to find them. Which in my case is /target/myapp/WEB-INF/classes
  4. Click back to the terminal running jetty and hit the ENTER key. This causes jetty to reload the WAR. This is because by default I set the scan interval to 0 and jetty.reload to manual, so I can batch up multiple changes before reloading.
Overall I am happy with the results so far. Previously, it took 1-2 minutes and sometimes more to rebuild the war and hotdeploy to JBoss. Using the jetty plugin this now takes around 30 seconds. Again, I think this could be further improved by using JRebel.

Tips
I did have to update my MAVEN_OPTS environment variable to increase Java's PermGenSpace since jetty reloads the WAR each time and you'll quickly run out of memory. This was something I was already doing in JBoss. Here is what it is set to:

export MAVEN_OPTS="-Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=512m"

Sample Files
Here is my pom.xml
<plugin>
<groupId>org.mortbay.jetty</groupId>
<artifactId>maven-jetty-plugin</artifactId>
<version>6.1.22</version>
<configuration>
<jettyConfig>${project.build.testOutputDirectory}/jetty.xml</jettyConfig>
<scanIntervalSeconds>${jetty.scan.sec}</scanIntervalSeconds>
<useTestClasspath>true</useTestClasspath>
<webAppConfig>
<baseResource implementation="org.mortbay.resource.ResourceCollection">
<resourcesAsCSV>${project.build.directory}/myapp</resourcesAsCSV>
</baseResource>
</webAppConfig>
<systemProperties>
<systemProperty>
<name>jetty.port</name>
<value>${jetty.port}</value>
</systemProperty>
</systemProperties>
<systemProperties>
<systemProperty>
<name>jetty.reload</name>
<value>${jetty.reload}</value>
</systemProperty>
</systemProperties>
</configuration>
<dependencies>
<dependency>
<groupId>commons-dbcp</groupId>
<artifactId>commons-dbcp</artifactId>
<version>1.2.1</version>
</dependency>
<dependency>
<groupId>net.sourceforge.jtds</groupId>
<artifactId>jtds</artifactId>
<version>1.2</version>
</dependency>
<dependency>
<groupId>com.oracle.jdbc</groupId>
<artifactId>ojdbc14</artifactId>
<version>10.2.0</version>
</dependency>
</dependencies>
</plugin>
.....
<properties>
<jetty.port>8080</jetty.port>
<jetty.scan.sec>10</jetty.scan.sec>
<jetty.reload>manual</jetty.reload>
</properties>

Here is my jetty.xml located under /src/test/resources used to define the Datasource.
<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Mort Bay Consulting//DTD Configure//EN" "http://jetty.mortbay.org/configure.dtd">

<Configure id="Server" class="org.mortbay.jetty.Server">

<New id="MYAPP-DS" class="org.mortbay.jetty.plus.naming.Resource">
<Arg>jdbc/MYAPP-DS</Arg>
<Arg>
<New class="org.apache.commons.dbcp.BasicDataSource">
<Set name="driverClassName">oracle.jdbc.driver.OracleDriver</Set>
<Set name="url">jdbc:oracle:thin:@localhost:1521:XE</Set>
<Set name="username">user</Set>
<Set name="password">password</Set>
</New>
</Arg>
</New>

</Configure>