talkingCode

iPojoRC – An iPOJO-based OSGi IRC Bot

I’ve been doing a fair bit of work recently in OSGi land. OSGi is a dynamic module system for Java designed to facilitate the creation, distribution and consumption of modular Java code. The idea is that anybody who has an application running on an OSGi platform implementation (Felix, Equinox) can retrieve your module (optionally stored in an OBR) and have it load in to their running system without upsetting anything. Exactly how well the hot-swapping of bundles works depends heavily on how well the application is written and how well the bundle being loaded / unloaded is written, but it can be made to work.

The Whiteboard Pattern

I’m not quite sure why it’s called a whiteboard pattern, but one of the neat things about the OSGi platform is the service registry. Service consumers and producers interact through the service registry that the platform provides, which takes care (in principle) of service lifecycle and dependency resolution. A correctly written service only becomes available to a correctly written consumer when the service’s dependencies have been met and its initialisation is complete. If the dependencies of any service are no longer satisfied (because of the disappearance of a required bundle), the service is removed from the registry and consumers are notified. Not rocket science, but saves developers a fair bit of effort and makes interoperable components easier to write.

The canonical example of the whiteboard pattern is an IRC bot whose commands are implemented as services. In this example, an IRC Bot is created that maintains its connection to the IRC server while commands are loaded and unloaded from the Bot on the fly. Here, the ServiceTracker is used to listen for the appearance and disappearance of bundles. Neat, but there’s still some associated boilerplate:

  private ServiceTracker serviceTracker;

  public void start(BundleContext context) throws Exception {
    serviceTracker = new ServiceTracker(context, 
                              MyTrackedService.class.getName(), null);
    serviceTracker.open();
    ... startup code...
  }

  public void doStuff() {
    Object[] implementors = serviceTracker.getServices();
    if (implementors != null && implementors.length > 0) {
      for (Object o : implementors) {
        MyService myService = (MyService) o;
        ... do stuff...
      }
    }
  }

  public void stop(BundleContext context) throws Exception {
    serviceTracker.close();
    serviceTracker = null;
    ... shutdown code ...
  }

iPOJO

iPOJO attempts to reduce this boilerplate and make service registration and listening even simpler. Under iPOJO, the boilerplate is reduced to:

  @Requires
  private MyService[] implementors;

plus some hand waving. iPOJO takes care of making sure that the implementors array contains registered implementations of MyService – all your application has to do is iterate over them.

iPojoRC

With a view to having a play around with this stuff, I thought I’d implement the IRC Bot using iPOJO. You can find the fruits of that labour on my github account. It seems to work pretty well. I’ve so far not had a problem adding and removing commands on the fly. It’s even possible (if you’re really careful with versions) to rev parts of the IRCCommand API and have the old and new versions running side by side, but in practice it’s much easier just to mvn clean install.

Why?

Summut to do, innit. More seriously, OSGi seems to be gaining ground as a way of building and distributing Java applications. It plays very nicely with Maven and there are some quite neat bundles available that you can just drop in to your application with very little wiring. That said, one of the challenges of OSGi is dealing with packages and dependencies when you create your own bundles. Most of the tutorials recommend starting with all your code in a single bundle, with good reason – I invariably spend more time sorting out dependency and classloader issues with my bundles than I do writing the code inside them. If you’re going to start a serious OSGi project, it’s good to have experience of bundling simple things and the IRC Bot is a great example of simple code in many bundles.

Abstract Type Members – Augmenting Scala classes

So, Scala’s cool.

About a month ago, a colleague was whining about having to write Java and it was hard not to sympathise. He’s a Python man, and Java’s Kingdom Of Nouns can seem a little clumsy. Sometimes, you just want to map{}, and trying to do that in Java can leave you feeling dirty.

There are proposals for first class functions and closures in Java – Neal Gafter is your man there. They look fun n’all, but they are apparently a little way off. What I needed in order to silence my colleague was less irritating Java in production straight away. In order to satisfy the needs of m’colleague, m’self and m’company I would need a couple of things.

Pour lui

• Concise variable declarations – none of this

  Map<String,List<Integer>> map = new HashMap<String, List<Integer>>()

  nonsense

• Functions as first class values
• Excellent library support

Pour moi

• Strong, static typing
• Eclipse integration
• Monads. Everybody loves monads

For practicality’s sake

• Maven integration
• Interoperation with the existing Java code base

I had a look on Wikipedia for a suitable JVM language since that would ensure the easy interoperation and give us a decent set of libraries. It turns out Scala has everything we need. It’s mixed paradigm functional / object-oriented, it’s got a Hindley-Milner style type inference system, and it’s got a really active community. The eclipse/maven integration isn’t quite there yet, but it gets much better as you get closer to the bleeding edge.

Some Code

There have been too many cool things to mention really, but here’s the first one that seemed apt for a blog post. Scala has some nice XML support, but lacks some of the functions you might like on an XML Node like, for example, ‘getAttributeValue(attributeName)’. In Haskell, you could define a new typeclass and create an instance for Node. In Java / C++ you could delegate (if you like pain). In Ruby, you could mixin. The answer in Scala appears to be… an Abstract Type Member.

Disclaimer:
I don’t know what the right answer actually is, but this works and seems kinda cool

import scala.xml.NodeSeq
import scala.xml.Node

object XmlTest
{
  implicit def nodeToNodePlus(node: Node):NodePlus =
  {
    return new NodePlus { type T = Node; val init = node }
  }

  abstract class NodePlus
  {
    type T <: scala.xml.Node
    val init: T
    private var value: T = init

    def getAttributeValue(name: String): String =
    {
      return value.attribute(name).get.first.text
    }
  }

  def main(args: Array[String])
  {
    println("Hello")
    val xmlDoc = <hi><hello id="world"/></hi>
    val hello = (xmlDoc\"hello").first
    println(hello.getAttributeValue("id"))
  }
}

The mechanics of what's going on there are... err... pretty opaque to me. But to break that down a little:

• An object is a singleton class, equivalent in most senses to a static class in Java.
• An implicit is a function from A to B which the compiler will use whenever inference tells it that you need a B but have an A (yes, evil, but okay if used carefully).
• type T is an Abstract Type Member. I think.
• <: is a subclass restriction

... all of which allows me to introduce a new function to an existing class without access to that class's constructor. Handy.

That's probably garbage. No doubt I'll find out that's not the way to do it or that it won't do what I expect, but it seems to work. The take-away is that Scala is an easy transition from Java, is much prettier and cooler, and that I can't think of a good reason to write any more Java code.

... having slept on it...

Ah. You see. What I did there was... I got carried away. My NodePlus class is actually no better than a wrapper - I can't actually do any Node operations on it. The implicit mechanism makes it look like I can, but I can't. And the Abstract Type Member was a bit of a red herring. What the Abstract Type Member is doing is letting me parameterise my type which is, I guess, a kind of useful. Given that Scala types accept formal parameters, though, it's not completely clear how much win is involved.

I'll let you know if I find out how to do the thing I actually want to do.

Un-overriding hashCode in Java

Problem: you are inheriting from a class in Java. This class has overridden Object.hashCode(). However, you do not want the behaviour of the overriding method, but rather that of the original implementation in the Object class.

Solution: in your new class, create a private field of type Object. Override the hashCode() method in the class to call hashCode() on that field. The result should follow a pattern similar to that below:

public class CustomHashSet<T> extends HashSet<T> {
    private Object hasher;

    public CustomSpecHashSet() {
        super();
        hasher = new Object();
    }

    @SuppressWarnings("unchecked")
    public synchronized Object clone() {
        CustomHashSet<T> copy = (CustomHashSet<T>)super.clone();
        copy.hasher = new Object();
        return copy;
    }

    public int hashCode() { return hasher.hashCode(); }

    public boolean equals(Object o) { return this == o; }
}

Discussion

In Java, once a method has been overridden, it is impossible to get at the original implementation from any inheriting classes. I recently had to deal with this when I wanted to keep some objects that inherited from various Collections inside HashSets, and wanted distinct objects in the set to not be treated as equals just because they happened to contain the same elements. In other words, I needed to use the Object implementations of the hashCode() and equals() methods, rather than the versions defined by the Collections. This is because Object.hashCode() returns a unique integer for every distinct object (typically by using the memory address of an object).

At first, I thought this problem could be solved with reflection. The soloution should simply involve getting a Method instance for Object.hashCode(), and invoke that on any object. Unfortunately, that does not work; calling a method using reflection in Java results in a dynamic method lookup based on the object the method is being invoked on.

Since Java does not permit un-overriding methods, one has to cheat. An object that is never leaked to the outside world can be used to provide a unique integer for each instance. However, care must be taken when implementing classes that support cloning (i.e. implement the Cloneable interface, or inherit from a class that does). Since the clone() method only does a shallow copy of an object, it is essential to override clone(), and make sure that any copies created there get their own instance of the hashing object. There is another gotcha; in the presense of multiple threads accessing the same object, the following clone() implementation would be unsafe:

public Object clone() {
    MyObject copy = (MyObject)super.clone();
    copy.hasher = new Object();
    return copy;
}

This is because the Java memory model only guarantees initialization safety for objects created using a constructor. What this means is that, for an object created using clone() and changing fields after the call to clone(), the updates to those fields are not guaranteed to be seen immediately by other threads accessing that object. A simple way to address this is to put the initialization of the cloned object inside a synchronized block, as in the code given at the top of this recipe.

Java, SSL, and the Keystore of Doom

In a break from our Haskell programming…

I used to be a staunch defender of Java as a language. I still think it’s relatively good (though mostly for the tool support), but there are things about it that make me want to scream.

Imagine, for example, you’d like to make an SSL authenticated fetch from a webserver. You have a client certificate to authenticate your client, and a server certificate to authenticate the server, and you’ve generated them both from your own CA. Shouldn’t be that hard, right? Wrong Everything SSL has to be configured via the key stores, so you need to import your private certificate and the server’s public certificate in to your key store in order to make anything go.

There are two stores – the Trust Store and the Key Store. The Trust store contains the certificates you trust (CAs, etc.). The Key Store contains certificates for which you have the private key and against which you’ll encrypt challenges to verify your identity. All you have to do is populate them…

Step 1: Don’t use GCJ
There are a lot of great things to be said for the Open Source outlook on life. GCJ isn’t one of them. It works quite like Java, except when you try and run anything. Unfortunately it works sufficiently like Java that you don’t necessarily know you’re using it, and it’s installed as the default on a lot of Debian machines.

keytool error: java.lang.IllegalStateException: masked envelope

That was the first cryptic clue that I was using GCJ. Other clues are random GC messages on the console. Here’s a quick way to tell if you’re infected:

$ ls -l /etc/alternatives/ | grep -c java-gcj
24

The number you’re looking for is ’0′ on a correctly configured system. Specifically you want to see:

$ chase `which keytool`
/usr/lib/jvm/java-1.5.0-sun-1.5.0.15/jre/bin/keytool
$ chase `which java`
/usr/lib/jvm/java-1.5.0-sun-1.5.0.15/jre/bin/java

If you’re not getting that, reconfigure the alternative:

# update-alternatives --config java

There are 7 alternatives which provide `java'.

  Selection    Alternative
-----------------------------------------------
          1    /etc/alternatives/kaffe-system/bin/java
          2    /usr/bin/gij-wrapper-4.0
*         3    /usr/lib/jvm/java-1.5.0-sun/jre/bin/java
          4    /usr/bin/gij-4.1
          5    /usr/bin/gij-4.3
 +        6    /usr/lib/jvm/java-gcj/jre/bin/java
          7    /usr/bin/gij-4.2

Step 2: Import the CA
Now we’re running the right JVM, it should be a simple matter of:

$ #Create a trust store with a CA Cert in it (teststore.jks doesn't yet exist)
$ keytool -import -v -trustcacerts -alias myalias -file cacert.pem -keystore teststore.jks
Enter keystore password: 
keytool error: java.lang.NullPointerException

Oh. There may be a way to use blank passwords on keystores, but keytool ain’t it. Let’s try again with a password:

$ keytool -import -v -trustcacerts -alias myalias -file cacert.pem -keystore teststore.jks
Enter keystore password: password

Smashing. That’ll mean we can at least connect to the remote host. But the SSL handshake will still fail when the host sees our lack of client certificate.

main, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

Step 3: Import the Client certificate
My client certificate comes as two files – a certificate PEM file (the public part) and a key PEM file (the private part). Naïvely, I tried just installing the PEM part:

$ keytool -import -v -alias myalias2 -file signup1-cert.pem -keystore teststore.jks
Enter keystore password:  password
keytool error: java.lang.Exception: Input not an X.509 certificate

*sigh*. That’ll happen. Fortunately, we can convert from PEM to ‘DER’, which is something that keytool understands, using ‘openssl’:

$ openssl x509 -in signup1-cert.pem -inform PEM -out signup1-cert.der -outform DER
$ keytool -import -v -alias myalias2 -file signup1-cert.der -keystore teststore.jks
Enter keystore password:  password
Certificate was added to keystore
[Storing teststore.jks]

It’s stored, but unfortunately a) it doesn’t work and b) keytool thinks this is a ‘trustedCertEntry’ rather than a ‘keyEntry’:

$ keytool -v -list -keystore teststore.jks
...
Entry type: trustedCertEntry
...

Now, we can use ‘openssl’ to convert our certificate and key into a PKCS#12 combined key file:

$ openssl pkcs12 -export -in signup1-cert.pem -inkey signup1-key.pem -out signup1.p12

Even better, according to the documentation, PKCS#12 format files are valid key stores… unless you try and use them:

default context init failed: java.io.IOException: Invalid keystore format

Right. Let’s try that as a combined PEM format file then:

$ openssl pkcs12 -in mykey.p12 -out keystore.pem -nodes 
$ keytool -import -v -alias clientcert -file keystore.pem -keystore keystore.jks
Enter keystore password:  password
keytool error: java.lang.Exception: Input not an X.509 certificate

True. (Incidentally, that’s No DES, not ‘nodes’) But we can convert PEM files to DER files

$ openssl x509 -in keystore.pem -inform PEM -out keystore.der -outform DER
$ keytool -import -v -alias clientcert -file keystore.der -keystore keystore.jks
Enter keystore password:  password

Step 4: Using the keystore in your program
You can configure the keystore at runtime as follows:

    System.setProperty("javax.net.ssl.keyStore", context.getRealPath(KEYSTORE));
    System.setProperty("javax.net.ssl.keyStorePassword", "password");
    System.setProperty("javax.net.ssl.trustStore", context.getRealPath(TRUSTSTORE));
    System.setProperty("javax.net.debug", "ssl");
    HttpClient httpClient = new HttpClient();
    GetMethod httpGet = new GetMethod("https://something.com");
    httpClient.executeMethod(httpGet);
    return new String(httpGet.getResponseBody());

javax.net.debug=ssl truly is a magic rune. I don’t know if you can get a list of such runes, but commit that one to memory. The debug output is pretty handy, if I little hard to follow.

Step 5: Become frustrated
What I didn’t mention, and perhaps should have mentioned above, is that I still hadn’t managed to import my key as a keyEntry, so this code still didn’t work. I downloaded the source code to the JDK and tried single-stepping KeyTool, but that also didn’t help. [Aside: People who create compressed archives without a top-level folder should be shot].

Step 6: Use KeyMan
KeyMan can be downloaded from IBM Alphaworks at time of writing:
http://www.alphaworks.ibm.com/tech/keyman/download
It ‘just works’. It lets you import your certificate and create a valid key store. Thanks IBM! (Source code plz).

Buzzword Bingo – Tomcat, Jetty, Cactus, Derby, Velocity, Maven

I’ve been writing a webapp at work recently. Because I’m writing it in Java, the scope for me to post interesting articles about what I’m doing has been pretty limited – I don’t know that much about enterprisey Java, and I spend most of my time bashing my head against a Java-shaped wall. A quick overview is probably worth doing though.

Apache
If you’ve not heard of the Apache Foundation (outside the context of their fine webserver), go visit their site now. Amongst other things, Apache are creating an incredible resource on that site in the form of a huge code commons. If you’re not sure what a commons is, a) shame on you and b) read this fine book (in fact, read it even if you are sure). The majority of the code on the Apache site is Java, so if you’re developing an application in Java it’s well worth checking the links on the left-hand side of the main page to see if they’ve already written what you’re writing, or something that will help you. The code is all Apache-licensed – a BSD-style license that is amenable to reuse in commercial projects as well as Free Software projects. It’s also very high quality code. You basically can’t lose.

strace is your friend
One of the biggest irritations, I’ve found, in using the Apache Java projects (and this is more of a function of Java than Apache) is trying to make Java find the appropriate config files. It’s pretty difficult to tell which config files Java is seeing, if any. You can download the source to the component you’re using and single step it in Eclipse, but this tends to get tedious at about the point you hit the class loader, which is exactly the point at which it’ll do any resource location. What you can do, though, is:

strace -f -p[YOUR_PID] 2>&1 | grep "your_config_file.xml"

You’re looking for things like calls to ‘stat’ or results of ‘ENOENT’. This will generally tell you what Java’s looking for and where. That is, I grant you, a sledgehammer-nut solution, but I’ve found it quicker than anything else. If you know a better way to work out where I should be putting my Velocity toolbox.xml in my Maven tree for the Webapp to run in Jetty under Cactus please let me know. It’s not in the FAQ.

My environment
What am I doing with all these projects?

• Maven – Your lifecycle management tool (build and dependency management). Maven2 is much improved over Maven for having the package repositories work 99% of the time rather than the 50% that was more common in the original. The whole POM thing is… errr… a little obtuse, I grant you, but you can use Maven in a dumb way without too much trouble. The Eclipse plugin is a great way to find the packages you need too.
• Tomcat – The application server. Can’t really do a Webapp without one of those. Tomcat isn’t the only server out there, but it’s pretty well used and, as at 5.5-ish, not too difficult to make go.
• Velocity – The templating engine. The web is awash with webapp ‘frameworks’; Apache have about four of them for starters. Sometimes, though, you’re not writing for web browsers. Sometimes, you just want a way to turn your POJOs into arbitrary markup. Velocity is fast, uncomplicated and fully featured.
• Cactus – The Servlet test framework. I’ve still not completely wrapped my head around Cactus, but basically for each of your unit tests you have setUp and tearDown to manipulate state on the server, and begin and end to manipulate state on the client, and Cactus will run web requests inside a test harness for you. Trés handy.
• Jetty – A lightweight servlet container. Controversially not an Apache project, Jetty is really quite handy for running your Cactus tests. Instead of pushing the WAR all the way to Tomcat, you can new up a Jetty server inside the Maven test task and run your functional tests there.
• Derby – The lightweight database. To avoid having to have a real database available in order to run your unit tests, I find it convenient to put some test fixtures in a Derby DB and run the servlet off the back of that. This plays really nicely with the Cactus / Jetty setup and has the added advantage that (unlike MySQL) if you try and create a prepared statement that your schema can’t possibly run, Derby will warn you about that at preparation time rather than execution time. With a bit of tweaking, then, you can statically check your statements against the schema.

Making all that play together has been nightmarish, either because of my relative inexperience or because it’s genuinely difficult. I’ve embarrassingly failed to write down all the errors I encountered and how to fix them, but I may cover some of the more tricky parts of the setup in future articles. For all the setup work, though, being able to type mvn install and feel confident that your latest refactoring hasn’t broken the complex application you’ve written is worth almost any amount of blood, sweat and tears.