talkingCode

iPojoRC - An iPOJO-based OSGi IRC Bot

I’ve been doing a fair bit of work recently in OSGi land. OSGi is a dynamic module system for Java designed to facilitate the creation, distribution and consumption of modular Java code. The idea is that anybody who has an application running on an OSGi platform implementation (Felix, Equinox) can retrieve your module (optionally stored in an OBR) and have it load in to their running system without upsetting anything. Exactly how well the hot-swapping of bundles works depends heavily on how well the application is written and how well the bundle being loaded / unloaded is written, but it can be made to work.

The Whiteboard Pattern

I’m not quite sure why it’s called a whiteboard pattern, but one of the neat things about the OSGi platform is the service registry. Service consumers and producers interact through the service registry that the platform provides, which takes care (in principle) of service lifecycle and dependency resolution. A correctly written service only becomes available to a correctly written consumer when the service’s dependencies have been met and its initialisation is complete. If the dependencies of any service are no longer satisfied (because of the disappearance of a required bundle), the service is removed from the registry and consumers are notified. Not rocket science, but saves developers a fair bit of effort and makes interoperable components easier to write.

The canonical example of the whiteboard pattern is an IRC bot whose commands are implemented as services. In this example, an IRC Bot is created that maintains its connection to the IRC server while commands are loaded and unloaded from the Bot on the fly. Here, the ServiceTracker is used to listen for the appearance and disappearance of bundles. Neat, but there’s still some associated boilerplate:

  private ServiceTracker serviceTracker;

  public void start(BundleContext context) throws Exception {
    serviceTracker = new ServiceTracker(context,
                              MyTrackedService.class.getName(), null);
    serviceTracker.open();
    ... startup code...
  }

  public void doStuff() {
    Object[] implementors = serviceTracker.getServices();
    if (implementors != null && implementors.length > 0) {
      for (Object o : implementors) {
        MyService myService = (MyService) o;
        … do stuff…
      }
    }
  }

  public void stop(BundleContext context) throws Exception {
    serviceTracker.close();
    serviceTracker = null;
    … shutdown code …
  }

iPOJO

iPOJO attempts to reduce this boilerplate and make service registration and listening even simpler. Under iPOJO, the boilerplate is reduced to:

  @Requires
  private MyService[] implementors;

plus some hand waving. iPOJO takes care of making sure that the implementors array contains registered implementations of MyService - all your application has to do is iterate over them.

iPojoRC

With a view to having a play around with this stuff, I thought I’d implement the IRC Bot using iPOJO. You can find the fruits of that labour on my github account. It seems to work pretty well. I’ve so far not had a problem adding and removing commands on the fly. It’s even possible (if you’re really careful with versions) to rev parts of the IRCCommand API and have the old and new versions running side by side, but in practice it’s much easier just to mvn clean install.

Why?

Summut to do, innit. More seriously, OSGi seems to be gaining ground as a way of building and distributing Java applications. It plays very nicely with Maven and there are some quite neat bundles available that you can just drop in to your application with very little wiring. That said, one of the challenges of OSGi is dealing with packages and dependencies when you create your own bundles. Most of the tutorials recommend starting with all your code in a single bundle, with good reason - I invariably spend more time sorting out dependency and classloader issues with my bundles than I do writing the code inside them. If you’re going to start a serious OSGi project, it’s good to have experience of bundling simple things and the IRC Bot is a great example of simple code in many bundles.

Abstract Type Members - Augmenting Scala classes

So, Scala’s cool.

About a month ago, a colleague was whining about having to write Java and it was hard not to sympathise. He’s a Python man, and Java’s Kingdom Of Nouns can seem a little clumsy. Sometimes, you just want to map{}, and trying to do that in Java can leave you feeling dirty.

There are proposals for first class functions and closures in Java - Neal Gafter is your man there. They look fun n’all, but they are apparently a little way off. What I needed in order to silence my colleague was less irritating Java in production straight away. In order to satisfy the needs of m’colleague, m’self and m’company I would need a couple of things.

Pour lui

• Concise variable declarations - none of this

  Map<String,List<Integer>> map = new HashMap<String, List<Integer>>()

  nonsense

• Functions as first class values
• Excellent library support

Pour moi

• Strong, static typing
• Eclipse integration
• Monads. Everybody loves monads

For practicality’s sake

• Maven integration
• Interoperation with the existing Java code base

I had a look on Wikipedia for a suitable JVM language since that would ensure the easy interoperation and give us a decent set of libraries. It turns out Scala has everything we need. It’s mixed paradigm functional / object-oriented, it’s got a Hindley-Milner style type inference system, and it’s got a really active community. The eclipse/maven integration isn’t quite there yet, but it gets much better as you get closer to the bleeding edge.

Some Code

There have been too many cool things to mention really, but here’s the first one that seemed apt for a blog post. Scala has some nice XML support, but lacks some of the functions you might like on an XML Node like, for example, ‘getAttributeValue(attributeName)’. In Haskell, you could define a new typeclass and create an instance for Node. In Java / C++ you could delegate (if you like pain). In Ruby, you could mixin. The answer in Scala appears to be… an Abstract Type Member.

Disclaimer:
I don’t know what the right answer actually is, but this works and seems kinda cool

import scala.xml.NodeSeq
import scala.xml.Node

object XmlTest
{
  implicit def nodeToNodePlus(node: Node):NodePlus =
  {
    return new NodePlus { type T = Node; val init = node }
  }

  abstract class NodePlus
  {
    type T <: scala.xml.Node
    val init: T
    private var value: T = init

    def getAttributeValue(name: String): String =
    {
      return value.attribute(name).get.first.text
    }
  }

  def main(args: Array[String])
  {
    println("Hello")
    val xmlDoc = <hi><hello id="world"/></hi>
    val hello = (xmlDoc\"hello").first
    println(hello.getAttributeValue("id"))
  }
}

The mechanics of what’s going on there are… err… pretty opaque to me. But to break that down a little:

• An object is a singleton class, equivalent in most senses to a static class in Java.
• An implicit is a function from A to B which the compiler will use whenever inference tells it that you need a B but have an A (yes, evil, but okay if used carefully).
• type T is an Abstract Type Member. I think.
• <: is a subclass restriction

… all of which allows me to introduce a new function to an existing class without access to that class’s constructor. Handy.

That’s probably garbage. No doubt I’ll find out that’s not the way to do it or that it won’t do what I expect, but it seems to work. The take-away is that Scala is an easy transition from Java, is much prettier and cooler, and that I can’t think of a good reason to write any more Java code.

… having slept on it…

Ah. You see. What I did there was… I got carried away. My NodePlus class is actually no better than a wrapper - I can’t actually do any Node operations on it. The implicit mechanism makes it look like I can, but I can’t. And the Abstract Type Member was a bit of a red herring. What the Abstract Type Member is doing is letting me parameterise my type which is, I guess, a kind of useful. Given that Scala types accept formal parameters, though, it’s not completely clear how much win is involved.

I’ll let you know if I find out how to do the thing I actually want to do.

Pixmap, Pixbuf and memory lane

So how’s the Haskell project going? So far, so good. When I last wrote, I’d managed to load a “Hello World!” GUI for my application. Since then I’ve managed to parse up some content and render it to the screen. Current revision is 121ed7f2.

Antediluvian file formats

The project, in case you’ve not been following on github, is to resurrect a long-since abandoned game. This game was released as a cover disk on an Amiga magazine back in “the day” and the code has since been made available by the authors (in 68000 assembly), ported to BlitzBasic and (abortively) ported to C++/SDL.

The maps for the BlitzBasic game were stored as tile maps. Each game level combines a tile set and a map to place the tiles on the screen. For additional kicks, some of the files are compressed under a scheme called “BPCK” which involves fairly simple run-length encoding. The code to parse up those files was… err… relatively straightforward to write although I did have some issues with bit-order and palette mapping.

Tiles as Pixbufs

The tiles are stored as sequences of nibbles mapping pixels on to a 16-colour palette. In order to draw these, I extracted the RGB for the nibbles and inserted that 3-byte-per-pixel sequence into a pixmap (this is the source):

buildTile :: [BPCK.PaletteEntry] -> BPCK.Gliph -> IO Pixbuf
buildTile palette g = do
    let gData = BPCK.gliphData g
    buf <- pixbufNew ColorspaceRgb False 8 gliphX gliphY
    pbData <- (pixbufGetPixels buf :: IO (PixbufData Int Word8))
    rowStride <- pixbufGetRowstride buf
    chan <- pixbufGetNChannels buf -- Hopefully this is 3 (R,G,B)
    bits <- pixbufGetBitsPerSample buf -- Hopefully this is 8
    doFromTo 0 (gliphX - 1) $ \y ->
      doFromTo 0 (gliphY - 1) $ \x -> do
        let pixbufoffset = x*chan + y*rowStride
        let gliphOffset = fromIntegral $ x + y*gliphX
        let paletteIndex = B.index gData gliphOffset
        let thiscolor = palette !! fromIntegral paletteIndex
        writeArray pbData (pixbufoffset) (fromIntegral $ BPCK.red thiscolor)
        writeArray pbData (1 + pixbufoffset) (fromIntegral $ BPCK.green thiscolor)
        writeArray pbData (2 + pixbufoffset) (fromIntegral $ BPCK.blue thiscolor)
    return buf
    where gliphX = BPCK.gliphWidth g
             gliphY = BPCK.gliphHeight g

Maps as Pixmaps

The difference between a pixbuf and a pixmap is, as I understand it, that a pixbuf is an X-client side block of data into which you can load pixel information and a pixmap is an X-server side drawable (in the GTK sense) object. Pixbuf is more like a brush for painting, and Pixmap is an off-screen canvas. So to build my map I blatted the tile pixbufs on to a pixmap:

createTiledPixmap :: BPCK.ParsedImage -> BPCK.ParsedTileMap -> IO Pixmap
createTiledPixmap tileSet tileMap = do
    putStrLn $ "Building tile pixmaps"
    tiles <- tilesFromImageData tileSet
    let tileCount = length tiles
    putStrLn $ "Creating new pixmap " ++ show totalWidthPixels ++ " x " ++ show totalHeightPixels
    pixmap <- pixmapNew (Nothing :: Maybe DrawWindow) totalWidthPixels totalHeightPixels (Just 24)
    gc <- gcNew pixmap
    doFromTo 0 (tilesHigh - 1) $ \iy ->
      doFromTo 0 (tilesAcross - 1) $ \ix -> do
        let tileIndex = ix + (iy * tilesAcross)
        let tileId = (min (fromIntegral (BPCK.tileMap tileMap !! tileIndex)) tileCount) `mod` tileCount
        let curX = ix * tileSizePixels
        let curY = iy * tileSizePixels
        postGUIAsync $ drawPixbuf pixmap gc (tiles !! tileId) 0 0 curX curY tileSizePixels tileSizePixels RgbDitherNone 0 0
    return pixmap
    where tileSizePixels = BPCK.gliphSize tileSet
             tilesAcross = BPCK.tilesAcross tileMap
             tilesHigh = BPCK.tilesHigh tileMap
             totalWidthPixels = tileSizePixels * tilesAcross
             totalHeightPixels = tileSizePixels * tilesHigh

Results

The finished article looks a bit like this:


or this:


and pressing the space bar cycles through the maps on account of this:

onKeyPress app (\x@(Key { eventKeyName = name,
                              eventKeyChar = char }) -> do
    case char of
      Just ' ' -> do
        putStrLn $ "Switching map"
        currentState <- readIORef mapStateRef
        nextState <- nextMapState currentState
        writeIORef mapStateRef nextState
        drawWin <- widgetGetDrawWindow canvas
        gc <- gcNew drawWin
        (width, height) <- drawableGetSize drawWin
        postGUIAsync $ drawDrawable drawWin gc (renderedMap nextState) 0 0 0 0 width height
      Just c -> putStrLn $ “Press ” ++ name ++ “(’” ++ [c] ++ “‘)”
      Nothing -> putStrLn $ “weird key: ” ++ name
    return (eventSent x))

Conclusions

I’m now a lot more comfortable with Haskell and think I’ve mastered the “word-for-word translation of imperative to pseudo-functional” style. The file parsing code I’ve written performs like a dog on account of that, but it’s not on a critical path for anything. One thing that has struck me is how easy it is to refactor Haskell code (as long as you use hanging ‘do’s to avoid formatting upsets). So I’ll stick at it and see if I can’t finish this project inside of a year.

Why not SDL?

FYYFDANSEIC, etc.

Un-overriding hashCode in Java

Problem: you are inheriting from a class in Java. This class has overridden Object.hashCode(). However, you do not want the behaviour of the overriding method, but rather that of the original implementation in the Object class.

Solution: in your new class, create a private field of type Object. Override the hashCode() method in the class to call hashCode() on that field. The result should follow a pattern similar to that below:

public class CustomHashSet<T> extends HashSet<T> {
    private Object hasher;

    public CustomSpecHashSet() {
        super();
        hasher = new Object();
    }

    @SuppressWarnings("unchecked")
    public synchronized Object clone() {
        CustomHashSet<T> copy = (CustomHashSet<T>)super.clone();
        copy.hasher = new Object();
        return copy;
    }

    public int hashCode() { return hasher.hashCode(); }

    public boolean equals(Object o) { return this == o; }
}

Discussion

In Java, once a method has been overridden, it is impossible to get at the original implementation from any inheriting classes. I recently had to deal with this when I wanted to keep some objects that inherited from various Collections inside HashSets, and wanted distinct objects in the set to not be treated as equals just because they happened to contain the same elements. In other words, I needed to use the Object implementations of the hashCode() and equals() methods, rather than the versions defined by the Collections. This is because Object.hashCode() returns a unique integer for every distinct object (typically by using the memory address of an object).

At first, I thought this problem could be solved with reflection. The soloution should simply involve getting a Method instance for Object.hashCode(), and invoke that on any object. Unfortunately, that does not work; calling a method using reflection in Java results in a dynamic method lookup based on the object the method is being invoked on.

Since Java does not permit un-overriding methods, one has to cheat. An object that is never leaked to the outside world can be used to provide a unique integer for each instance. However, care must be taken when implementing classes that support cloning (i.e. implement the Cloneable interface, or inherit from a class that does). Since the clone() method only does a shallow copy of an object, it is essential to override clone(), and make sure that any copies created there get their own instance of the hashing object. There is another gotcha; in the presense of multiple threads accessing the same object, the following clone() implementation would be unsafe:

public Object clone() {
    MyObject copy = (MyObject)super.clone();
    copy.hasher = new Object();
    return copy;
}

This is because the Java memory model only guarantees initialization safety for objects created using a constructor. What this means is that, for an object created using clone() and changing fields after the call to clone(), the updates to those fields are not guaranteed to be seen immediately by other threads accessing that object. A simple way to address this is to put the initialization of the cloned object inside a synchronized block, as in the code given at the top of this recipe.

GTK, Glade, Haskell and gnome_program_init()

So. I finished the book (at length, and to be fair I mostly skimmed the last chapters), which means it’s time for me to start actually writing Haskell code. I thought I’d start with a simple GUI app, but it turned out not to be quite so simple.

I’ve put the code that I’m working on up on GitHub because that’s what all the cool kids are doing. My project is called GP3 for reasons that ought eventually to become clear. HEAD at time of writing is 7b01940

Glade
Glade is a GTK UI designer. I won’t go in to a lot of detail - there’s been plenty written about it. What I will say, though, is that at version 3, you can often find yourself creating unexpected dependencies for your program by using the more complex widgets. I unwittingly picked something from the “GNOME User Interface” toolbox, which has a Glade class of “GnomeApp”. This includes a “BonoboDock” and a “BonoboDockItem”.

Launching your app
Borrowing heavily from the book, here’s part of the code I was using to launch my app:

main :: FilePath -> IO ()
main gladepath =
  do
    unsafeInitGUIForThreadedRTS
    timeoutAddFull (yield >> return True) priorityDefaultIdle 100
    gui <- loadGlade gladepath
    connectGui gui
    windowPresent (mainApp gui)
    mainGUI

gnome_program_init()
Having made the mistake of using a GNOME-UI widget, I saw this when I ran my app:

GnomeUI-ERROR **: You must call gnome_program_init()
          before creating a GnomeApp

This is because my Glade UI requires libgnomeui to be initialised. To make matters worse, libgnomeui isn’t linked by default in to Gtk2Hs. In my limited understanding of Haskell and Gnome, there are two options at this point. One is to import the gnome_program_init function from libgnomeui over FFI. The other is to write a C program to wrap a call to gnome_program_init and re-export a simpler function for you to import over FFI. I chose the latter option:

// gtk_docker.c
#include 

void do_gnome_init()
{
  static char **argv = NULL;
  if (argv == NULL)
  {
    argv = malloc(2);
    argv[0] = “gtk_docker”;
    argv[1] = ‘\0′;
  }
  gnome_init(”my-app”, “my-version”, 1, argv);
}

… and a header file

// gtk_docker.h
void do_gnome_init(void);

Astute observers will see that this is a bit of a cheat. GnomeUI wants the command line arguments that were passed to the executable. It would be possible, but irritating, to arrange this. I couldn’t easily divine how to pass an array of CStrings over FFI, so I wimped out. Also, I’m not technically calling gnome_program_init - this call appears to be deprecated in favour of gnome_init, and the latter call also silences the error message.

Compilation
We haven’t solved the compilation problem yet. The compiler still needs to know where to find gnome.h and its included headers, and needs to know where to find the associated libraries for linking. There are good ways and bad ways to solve this problem… here’s a bad way:

#Makefile
LDFLAGS = -lgnomeui-2 -lcairo -lglade-2.0
CFLAGS = -I/usr/include/libgnomeui-2.0 -I/usr/include/gtk-2.0/ \
              -I/usr/include/cairo/ -I/usr/include/glib-2.0/ \
              -I/usr/lib/glib-2.0/include/ \
              -I/usr/include/pango-1.0/ \
              -I/usr/lib/gtk-2.0/include/ \
              -I/usr/include/atk-1.0/ \
              -I/usr/include/libgnome-2.0/ \
              -I/usr/include/libbonobo-2.0/ \
              -I/usr/include/libgnomecanvas-2.0/ \
              -I/usr/include/libart-2.0/ \
              -I/usr/include/libbonoboui-2.0/ \
              -I/usr/include/gnome-vfs-2.0/ -Werror -Wall

GHCC=ghc

default: gp3

gp3: gtk_docker.o GP3Main.hs GP3GUI.hs
        $(GHCC) --make $(LDFLAGS) $^

clean:
  rm -f *.o *.hi GP3Main

The real answer probably involves GNU AutoTools for the C toolchain or some craziness with Cabal. I’m sure I’ll get round to that

Calling the function
Now to clear that error message. We just need to…

{-# LANGUAGE ForeignFunctionInterface #-}

foreign import ccall unsafe "gtk_docker.h do_gnome_init"
      c_gnome_init :: IO ()

and

main :: FilePath -> IO ()
main gladepath =
  do
    unsafeInitGUIForThreadedRTS
    c_gnome_init
    ...

and we’re done.

Portability
Linking libgnomeui probably makes my code a lot less portable. Hard-coding the include paths certainly does. Fortunately I don’t have to care about other users just yet, and I’m unlikely ever to care about other platforms

Haskell, GTK and Multi-Threading

I have been working on an application in Haskell, using Gtk2Hs for the user interface. Now, you normally want a graphical user interface (GUI) to be responsive, so you avoid doing tasks that take a long time in the thread that handles the GUI. Instead, your main computation happens in other application threads, and all that happens in the GUI thread is updating of interface elements. It turns out that there are several issues that come up when you mix Haskell, GTK and multiple threads, which is why this post is here.

Part 1: The GTK event loop

Most programs that use Gtk2Hs first do all the GUI-related initialization, and then execute the mainGUI computation. This is actually a loop that processes GTK-related events until the user quits the program. Because of the way thread switching works in Haskell, a thread executing a loop like that will not let any other “lightweight” threads run. (Lightweight threads are threads created using the forkIO computation.) One frequently suggested way to solve this problem is to make the GTK event loop periodically yield to any other Haskell threads by adding the following to your GUI initialization code:

timeoutAddFull (yield >> return True) priorityDefaultIdle 100

Note that this is only an issue when all lightweight Haskell threads run on top of a single operating system thread, as in the single-threaded RTS of GHC. If Haskell threads are allowed to run on multiple OS threads, then yielding is not necessary.

Part 2: GTK thread safety

It turns out that that GTK (the C library wrapped by Gtk2Hs) is not thread-safe. What this means is that all modifications of GTK state must happen from a single OS thread, which also must be the same thread that is executing the GTK event loop. If all Haskell threads are run on top of a single OS thread, this is easy to ensure. However, if you use a system where lightweight threads may be mapped to different OS threads (such as in the multi-threaded RTS in GHC), care must be taken when accessing a GUI. Essentially, application threads need to put GUI modification events onto a queue, with the thread that runs the GTK event loop processing the events from that queue. Fortunately, Gtk2Hs comes with such a queue built-in; you can use the postGUI functions to give the event loop blocks of IO to execute.

Note that, by default, Gtk2Hs will produce a warning when running under the GHC multi-threaded RTS. To get rid of this, use unsafeInitGUIForThreadedRTS instead of the usual initGUI to perform GTK initialization. The “unsafe” part of the computation name signifies that you are aware of the requirement to only modify GTK state from the correct OS thread.

Part 3: Deconstructing the GTK event loop

There might come a time when you want to do something more complicated than the above solutions allow. In such a case, you can actually substitute the mainGUI event loop with your own. Gtk2Hs provides functions that will process a single event at a time off the GTK event queue. Put these in a loop, sprinkle with your custom logic (such as pulling events off multiple queues), and you’re done!

Context

This was written with the GHC environment in mind. Other Haskell compilers and/or interpreters may differ in their implementations.

Java, SSL, and the Keystore of Doom

In a break from our Haskell programming…

I used to be a staunch defender of Java as a language. I still think it’s relatively good (though mostly for the tool support), but there are things about it that make me want to scream.

Imagine, for example, you’d like to make an SSL authenticated fetch from a webserver. You have a client certificate to authenticate your client, and a server certificate to authenticate the server, and you’ve generated them both from your own CA. Shouldn’t be that hard, right? Wrong Everything SSL has to be configured via the key stores, so you need to import your private certificate and the server’s public certificate in to your key store in order to make anything go.

There are two stores - the Trust Store and the Key Store. The Trust store contains the certificates you trust (CAs, etc.). The Key Store contains certificates for which you have the private key and against which you’ll encrypt challenges to verify your identity. All you have to do is populate them…

Step 1: Don’t use GCJ
There are a lot of great things to be said for the Open Source outlook on life. GCJ isn’t one of them. It works quite like Java, except when you try and run anything. Unfortunately it works sufficiently like Java that you don’t necessarily know you’re using it, and it’s installed as the default on a lot of Debian machines.

keytool error: java.lang.IllegalStateException: masked envelope

That was the first cryptic clue that I was using GCJ. Other clues are random GC messages on the console. Here’s a quick way to tell if you’re infected:

$ ls -l /etc/alternatives/ | grep -c java-gcj
24

The number you’re looking for is ‘0′ on a correctly configured system. Specifically you want to see:

$ chase `which keytool`
/usr/lib/jvm/java-1.5.0-sun-1.5.0.15/jre/bin/keytool
$ chase `which java`
/usr/lib/jvm/java-1.5.0-sun-1.5.0.15/jre/bin/java

If you’re not getting that, reconfigure the alternative:

# update-alternatives --config java

There are 7 alternatives which provide `java'.

  Selection    Alternative
-----------------------------------------------
          1    /etc/alternatives/kaffe-system/bin/java
          2    /usr/bin/gij-wrapper-4.0
*         3    /usr/lib/jvm/java-1.5.0-sun/jre/bin/java
          4    /usr/bin/gij-4.1
          5    /usr/bin/gij-4.3
 +        6    /usr/lib/jvm/java-gcj/jre/bin/java
          7    /usr/bin/gij-4.2

Step 2: Import the CA
Now we’re running the right JVM, it should be a simple matter of:

$ #Create a trust store with a CA Cert in it (teststore.jks doesn't yet exist)
$ keytool -import -v -trustcacerts -alias myalias -file cacert.pem -keystore teststore.jks
Enter keystore password:
keytool error: java.lang.NullPointerException

Oh. There may be a way to use blank passwords on keystores, but keytool ain’t it. Let’s try again with a password:

$ keytool -import -v -trustcacerts -alias myalias -file cacert.pem -keystore teststore.jks
Enter keystore password: password

Smashing. That’ll mean we can at least connect to the remote host. But the SSL handshake will still fail when the host sees our lack of client certificate.

main, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

Step 3: Import the Client certificate
My client certificate comes as two files - a certificate PEM file (the public part) and a key PEM file (the private part). Naïvely, I tried just installing the PEM part:

$ keytool -import -v -alias myalias2 -file signup1-cert.pem -keystore teststore.jks
Enter keystore password:  password
keytool error: java.lang.Exception: Input not an X.509 certificate

*sigh*. That’ll happen. Fortunately, we can convert from PEM to ‘DER’, which is something that keytool understands, using ‘openssl’:

$ openssl x509 -in signup1-cert.pem -inform PEM -out signup1-cert.der -outform DER
$ keytool -import -v -alias myalias2 -file signup1-cert.der -keystore teststore.jks
Enter keystore password:  password
Certificate was added to keystore
[Storing teststore.jks]

It’s stored, but unfortunately a) it doesn’t work and b) keytool thinks this is a ‘trustedCertEntry’ rather than a ‘keyEntry’:

$ keytool -v -list -keystore teststore.jks
...
Entry type: trustedCertEntry
...

Now, we can use ‘openssl’ to convert our certificate and key into a PKCS#12 combined key file:

$ openssl pkcs12 -export -in signup1-cert.pem -inkey signup1-key.pem -out signup1.p12

Even better, according to the documentation, PKCS#12 format files are valid key stores… unless you try and use them:

default context init failed: java.io.IOException: Invalid keystore format

Right. Let’s try that as a combined PEM format file then:

$ openssl pkcs12 -in mykey.p12 -out keystore.pem -nodes
$ keytool -import -v -alias clientcert -file keystore.pem -keystore keystore.jks
Enter keystore password:  password
keytool error: java.lang.Exception: Input not an X.509 certificate

True. (Incidentally, that’s No DES, not ‘nodes’) But we can convert PEM files to DER files

$ openssl x509 -in keystore.pem -inform PEM -out keystore.der -outform DER
$ keytool -import -v -alias clientcert -file keystore.der -keystore keystore.jks
Enter keystore password:  password

Step 4: Using the keystore in your program
You can configure the keystore at runtime as follows:

    System.setProperty("javax.net.ssl.keyStore", context.getRealPath(KEYSTORE));
    System.setProperty("javax.net.ssl.keyStorePassword", "password");
    System.setProperty("javax.net.ssl.trustStore", context.getRealPath(TRUSTSTORE));
    System.setProperty("javax.net.debug", "ssl");
    HttpClient httpClient = new HttpClient();
    GetMethod httpGet = new GetMethod("https://something.com");
    httpClient.executeMethod(httpGet);
    return new String(httpGet.getResponseBody());

javax.net.debug=ssl truly is a magic rune. I don’t know if you can get a list of such runes, but commit that one to memory. The debug output is pretty handy, if I little hard to follow.

Step 5: Become frustrated
What I didn’t mention, and perhaps should have mentioned above, is that I still hadn’t managed to import my key as a keyEntry, so this code still didn’t work. I downloaded the source code to the JDK and tried single-stepping KeyTool, but that also didn’t help. [Aside: People who create compressed archives without a top-level folder should be shot].

Step 6: Use KeyMan
KeyMan can be downloaded from IBM Alphaworks at time of writing:
http://www.alphaworks.ibm.com/tech/keyman/download
It ‘just works’. It lets you import your certificate and create a valid key store. Thanks IBM! (Source code plz).

More Haskell fun

“Define a tree type that has only one constructor, like our Java example. Instead of the Empty constructor, use the Maybe type to refer to a node’s children.”

Fair enough. How bout this?

data MaybeTree a = MaybeNode (Maybe (a (MaybeTree a) (MaybeTree a)))

It’s a real type. I can even make real values in the type:

maybeTree = MaybeNode (Just ("fish",
             (MaybeNode (Just ("left",
                               (MaybeNode Nothing),
                               (MaybeNode Nothing)))),
             (MaybeNode (Just ("right",
                               (MaybeNode Nothing),
                               (MaybeNode Nothing))))))

Buuut I can’t print them. Because I don’t derive Show. If I try to derive Show:

    No instance for (Show (a (MaybeTree a) (MaybeTree a)))
      arising from the 'deriving' clause of a data type declaration
                   at working.hs:(51,0)-(52,30)
    Possible fix:
      add an instance declaration for
      (Show (a (MaybeTree a) (MaybeTree a)))
    When deriving the instance for (Show (MaybeTree a))

That’s fair enough. And GHC’s even provided me with a hint. So I just… err… instance Show something, right? Wrong

Let’s write the show function for my tree type:

showTree (MaybeNode Nothing) = "empty"
showTree (MaybeNode (Just (a, b, c))) = "Node: " ++ (show a) ++
              ", Left: " ++ (showTree b) ++
              ", Right: " ++ (showTree c)

s’all good. But the type of that function?

showTree :: (Show t) => MaybeTree ((,,) t) -> [Char]

See that ((,,) t)? That’s the badness.

*Main> :kind (,,)
(,,) :: * -> * -> * -> *

Yeah. So it’s a tuple that has three type variables. Fun. Now I’m pretty sure that I ought to be able to define an instance of Show for my type, but I can’t for the life of me work out what the syntax is going to be

instance (Show a) => Show (MaybeTree ((,,) a)) where
       show t = showTree t

working.hs:58:0:
    Illegal instance declaration for `Show (MaybeTree ((,,) a))'
        (All instance types must be of the form (T a1 ... an)
         where a1 ... an are distinct type *variables*
         Use -XFlexibleInstances if you want to disable this.)
    In the instance declaration for `Show (MaybeTree ((,,) a))'

??? How about

instance (Show a) => Show (a, MaybeTree a, MaybeTree a) where
        show t = showTree t

working.hs:61:40:
    Kind mis-match
    Expected kind `* -> * -> *', but `a' has kind `*'
    In the type `MaybeTree a'
    In the type `(a, MaybeTree a, MaybeTree a)'
    In the type `(Show a) => Show (a, MaybeTree a, MaybeTree a)'

Any answers much appreciated, glorious lazyweb. I’m adding this to the list of exercises in the book that you can’t answer at the point you’ve reached in the book (this is chapter 4). Even after reading around about types and kinds and other peoples’ use of instance, I’m clueless.

And changing my type to be something sane doesn’t count. If it’s not possible to Show my type, I’d like to know why

Update:
After much discussion with sffubs and sos, it seems the only reasonable thing to do is to

{-# LANGUAGE FlexibleInstances #-}
instance (Show t) => Show (MaybeTree ((,,) t)) where
  show = showTree

We’re not quite sure what Flexible Instances are, but it seems that’s what’s needed to make this datatype work. The real answer is obviously not to use a crazy datatype:

data FooTree a = Maybe a ((FooTree a), (FooTree a)) deriving (Show)

Thanks both.

Getting started with Haskell… still

I can’t help but think there’s a bit of a gap in the market for introductory texts on Haskell. I say this in part because, at time of writing, if you google (hah! I’m using it as a verb! Trademark that!) “Getting Started Haskell”, you might end up here =/

I’d resolved to try getting started again on account of continuing to hear people rave about the language, so last night I did what I always do when learning something new - I googled “Getting Started X”. I found this awesome e-book / blog:

http://book.realworldhaskell.org/beta/index.html

It’s very well written (if a little rough round the edges - beta is the word), but I still think the learning curve presented is a _little_ steep for simpletons like myself. Bear with me while I expose my ignorance.

We’re using GHC. GHC is recommended by the book, it’s recommended by Don (who is indescribably leet: http://cgi.cse.unsw.edu.au/~dons/blog/), and it’s recommended by my n-sim colleagues (who mostly are, except for me: http://www.n-sim.com).

# apt-get install ghc6
# ghci
GHCi, version 6.8.2: http://www.haskell.org/ghc/  : ? for help
Loading package base ... linking ... done.
Prelude>

aaah. Prelude. Don’t I feel at home. In fact I don’t - this is all pretty weird, but working through the first couple of chapters of the book was fun. Walk with me a while…

Type Porn
If types don’t excite you, this probably isn’t the language for you. But they should; they’re awesome.

Prelude> :set +t
Prelude> 1337
1337
it :: Integer

In GHCI (interactive GHC interpreter), setting “+t” makes the interpreter print the type of whatever you’ve just evaluated. Technically, it prints the type of “it” - the special value in to which your last evaluated expression is loaded (there is no spoon, there are no variables). I know what “1337″ is, I know what “it” is, but what’s “Integer”?

Prelude> :info Integer
data Integer
  = GHC.Num.S# GHC.Prim.Int#
  | GHC.Num.J# GHC.Prim.Int# GHC.Prim.ByteArray#
        -- Defined in GHC.Num
instance Enum Integer -- Defined in GHC.Num
instance Eq Integer -- Defined in GHC.Num
instance Integral Integer -- Defined in GHC.Real
instance Num Integer -- Defined in GHC.Num
instance Ord Integer -- Defined in GHC.Num
instance Read Integer -- Defined in GHC.Read
instance Real Integer -- Defined in GHC.Real
instance Show Integer -- Defined in GHC.Num

ooo… fancy. What does that all mean? Well, I’m only on chapter 3, but in my simplistic Object Oriented view of the world, we’re effectively saying that Integer implements the interfaces Enum, Eq, Integral, Num, Org, Read, Real and Show (but the truth is a little more involved).

Prelude> :info Enum
class Enum a where
  succ :: a -> a
  pred :: a -> a
  toEnum :: Int -> a
  fromEnum :: a -> Int
  enumFrom :: a -> [a]
  enumFromThen :: a -> a -> [a]
  enumFromTo :: a -> a -> [a]
  enumFromThenTo :: a -> a -> a -> [a]
        — Defined in GHC.Enum
instance Enum Integer — Defined in GHC.Num
instance Enum Float — Defined in GHC.Float
instance Enum Double — Defined in GHC.Float
instance Enum Bool — Defined in GHC.Enum
instance Enum Ordering — Defined in GHC.Enum
instance Enum Char — Defined in GHC.Enum
instance Enum () — Defined in GHC.Enum
instance Enum Int — Defined in GHC.Enum

You kind of have to be comfortable with looking at types of the form:

a -> a -> a -> [a]

“enumFromThenTo” obviously takes three values and returns a list of values. (The joy of types, right? You know what it does by what its type is.) Moreover, we can see that it’s defined for the instances Integer, Float, Double, Bool, Ordering, Char, () and Int.

Prelude> enumFromThenTo 1 2 8
[1,2,3,4,5,6,7,8]
Prelude> enumFromThenTo () () ()
[(),(),(),(),(),()^CInterrupted.

and we can type functions too:

Prelude> :type fst
fst :: (a, b) -> a

What does that one do? There's only one thing it can do! Yes, I know, that's practically pornographic.

The exercise
"Write a function lastButOne, that returns the element before the last."

Trivial, right? A five year old could do it. Well, excuse me while I have a quick flashback to ML ticks (PDF) and rock gently back and forth in the corner. You have to bear in mind that, at this point in the book, we don't know there's a 'length' function in Prelude, we've not been taught pattern matching or case statements, and we're still simplistically minded imperative programmers. So naïvely, the best we might be able to do is:

-- in add.hs
count n [] = n
count n xs = count (n+1) (drop 1 xs)

myLastButOne xs = head (drop ((count 0 xs) - 2) xs)

Prelude> :load add.hs
[1 of 1] Compiling Main             ( add.hs, interpreted )
Ok, modules loaded: Main.
*Main> myLastButOne [1,2..10]
9

Repeat after me… “ewww”. And even to do that, I’ve had to use mysterious pattern matching which hasn’t been explained at that point in the book. Now, we could assume that a resourceful reader might find the ‘length’ function:

myLastButOne xs = head (drop ((length xs) - 2) xs)

But that’s still pretty unsatisfactory. For all I know, length is O(n) in the length of the list, so I’ll be going down the list twice. I daren’t imagine what Don would say. Even if it’s O(1), it doesn’t feel right. After a bit of head scratching and syntax guessing, I came to:

lastButOne (h:t) = case t of
                       (a:[]) -> h
                       (a:b) -> lastButOne t

which, for me at least, feels a little better. But I don’t know it’s right. I’m welcoming any pointers here. Now obviously for the “Find the last but nth item”, going down the list twice is looking less unattractive:

myLastButN n xs = head (drop ((length xs) - (n+1)) xs)

It’s still not great though. Would that I could list[-n]. But that’s not the point.

Summary
I’m determined to learn more Haskell and continue to expose my ignorance on this blog. Any pointers to good docs are welcome - “Haskell for simpletons”, that sort of thing. Meantimes I’ll continue to read the book. My stretch goal is to understand the things written on Don’s blog and on the following:

Conal Elliott:
http://conal.net/blog/

Kenn Knowles:
http://www.kennknowles.com/blog/

Removing bytes from a file

Morning,

I copied and pasted a (Ruby) script from a PDF this morning, and on executing it I got a whole pile of:

webservice.rb:33: Invalid char `\240' in expression
webservice.rb:33: Invalid char `\302' in expression

which was annoying. For reasons best known to KPDF (or oowriter, or my window manager’s cut-and-paste buffer), the spaces in the script (” “) had been encoded as 0xc2 0xa0, which is sort of UTF16 if you look at it sideways, but essentially useless to me.

So how do you remove 200 instances of a 2-byte sequence from a file? I didn’t have a good way, but this bad way sufficed:

cat > rm.c << EOF
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>

int main(int arcg, char *argv[])
{
  unsigned char c;

  while (read(0, &c, 1)==1)
  {
    if (c != 0xc2 && c != 0xa0)
    {
      write(1, &c, 1);
    }
    else if (c == 0xc2)
    {
      write(1, ” “, 1);
    }
  }
}
EOF
make rm
cat webservice.rb | ./rm > output.rb

So, my dearest lazyweb… better answers?