talkingCode

Buzzword Bingo – Tomcat, Jetty, Cactus, Derby, Velocity, Maven

I’ve been writing a webapp at work recently. Because I’m writing it in Java, the scope for me to post interesting articles about what I’m doing has been pretty limited – I don’t know that much about enterprisey Java, and I spend most of my time bashing my head against a Java-shaped wall. A quick overview is probably worth doing though.

Apache
If you’ve not heard of the Apache Foundation (outside the context of their fine webserver), go visit their site now. Amongst other things, Apache are creating an incredible resource on that site in the form of a huge code commons. If you’re not sure what a commons is, a) shame on you and b) read this fine book (in fact, read it even if you are sure). The majority of the code on the Apache site is Java, so if you’re developing an application in Java it’s well worth checking the links on the left-hand side of the main page to see if they’ve already written what you’re writing, or something that will help you. The code is all Apache-licensed – a BSD-style license that is amenable to reuse in commercial projects as well as Free Software projects. It’s also very high quality code. You basically can’t lose.

strace is your friend
One of the biggest irritations, I’ve found, in using the Apache Java projects (and this is more of a function of Java than Apache) is trying to make Java find the appropriate config files. It’s pretty difficult to tell which config files Java is seeing, if any. You can download the source to the component you’re using and single step it in Eclipse, but this tends to get tedious at about the point you hit the class loader, which is exactly the point at which it’ll do any resource location. What you can do, though, is:

strace -f -p[YOUR_PID] 2>&1 | grep "your_config_file.xml"

You’re looking for things like calls to ‘stat’ or results of ‘ENOENT’. This will generally tell you what Java’s looking for and where. That is, I grant you, a sledgehammer-nut solution, but I’ve found it quicker than anything else. If you know a better way to work out where I should be putting my Velocity toolbox.xml in my Maven tree for the Webapp to run in Jetty under Cactus please let me know. It’s not in the FAQ.

My environment
What am I doing with all these projects?

• Maven – Your lifecycle management tool (build and dependency management). Maven2 is much improved over Maven for having the package repositories work 99% of the time rather than the 50% that was more common in the original. The whole POM thing is… errr… a little obtuse, I grant you, but you can use Maven in a dumb way without too much trouble. The Eclipse plugin is a great way to find the packages you need too.
• Tomcat – The application server. Can’t really do a Webapp without one of those. Tomcat isn’t the only server out there, but it’s pretty well used and, as at 5.5-ish, not too difficult to make go.
• Velocity – The templating engine. The web is awash with webapp ‘frameworks’; Apache have about four of them for starters. Sometimes, though, you’re not writing for web browsers. Sometimes, you just want a way to turn your POJOs into arbitrary markup. Velocity is fast, uncomplicated and fully featured.
• Cactus – The Servlet test framework. I’ve still not completely wrapped my head around Cactus, but basically for each of your unit tests you have setUp and tearDown to manipulate state on the server, and begin and end to manipulate state on the client, and Cactus will run web requests inside a test harness for you. Trés handy.
• Jetty – A lightweight servlet container. Controversially not an Apache project, Jetty is really quite handy for running your Cactus tests. Instead of pushing the WAR all the way to Tomcat, you can new up a Jetty server inside the Maven test task and run your functional tests there.
• Derby – The lightweight database. To avoid having to have a real database available in order to run your unit tests, I find it convenient to put some test fixtures in a Derby DB and run the servlet off the back of that. This plays really nicely with the Cactus / Jetty setup and has the added advantage that (unlike MySQL) if you try and create a prepared statement that your schema can’t possibly run, Derby will warn you about that at preparation time rather than execution time. With a bit of tweaking, then, you can statically check your statements against the schema.

Making all that play together has been nightmarish, either because of my relative inexperience or because it’s genuinely difficult. I’ve embarrassingly failed to write down all the errors I encountered and how to fix them, but I may cover some of the more tricky parts of the setup in future articles. For all the setup work, though, being able to type mvn install and feel confident that your latest refactoring hasn’t broken the complex application you’ve written is worth almost any amount of blood, sweat and tears.

Simple webservice client, Ruby

Haven’t really got anything useful to write about, so here’s a simple bit of code to make XML requests to a webservice. It’s useful for me as a reference because it covers things I want to do fairly regularly – MD5-summing, Base64 encoding, fetching a page over HTTP and parsing and dumping an XML document.

For the sake of a complete example, the service we’re looking at here is a relatively RESTful directory service, exposing nested resources by extending the request URL:

# Root of service

http://some.service.com/api_root/

# List of locations

http://some.service.com/api_root/locations

# List of categories for location ID 4

http://some.service.com/api_root/categories/location/4

Requests can also have query arguments appended to specify, for example, numbers of results to return and sort order. Additionally, an authentication token and username are sent as query parameters, so that a complete request might look like:

http://some.service.com/api_root/categories/location/4?

    count=20&sort=name&uid=someuser&hash=5ju5eVirhXRqjdobToZiGA%3D%3D

The code, then, for our simple client is:

#!/usr/bin/ruby

require 'digest/md5'
require 'base64'
require 'cgi'
require 'net/http'
require 'uri'
require 'rexml/document'
require 'rexml/xpath'

BASE="http://some.service.com/api_root/"
UID="username"
PASS="suitable_password"

# Generates a valid authentication token based on 'PASS' and the
# current timestamp
def token
  plaintext = Time.now().to_i.to_s + '.' + PASS
  md5 = Digest::MD5.digest(plaintext)
  return Base64.encode64(md5).strip
end

# Returns a valid service URL, including authentication tokens
def url_for(method, args, queryargs = Hash.new)
  queryargs['uid'] = UID
  queryargs['hash'] = token
  escaped_query_parts = queryargs.collect do |entry|
    entry.collect { |e| CGI.escape(e) }.join("=")
  end
  escaped_args = args.unshift(method).collect { |a| CGI.escape(a) }
  path = escaped_args.join("/") + "?" + escaped_query_parts.join('&')
  return BASE + path
end

# Fetches an XML document from the supplied URL
def fetch_xml(url)
  xml_string = Net::HTTP.get(URI.parse(url))
  if !xml_string
    puts "Request failed"
    exit
  end
  doc = REXML::Document.new xml_string
end

# Dumps out the 'name' and 'url' attributes for a nodelist
def dump_name_attributes(doc, path)
  REXML::XPath.each(doc, path) do |node|
    puts attribute_value(node, '@name') +" ("+ attribute_value(node, '@url') +")"
  end
end

# Fetches the value of the attribute with the supplied name, or nil
def attribute_value(node, path)
  attribute = REXML::XPath.first(node, path)
  if !attribute
    return nil
  end
  return attribute.value
end

… and we might make a request as follows:

puts "Category List:"
xml = fetch_xml(url_for("categories", ["location","4"],
               { "count" => "20",
                  "sort" => "name" }
        ))
dump_name_attributes(xml, 'xmlservice/categories/category')

assuming that the returned XML looks a little like this:

<xmlservice>
  <categories>
    <category name="Food" url="/api_root/category/food"/>
    <category name="Drink" url="/api_root/category/drink"/>
    <category name="Art" url="/api_root/category/art"/>
  </categories>
</xmlservice>

Building your own USB pendrive Linux image

I don’t know about you, but whenever I leave the house I have the same six items in my pockets (in order of certainty with which I believe I’m going to need them) – keys, wallet, phone, handkerchief, penknife and, as of the end of December, a 1GB USB stick with a basic Debian Live system on it.

Before December, had I been asked at a party “Exactly how geeky are you?”, I’d have meekly to have replied “Fairly.”. Henceforth, though, I shall be able to reach in to my pocket and say “Well, if you have a computer that can boot from USB mass storage and that’s configured to do so in the BIOS, I can show you…”. Of course, having such a device won’t just make you popular with the ladies – there’s a chance it’ll also have practical applications. Without further ado then…

First thing you’ll need is (at time of writing) a Debian Lenny (or more recent) install. Real men run unstable, but if you’re sitting there with Etch (running Firefox 0.0.1a on your Linux 2.0.34 machine) you can easily enough conjure up a suitable chroot with the standard:

debootstrap lenny chroot
chroot chroot

(There is, I should warn, some scope for confusion when you attempt the following inside a chroot, because the scripts create their own chroot)

Next thing to do is to install the helper packages:

apt-get install live-helper

We’re going to build the image in a fresh folder somewhere. You’ll want to pick a filesystem with plenty of disk space (~5GB) just to be on the safe side. In the fresh folder, typing lh_config will create the config directory for the image building scripts:

mkdir live
cd live
lh_config

We want to tweak some of the config files – it’d be pretty dull to go to this effort just to build a vanilla image. Below I’ve listed the changes I made to build my ‘dream’ install. The variables listed already exist in the files – you just need to change them to match what’s below:

# In 'config/binary'
LH_BINARY_IMAGES="usb-hdd"
LH_BOOTAPPEND_LIVE="locale=en_GB.UTF-8 keyb=uk"

# In 'config/bootstrap'
LH_SECTIONS="main contrib non-free"

# In 'config/chroot' (you won't want the linebreaks)
LH_PACKAGES="dns2tcp doc-base dsniff ettercap ettercap-common
less vim wireless-tools iceweasel icedove gaim curl openssh-client
openssh-server irssi centerim hexdump iproute iptables nemesis nmap
ntop privoxy socat tcpdump tor wireshark wireshark-common
firmware-ipw3945 ipw3945d ipw3945-modules-2.6-486 ncftp telnet
netcat fluxbox eterm xserver-xorg xfonts-base"

The idea with those package selections is a) to make you dangerous and b) to enable you to punch a hole through to the internet wherever possible. Once online, you can obviously install whatever else you need.

You’ll also want to:

mkdir -p config/chroot_local-includes/etc/skel/
echo fluxbox > config/chroot_local-includes/etc/skel/.xinit

This image boots to console, but typing startx will give you an X session (assuming it can detect and load the correct graphics driver).
To build the image, simply type

lh_build

Your completed image will appear as binary.img in that folder (weighing in at around 235Mb). In the event that something screws up, it’s important to understand that the state for the build process is tied up in the .stage folder. Until you notice that hiding there, the whole system can seem a bit mysterious.

All that remains is to transfer the image to your pendrive.
CAUTION: This next instruction is a dd onto a block device. My pendrive is /dev/sdb, but for all I know on your system that’s the SATA hard disk containing the only copy of your doctoral thesis.

dd if=binary.img of=/dev/sdb

You’re done. You should be able to reboot into that (assuming your BIOS supports it and is correctly configured). Couple of extra things to point out. The dd copies across a boot sector and the first partition. You can use fdisk to create further partitions on the device if you want a little read-write space. Specifically, if you type live persistent at the SysLinux prompt:

live-initramfs will look for persistent and snapshot partitions or files labeled “live-rw”, “home-rw”, and files called “live-sn*”, “home-sn*” and will try to, in order: mount as /cow the first, mount the second in /home, and just copy the contents of the latter in appropriate locations (snapshots). Snapshots will be tried to be updated on reboot/shutdown. Look at live-snapshot(1) for more informations.

More details in the manpage. To label a partition, you’re looking for:

mke2fs -L your_label /dev/sdbX

after you’ve created it in fdisk

The astute among you may spot the live-magic package. This is a GUI front-end to the process I’ve just described, but not something I’ve actually tried to use.

Long time no see

As is my wont, I took the last half of December as holiday (having carefully saved it up during the year) and got back in to work today, hence the slight drop in post frequency.

Of Blogging
If my first month of blogging taught me anything, it’s that it’s remarkably time consuming to write technical articles for a technical audience. Turning ‘that thing that I did at work today’ into an informative, accurate, and non-trivial (not to mention non-criminal) article takes more time than expected. For that reason I think I’m going to be dropping to two posts a week (if that) instead of three.
That’s the downside. The upside is that my digital log-book has actually served most / all of its intended purposes. I’ve used it for my own personal reference on more than one occasion and others have used it to help them get things done. Not only that, but I am now more intimately familiar with the seedy underbelly of the blogosphere – the SEO and the carefully selected topics titles and stubs. Analytics tells me I’ve had 540 unique visitors (which is more than I was expecting), the vast majority of whom are referred by Google search results (75%).
All in all, a successful little experiment and one I plan to continue for a while yet.

Of Holiday
I use the word ‘Holiday’ in the loosest possible sense. I mostly stayed in Cambridge pretending that Internet Radio didn’t exist, which was indescribably refreshing. I also, naturally, played a little with my computer. I have discovered that:

• Compiz now works in Sid (on Intel cards)
• If you do use Compiz, you can really only play videos with the GL renderers, and they play really badly with … err…. everything else on screen (which is obviously also GL). Also, don’t get too attached to being able to switch to console without crashing X.
• Debian has handy tools for building pendrive linux images (on which topic an article shortly)
• Xen won’t easily boot XP as an HVM guest from your pre-installed XP partition (on the same disk as your root partition). Steve said it would be fine, until I disclosed where the XP install was. Then he laughed. Then he said it was unsupported. Then, when pushed, said it was possible but ‘a really dumb idea’. He repeated that a couple of times. Then I did it anyway. Then he spent the best part of an afternoon helping me make it work. Then we gave up. I should listen to Steve.
• VMWare won’t boot XP from your pre-installed XP partition (or at least not without a Windows MBR, which I’ve not yet had a chance to copy from somewhere).
• `xset -dpms; xset s off` will turn off your screensaver, no matter how obtuse KDE is being.
• Epigenetics is fascinating. So is genetics, but I’d already heard of that. Props, as ever, to Radio 4 (and its fans)
• You really need a Raven login to access the University’s teaching materials
• Steve has a Raven login (mine’s apparently still ‘in the post’)
• If you want to play chess over the internet, you probably want to sign up at FreeChess and install pychess
• You can write your own BBC iPlayer (not sure if I’ll write an article about this), but you still can’t save the files
• I’ve forgotten how to play chess
• Sake is good

All in all a successful, relaxing and informative holiday.

Of the future
This year, apart from my New Year’s resolution to ‘Give people a break. They’re really not as bad as you make out’, I will be doing my best to sabotage the following organisations:

• Big Media – They seem to be doing a pretty good job on their own, to be fair. I was at home briefly this Christmas and for the first time in ages I saw an anti-piracy advert when I put a DVD in my parents’ player. Now don’t get me wrong, I have quite a lot of DVDs, I just don’t see the ad’s because playback software in Linux isn’t hideously crippled. If I had to watch those things, I’d certainly steal more films and buy fewer.
  I also had, as a result, to try to convince my father than in spite of what’s written in 36-point block caps on the screen, piracy is not theft. Piracy is copyright infringement; theft is theft. (Apparently the law disagrees).
• Apple – A lot of the cleverest people I know buy Apple. They don’t see that Apple is doing anything worse than your run-of-the-mill tech company and like the innovative products the company creates. If Apple are so nice, show me the offical documentation for DAAP. In fact, show me the docs for the Apple Accessory Protocol so that I can plug my iPod (I don’t own an iPod) into something that Apple didn’t make (I know such devices are available, but they all had to use the reverse engineered protocol). Apple are creating an ecosystem of devices with which you have no guarantees of interoperation, and just because the devices work and are shiny doesn’t make it good.
• The ISPs – ISPs are a necessary evil. That’s pretty much all I have to say on that.
• Adobe – When I tried to write my own BBC iPlayer, I discovered that there’s not a single place in the free world that you can find an implementation of RTMP (the Flash video streaming protocol). I found it hard to believe that there were still protocols that were secure by their obscurity, but there it is. It’s everywhere, streaming almost everything, and the only way you can view the streams is using an officially-sanctioned Flash player. Something must be done.

That’s part of the future. I’ll obviously be doing the turncoat thing of going to work everyday for what is effectively a media company, implementing existing DRM schemes, creating new ones of my own, buying DVDs, watching Flash videos and using the internet. But deep in my soul, I’ll be hating every minute of it