talkingCode

Simple webservice client, Ruby

Haven’t really got anything useful to write about, so here’s a simple bit of code to make XML requests to a webservice. It’s useful for me as a reference because it covers things I want to do fairly regularly - MD5-summing, Base64 encoding, fetching a page over HTTP and parsing and dumping an XML document.

For the sake of a complete example, the service we’re looking at here is a relatively RESTful directory service, exposing nested resources by extending the request URL:

# Root of service
http://some.service.com/api_root/

# List of locations
http://some.service.com/api_root/locations

# List of categories for location ID 4
http://some.service.com/api_root/categories/location/4

Requests can also have query arguments appended to specify, for example, numbers of results to return and sort order. Additionally, an authentication token and username are sent as query parameters, so that a complete request might look like:

http://some.service.com/api_root/categories/location/4?
    count=20&sort=name&uid=someuser&hash=5ju5eVirhXRqjdobToZiGA%3D%3D

The code, then, for our simple client is:

#!/usr/bin/ruby

require 'digest/md5'
require 'base64'
require 'cgi'
require 'net/http'
require 'uri'
require 'rexml/document'
require 'rexml/xpath'

BASE="http://some.service.com/api_root/"
UID="username"
PASS="suitable_password"

# Generates a valid authentication token based on 'PASS' and the
# current timestamp
def token
  plaintext = Time.now().to_i.to_s + '.' + PASS
  md5 = Digest::MD5.digest(plaintext)
  return Base64.encode64(md5).strip
end

# Returns a valid service URL, including authentication tokens
def url_for(method, args, queryargs = Hash.new)
  queryargs['uid'] = UID
  queryargs['hash'] = token
  escaped_query_parts = queryargs.collect do |entry|
    entry.collect { |e| CGI.escape(e) }.join(”=”)
  end
  escaped_args = args.unshift(method).collect { |a| CGI.escape(a) }
  path = escaped_args.join(”/”) + “?” + escaped_query_parts.join(’&')
  return BASE + path
end

# Fetches an XML document from the supplied URL
def fetch_xml(url)
  xml_string = Net::HTTP.get(URI.parse(url))
  if !xml_string
    puts “Request failed”
    exit
  end
  doc = REXML::Document.new xml_string
end

# Dumps out the ‘name’ and ‘url’ attributes for a nodelist
def dump_name_attributes(doc, path)
  REXML::XPath.each(doc, path) do |node|
    puts attribute_value(node, ‘@name’) +” (”+ attribute_value(node, ‘@url’) +”)”
  end
end

# Fetches the value of the attribute with the supplied name, or nil
def attribute_value(node, path)
  attribute = REXML::XPath.first(node, path)
  if !attribute
    return nil
  end
  return attribute.value
end

… and we might make a request as follows:

puts "Category List:"
xml = fetch_xml(url_for("categories", ["location","4"],
               { “count” => “20″,
                  “sort” => “name” }
        ))
dump_name_attributes(xml, ‘xmlservice/categories/category’)

assuming that the returned XML looks a little like this:

<xmlservice>
  <categories>
    <category name="Food" url="/api_root/category/food"/>
    <category name="Drink" url="/api_root/category/drink"/>
    <category name="Art" url="/api_root/category/art"/>
  </categories>
</xmlservice>

Reading the Economist - Hpricot, Ruby-RSS, Festival

Well, having the Economist read at any rate.

First, set up Festival (configuring it to use ALSA and an ‘English’ voice):

apt-get install festival
apt-get install festvox-rablpc16k
cat > ~/.festivalrc <<END
(Parameter.set 'Audio_Command "aplay -D plug:dmix -q -c 1 -t raw -f s16 -r \$SR \$FILE")
(Parameter.set 'Audio_Method 'Audio_Command)
(voice_rab_diphone)
END

Then liberally sprinkle some ruby:

#!/usr/bin/ruby

require 'rss/1.0'
require 'rss/2.0'
require 'open-uri'
require 'yaml'
require 'hpricot'
include YAML

TEMPFILE = "/tmp/economistreader"
puts "Fetching feed"
source = "http://www.economist.com/rss/full_print_edition_rss.xml"
content = ""
open(source) do |s| content = s.read end
rss = RSS::Parser.parse(content, false)

puts "Title: #{rss.channel.title}"
puts "Found #{rss.items.size} items"
for item in rss.items
  puts "#{item.title}"
  puts "Read? [Y/n]”
  if readline.strip.downcase =~ /^n/
    next
  end
  doc = Hpricot(open(item.link))
  paras = doc.search(”//div[@class='col-left']/p[@class='']“)
  File.open(”#{TEMPFILE}.body”, “w”) do |f|
    paras.each do |p|
      f.write(p.inner_text + “\n”)
      puts p.inner_text
    end
  end
  system(”festival”, “–tts”, “#{TEMPFILE}.body”)
end

I give it about 3 articles before the voice drives me completely insane. There’s a character-set issue that puts ‘?’s in odd places and causes Festival to get confused. Even without confusing characters, free text-to-speech software still isn’t ‘all that‘.
You could also, it’s worth pointing out, visit PimpMyNews. You’ll find the Economist’s feed under ‘Business/World Business News’. Unfortunately, they are lazy and their software only reads out the text from the RSS ‘Description’ field rather than parsing the whole article. That said, if what you want is to hear the first 200 words of every article in the Economist, that’s your badger.

Import into MySQL from CSV/Excel, Ruby

You have some data in a CSV file (or a spreadsheet that you’ve dumped to CSV) that you’d like to load into a MySQL database. Nothing very interesting to say about this except that faffing looking up both sets of docs is tedious for what are quite simple bits of code and a fairly occasional task. (If you spend a lot of your time dealing with CSV / Excel files, you’ve probably made some bad decisions in life, but at least you’ll know this by heart )

The source file is a list of resistor values - column headings ‘Value’ (read product code), ‘Resistor A’, ‘Resistor B’, ‘Resistor C’. The CSV looks something like:

0,1.1,27,180
7,4.7,22,47
21,2.2,18,4.7

The table looks something like:

CREATE TABLE resistor_configs (
  id INT AUTO_INCREMENT,
  value INT,
  resistor_a FLOAT,
  resistor_b FLOAT,
  resistor_c FLOAT,
  PRIMARY KEY  (id)
) DEFAULT CHARSET=utf8

…and the Ruby runs as follows:

#!/usr/bin/ruby
#
require 'csv'
require 'mysql'

my = Mysql::init()
# You can do any SSL stuff before the real_connect
# args: hostname, username, password, database
my.real_connect("localhost", "root", "", "products_development")

my.query("DELETE FROM resistor_configs")

CSV.open('/tmp/resistors.csv', 'r') do |row|
  # No escaping here, because I trust the input file. You may not
  my.query("INSERT INTO resistor_configs" +
               "(value, resistor_a, resistor_b, resistor_c)" +
               "VALUES (#{row[0]}, #{row[1]},” +
               “#{row[2]}, #{row[3]})”)
end

There you have it. Nothing very clever, but easier to copy and paste than to read the fine manual.

Byte-packing / reading binary data in Ruby

A lot of high-level languages don’t have wonderful native support for bit-level manipulation of data, so when you find yourself having to implement a proprietary wire protocol or parse a custom file header, you often feel a little lost. Fortunately you’re not the first person to feel lost, and some kind person has, by and large, already solved the problem for you. In Java, for example, there’s the fine commons.io library (in fact, all the commons libraries are pretty fine), and in Ruby you have BitStruct:

# Syntax here is
#    parsed-datatype
#    :symbol_name for the parsed data
#    field size in bits
#    comment
class BinaryHeader < BitStruct
  default_options :endian => :native
  char     :id, 4*8, "File ID"
  char     :format_string, 4*8, "Format String"
  unsigned :remainder, 32, "Remaining bytes"
  unsigned :trackId, 32, "Track ID"
  unsigned :formatId, 32, "Format ID"
  unsigned :codecId, 32, "Codec ID"
  unsigned :major, 16, "Major Version"
  unsigned :minor, 16, "Minor Version"
  unsigned :validation, 32, "Validation Data"
  unsigned :size, 32, "File Size"
end

There are a number of ways you can find binary data on your hands. You could read directly from a file / socket, you could generate some, or you could receive it Base64 encoded from a Webservices request:

def get_header(file)
  header_bytes_base64 = @webservice.getFileHeader(file.trackId)
  header_bytes = Base64.decode64(header_bytes_base64)
  if header_bytes.size != HEADER_SIZE
    puts "Invalid header size: #{header_bytes.size}"
    return nil
  end
  header = BinaryHeader.new(header_bytes)
  puts "Got Header:"
  puts header.inspect_detailed
  if header.id != "FILE"
    puts "Unknown file ID: #{id}"
    return nil
  end
  if header.format_string != "MP3X"
    puts "Unknown file format: #{format}"
    return nil
  end
  return header
end

Problem solved.

JSON “RPC” from Perl to Ruby / Webrick

So I have a bunch of software I’d written in Ruby because it was “The Right Choice™” and I need to make it talk to a stack of software I’d written in Perl (because “I Was An Idiot™”). Specifically, I need the Perl to be able to call the Ruby. I had a quick dig around, and there’s a Perl module Inline::Ruby which, on the face of it, would do the job. Unfortunately, Inline::Ruby is version 0.0.2 software and “Doesn’t Work So Good™”. Not only that, but I’d really like some persistence in the Ruby code so that I don’t have to new up the state every time I make a call. What this calls for, then, is some IPC

Linux IPC comes in three varieties - Sockets, Files, and Shared Memory. Files is obviously a pretty poor idea in that you’ll either be polling a lot or writing nasty dnotify stuff. Shared Memory is okay but extremely unportable. Sockets are a pretty good idea, and if you choose an IP socket you get the advantage that you can run the communicating processes on different machines (assuming they don’t need to share other local resources).

So you’ve selected TCP/IP as a transport, but you’ve then got all sorts of irritating high-level protocol implementation to do. Unless…

#!/usr/bin/ruby
# A Simple Webserver for JSON RPC

require 'webrick'
require 'json'
require 'yaml'
include WEBrick
include YAML

server = HTTPServer.new(:Port => 8000)
server.mount_proc("/rhapsodise") do |request, response|
  response['Content-Type'] = “application/json”
  response.body = handle_json_request(request)
end

trap(”INT”) { server.shutdown }

def handle_json_request(request)
  object = JSON.parse(request.body)
  YAML.dump(object)
end

server.start

and then…

#!/usr/bin/perl -w
#
# A Simple Perl Client for JSON RPC

use strict;
use JSON;
use LWP;

my $actionurl = "http://localhost:8000/rhapsodise";

my $ua = LWP::UserAgent->new();
$ua->agent("JSONClient/0.1");

my $object = { test => "fish",
               wibble => ["meep", "flange" ] };

my $json = objToJson($object);
print “$json\n”;
my $req = HTTP::Request->new(POST => $actionurl);
$req->content_type(’application/json’);
$req->content($json);

my $res = $ua->request($req);
if ($res->is_success)
{
  print “Succeeded:\n” . $res->content;
  my $result = jsonToObj($res->content);
}
else
{
  print “Failed\n”;
}

… et voila. For less code than it might cost you to bind the socket in C(!) you’ve got a nice, portable way of making Ruby calls from your Perl through the Webrick webserver. Even better, you get to use Perl and JSON, as exhorted in my previous post. A dash more code for the server, and it’s very nearly useful:

def error_object(message)
  return { :error => message }.to_json
end

def handle_json_request(request)
  command = JSON.parse(request.body)
  if command["method"] == nil
    return error_object(”No method suppled”)
  end
  if command["method"] =~ /^rd_/
    return self.send(command["method"], command)
  end
  return error_object(”No matching method”)
end

def method_missing(m, *args)
  return error_object(”Invalid command: #{m}”)
end

Notice that I’m prefixing my RPC-able methods with ‘rd_’. This probably isn’t much more secure than not bothering, but is a useful kind of Hungarian Notation for the methods. I’m not worrying too much about security on this one since I write both ends and I trust the link across which the packets run - you’d need to take appropriate precautions if that wasn’t true.