Ben Humphreys

Someone disconnects completely from technology for 90 days....

Wed, 08 Feb 2012 08:40:37 +0900

Someone disconnects completely from technology for 90 days. I’d love to be able to do this, but it’s kind of impossible.

I haven’t been drunk in 3 years... and I’ve been partying way more than you.

Mon, 06 Feb 2012 19:44:46 +0900

I haven’t been drunk in 3 years... and I’ve been partying way more than you.

Fast Ruby Parsers

Wed, 01 Feb 2012 14:38:44 +0900

Fast Ruby Parsers:

This thread mentions a bunch of Ruby parsers, with the focus on being fast for small languages. This is more for my own future reference than anything, it’s a great summary.

Parsing Wiki Text (Part 2)

Wed, 01 Feb 2012 11:03:56 +0900

More on parsing Wikipedia markup. So you’ve abandoned the idea of using regular expressions to parse the markup, congratulations. This should save yourself from insanity and oblivion.

mwlib is a Python library for parsing Wiki markup. It seems semi-official. Whatever, it’s what we’re going to use. It can parse the zipped XML dumps as I’ve shown in previous tutorials. However you have to provide it with an ariticle name, and today we’re feeling more John Wayne.

How can we feed mwlib raw Wiki markup and get the same hierarchical deliciousness as output? Here’s how:

A more involved example:

Dealing with Unicode CSV files in Python

Tue, 31 Jan 2012 17:04:08 +0900

Today I found out that Python’s csv module cannot read UTF-8 files (see the warning at the top of the page).

There are two Stackoverflow questions on the topic (here and here) but the solutions don’t seem to work for Python 2.6.

There code examples on the official page borrow from StackOverflow (or the other way around), and don’t seem to help much.

The more I learn about Python the more I appreciate Ruby.

Video

Sun, 29 Jan 2012 17:01:01 +0900

Star Wars Uncut: Director’s Cut: Wonderfully cute and...

Thu, 26 Jan 2012 20:42:00 +0900

Star Wars Uncut: Director’s Cut: Wonderfully cute and funny clips by hundreds of fans, stuck together to make the entire first film (via TokyoJoe).

Hand-made outfits, stop-motion segments, entire clips rendered on 80’s Macintoshes, Star Trek uniforms. Also My Little Pony at 0:37.

Learn Languages Through Games on Steam

Thu, 26 Jan 2012 20:36:50 +0900

If you like playing PC games, and want to learn a language at the same time, try this. Set your Steam language to the foreign language you’re trying to learn, and you can play most of your library in that language. It’s under Preferences > Interface then choose your preferred language from the first drop-down.

Sometimes Steam will need to re-download the data to get the localised content, but once that’s all done, you should be able to play without any trouble. For example, both Portal games, Civilisation, Team Fortress 2 and Skyrim all work in French. Check the languages section of Steam store page for that game for what languages are available.

The only problem with learning from games is you might learn a lot of weird vocabulary. Just think of it as gaming with a bonus, rather than actual language practice.

Parsing Edict XML with Perl and XML::LibXML

Wed, 25 Jan 2012 15:59:46 +0900

Edict is a Japanese-English dictionary that is free to use for research (as far as I know). It’s available in a few formats, the most useful of which is XML dump of English-only data.

It might help someone sometime, so I’ve posted a short Perl snippet of how to parse the format.

A single entry looks like:

Print Wiki Parse Tree with `mwlib`

Fri, 20 Jan 2012 09:40:53 +0900

It’s hard to see how mwlib treats Wiki markup. You can use this small snippet to output the tree of an article and see what you’re dealing with. It’s recursive so if you have a Huge document it’s possible you’ll get some problems, although I haven’t hit any yet.

Output is something like:

Section
  Node
    Text
  Node
    Paragraph
      Text
    Paragraph
      Style
        Node
          Node
            ArticleLink
            Text
            ArticleLink
            Text
...

There’s probably a function within mwlib for doing this, but I couldn’t find it.

Python Regex Global Replace

Wed, 18 Jan 2012 18:40:55 +0900

Today’s fun Python gotcha: I couldn’t find out how to do a global regular expression replace. In Perl it would look like:

$my_str =~ s/foo/bar/g;

In Python it’s:

import re
re.sub('foo', 'bar', my_text)

It turns out I wasn’t reading the sub() documentation properly. The amazing @puresock saved the day and pointed out this from the documentation:

The optional argument count is the maximum number of pattern occurrences to be replaced; count must be a non-negative integer. If omitted or zero, all occurrences will be replaced. Empty matches for the pattern are replaced only when not adjacent to a previous match, so sub('x*', '-', 'abc') returns '-a-b-c-'.

tl;dr Python’s regex-replace function sub() defaults to global. So you don’t need a global flag.

Kamogawa river, Kyoto

Tue, 17 Jan 2012 12:55:15 +0900

Kamogawa river, Kyoto

Parsing Wiki Text (Part 1)

Mon, 16 Jan 2012 17:56:00 +0900

I’ll be dealing with Wiki text a lot in the coming months, so I thought I’d post what I learn, as I learn it.

This is just a quick introduction on what I did today pieced together from mailing list posts and various other posts online.

Wikimedia Markup

First some points on the markup used on Wikipedia. There seems to be a few ways to call this syntax, wikitext, wikimedia markup, wiki something. By some weird definitions it’s not a real grammar, merely something that can be translated into HTML easily.

From what I’ve read online, parsing it naïvely using regular expressions is not a great idea. There are too many edge cases, and weird ways in which things can be written for it to be viable. However I have a feeling that using regular expressions only would be a lot faster than the grammar-like parsers that are available. Maybe I’ll do a short benchmark another time.

For now, rather than roll my own, I decided to use the Python library mwlib to parse the Wiki markup.

Install

Let’s install mwlib then. Unfortunately the documentation for mwlib is somewhere between non-existent and useless.

If you’re on a shared setup, I found the easiest way to install is using virtualenv.py:

$ curl -O https://raw.github.com/pypa/virtualenv/master/virtualenv.py
$ python virtualenv.py wiki
$ . wiki/bin/activate
(wiki)$ pip install mwlib

Setup

You then need to download the dump(s) of the data you want to deal with. Find it from the full list of dumps available. It’s likely you’ll just want the Articles, but refer to the documentation for more details of what’s on offer. We want the articles-only dump of Simple English:

wget http://dumps.wikimedia.org/simplewiki/20120112/simplewiki-20120112-pages-articles.xml.bz2

mwlib uses a db format to speed up access. Set up that DB with:

mw-buildcdb --input=some-dump.xml.bz2 --output=some_output_dir

This will take a couple of minutes, depending on the size of your dump. Coffee Time Once this is done, your some_output_dir should contain 3 files that we’ll access from mwlib later on.

wikiidx.cdb - the articles index file.
wikidata.bin - the wikitext for the articles.
wikiconf.txt - config file for mwlib

Parsing

We’re going to parse Simple English wiki’s page on Cheese mainly because I’m hungry. Check out the raw Wiki syntax on the edit page

Run this with Python, and you should get some wonderful output.

...
Inline link: Super Adams Smelly Cheese
Inline link: Emmental cheese
Inline link: String Cheese
Inline link: Marble cheese
Inline link: Wikimedia Commons
Inline link: media
Linked Japanese article: ja:チーズ

I’ll post more as I get further. For now, you should have a good craving for cheese.

Just found out my girlfriend posted this to Programmer Ryan...

Mon, 16 Jan 2012 15:56:00 +0900

Just found out my girlfriend posted this to Programmer Ryan Gosling. So happy :D

Even mentions all-important tea. happy sigh

Installing Python Junk without Hassle

Mon, 16 Jan 2012 15:26:30 +0900

Note to self, how to install Python stuff with minimum fuss. Looks like virtualenv.py is sort of like RVM for Python.

$ curl -O https://raw.github.com/pypa/virtualenv/master/virtualenv.py
$ python virtualenv.py my_new_env
$ . my_new_env/bin/activate
(my_new_env)$ pip install ...

RECYCLE OR DIE

Sun, 15 Jan 2012 16:36:50 +0900

RECYCLE OR DIE

Japanese Equivalent of The Onion

Wed, 11 Jan 2012 17:20:38 +0900

I love The Onion. For a long time I wished there was an equivalent satrirical site in Japanese. Someone just told me about the Kyoko Shimbun, a Japanese site full of made-up amusing stories.

For example they have a story on McDonalds Japan releasing the McDonal-don, a bowl of rice (don) topped with a burger.

Another story on naturally drying baumkuhen cakes in the sun.

Discovering that pi is only 10 digits long. Calculations until now having been a bug in the program running them. The quote from the researcher at the end is great.

The stories are short enough and have a good variety of vocabulary to be a pretty good way to practice Japanese I think.

Papers Incorrectly Identifies Duplicates

Wed, 11 Jan 2012 10:07:40 +0900

For those of you who use Papers like me, a quick warning - I’ve found it often incorrectly identifies non-duplicate papers as duplicates. I’m not sure why it does this, maybe because part of the download URL is the same.

Just double-check whether it really is a duplicate. It’s happened to me so often that I no longer check and just hit “Ignore” every time. It’s safer than losing a bunch of papers I just imported.

Learn French with Bouletcorp

Sun, 08 Jan 2012 20:34:31 +0900

Bouletcorp is a superb French comic blog, with a combination of mind-expanding topics, humour and great artwork. The French is quite challenging and idiomatic, which makes it great to study but tricky to look up. But luckily most of the comics are translated into English. So open up both versions in your browser and get reading.

The image is taken from the French and English comic called Kitchen Darwinism.

2012

Sun, 01 Jan 2012 15:41:34 +0900

2012

Ben Humphreys

Someone disconnects completely from technology for 90 days....

I haven’t been drunk in 3 years... and I’ve been partying way more than you.

Fast Ruby Parsers

Parsing Wiki Text (Part 2)

Dealing with Unicode CSV files in Python

Video

Star Wars Uncut: Director’s Cut: Wonderfully cute and...

Learn Languages Through Games on Steam

Parsing Edict XML with Perl and XML::LibXML

Print Wiki Parse Tree with `mwlib`

Python Regex Global Replace

Kamogawa river, Kyoto

Parsing Wiki Text (Part 1)

Wikimedia Markup

Install

Setup

Parsing

Next

Just found out my girlfriend posted this to Programmer Ryan...

Installing Python Junk without Hassle

RECYCLE OR DIE

Japanese Equivalent of The Onion

Papers Incorrectly Identifies Duplicates

Learn French with Bouletcorp

2012