Parsing Wiki Text (Part 2)
More on parsing Wikipedia markup. So you’ve abandoned the idea of using regular expressions to parse the markup, congratulations. This should save yourself from insanity and oblivion.
mwlib is a Python library for parsing Wiki markup. It seems semi-official. Whatever, it’s what we’re going to use.
It can parse the zipped XML dumps as I’ve shown in previous tutorials.
However you have to provide it with an ariticle name, and today we’re feeling more John Wayne.
How can we feed mwlib raw Wiki markup and get the same hierarchical deliciousness as output? Here’s how:
A more involved example:
Dealing with Unicode CSV files in Python
Today I found out that Python’s csv module cannot read UTF-8 files (see the warning at the top of the page).
There are two Stackoverflow questions on the topic (here and here) but the solutions don’t seem to work for Python 2.6.
There code examples on the official page borrow from StackOverflow (or the other way around), and don’t seem to help much.
The more I learn about Python the more I appreciate Ruby.
Star Wars Uncut: Director’s Cut: Wonderfully cute and funny clips by hundreds of fans, stuck together to make the entire first film (via TokyoJoe).
Hand-made outfits, stop-motion segments, entire clips rendered on 80’s Macintoshes, Star Trek uniforms. Also My Little Pony at 0:37.
Source: tokyojoe0985
Learn Languages Through Games on Steam
If you like playing PC games, and want to learn a language at the same time, try this. Set your Steam language to the foreign language you’re trying to learn, and you can play most of your library in that language. It’s under Preferences > Interface then choose your preferred language from the first drop-down.
Sometimes Steam will need to re-download the data to get the localised content, but once that’s all done, you should be able to play without any trouble. For example, both Portal games, Civilisation, Team Fortress 2 and Skyrim all work in French. Check the languages section of Steam store page for that game for what languages are available.
The only problem with learning from games is you might learn a lot of weird vocabulary. Just think of it as gaming with a bonus, rather than actual language practice.
Parsing Edict XML with Perl and XML::LibXML
Edict is a Japanese-English dictionary that is free to use for research (as far as I know). It’s available in a few formats, the most useful of which is XML dump of English-only data.
It might help someone sometime, so I’ve posted a short Perl snippet of how to parse the format.
A single entry looks like:
Print Wiki Parse Tree with `mwlib`
It’s hard to see how mwlib treats Wiki markup. You can use this small snippet to output the tree of an article and see what you’re dealing with. It’s recursive so if you have a Huge document it’s possible you’ll get some problems, although I haven’t hit any yet.
Output is something like:
Section
Node
Text
Node
Paragraph
Text
Paragraph
Style
Node
Node
ArticleLink
Text
ArticleLink
Text
...
There’s probably a function within mwlib for doing this, but I couldn’t find it.
Python Regex Global Replace
Today’s fun Python gotcha: I couldn’t find out how to do a global regular expression replace. In Perl it would look like:
$my_str =~ s/foo/bar/g;
In Python it’s:
import re
re.sub('foo', 'bar', my_text)
It turns out I wasn’t reading the sub() documentation properly. The amazing @puresock saved the day and pointed out this from the documentation:
The optional argument count is the maximum number of pattern occurrences to be replaced; count must be a non-negative integer. If omitted or zero, all occurrences will be replaced. Empty matches for the pattern are replaced only when not adjacent to a previous match, so
sub('x*', '-', 'abc')returns'-a-b-c-'.
tl;dr Python’s regex-replace function sub() defaults to global. So you don’t need a global flag.
Kamogawa river, Kyoto
Parsing Wiki Text (Part 1)
I’ll be dealing with Wiki text a lot in the coming months, so I thought I’d post what I learn, as I learn it.
This is just a quick introduction on what I did today pieced together from mailing list posts and various other posts online.
Wikimedia Markup
First some points on the markup used on Wikipedia. There seems to be a few ways to call this syntax, wikitext, wikimedia markup, wiki something. By some weird definitions it’s not a real grammar, merely something that can be translated into HTML easily.
From what I’ve read online, parsing it naïvely using regular expressions is not a great idea. There are too many edge cases, and weird ways in which things can be written for it to be viable. However I have a feeling that using regular expressions only would be a lot faster than the grammar-like parsers that are available. Maybe I’ll do a short benchmark another time.
For now, rather than roll my own, I decided to use the Python library mwlib
to parse the Wiki markup.
Install
Let’s install mwlib then. Unfortunately the documentation for mwlib is somewhere between non-existent and useless.
If you’re on a shared setup, I found the easiest way to install is using
virtualenv.py:
$ curl -O https://raw.github.com/pypa/virtualenv/master/virtualenv.py
$ python virtualenv.py wiki
$ . wiki/bin/activate
(wiki)$ pip install mwlib
Setup
You then need to download the dump(s) of the data you want to deal with. Find it from the full list of dumps available. It’s likely you’ll just want the Articles, but refer to the documentation for more details of what’s on offer. We want the articles-only dump of Simple English:
wget http://dumps.wikimedia.org/simplewiki/20120112/simplewiki-20120112-pages-articles.xml.bz2
mwlib uses a db format to speed up access. Set up that DB with:
mw-buildcdb --input=some-dump.xml.bz2 --output=some_output_dir
This will take a couple of minutes, depending on the size of your dump. Coffee Time
Once this is done, your some_output_dir should contain 3 files that we’ll
access from mwlib later on.
wikiidx.cdb- the articles index file.wikidata.bin- the wikitext for the articles.wikiconf.txt- config file for mwlib
Parsing
We’re going to parse Simple English wiki’s page on Cheese mainly because I’m hungry. Check out the raw Wiki syntax on the edit page
Run this with Python, and you should get some wonderful output.
...
Inline link: Super Adams Smelly Cheese
Inline link: Emmental cheese
Inline link: String Cheese
Inline link: Marble cheese
Inline link: Wikimedia Commons
Inline link: media
Linked Japanese article: ja:チーズ
Next
I’ll post more as I get further. For now, you should have a good craving for cheese.
