miscellaneous factZ

A new version (v1.2) of my python script for converting markdown to latex is now done. markdown2latex (renamed from mkdn2latex) has been extensively refactored to become a proper python-markdown extension. This means it can be used seemlessly alongside plain markdown conversion, as well as independently whether as a module or, in its classic form, from the command line.

In addition for ease of installation it has also been turned into a proper python package and registered on pypi so you can just do:

$ easy_install markdown2latex

Alternatively you can still get it straight from the repository at:

http://knowledgeforge.net/okftext/svn/trunk/python/markdown2latex/

An (ongoing) summary of my experience with some of the utilities available for plotting from a python perspective.

Last updated: 2008-03-06

Ploticus

• (+) Fast, powerful, mature, well-documented
• (-) Not python based

C-based rather than python-based but fast and powerful. There is a (fairly crude) set of python bindings available here: http://www.srcc.lsu.edu/~davids/ploticus_module.html. Alternatively one can just call the ploticus command from a python script.

Matplotlib

• (+) Fairly powerful, mature, well-documented, nice pure python API
• (-) A little slow; requires a backend to be installed (so installation on a server is a problem)
• Could support object-orientation better

PyChart

http://home.gna.org/pychart/

• (+) Pure python, quite simple to use, good documentation
• (-) Not quite as nice looking or as powerful as e.g. ploticus

Biggles

http://biggles.sourceforge.net/

• last updated: 2004-03-08
• looks fine but does not seem to be actively developed any longer

Example

See: http://home.gna.org/pychart/examples/index.html. This is the bar/line example from there:

from pychart import *
theme.get_options()

data = [(10, 20, 30), (20, 65, 33),
    (30, 55, 30), (40, 45, 51),
    (50, 25, 27), (60, 75, 30)]

ar = area.T(size = (150,120),
            y_grid_interval=10,
            x_axis=axis.X(label="X label", label_offset=(0,-7)),
            y_axis=axis.Y(label="Y label"),
            legend = legend.T(), y_range = (0, None))

ar.add_plot(bar_plot.T(label="foo", data=data),
            line_plot.T(label="bar", data=data, ycol=2))
ar.draw()

import readline
readline.write_history_file('my_history.py')

I’ve been thinking about how to have a versioned domain model similar to the way we have versioned filesystems (e.g. subversion) for over two years. Over the last few months whatever bits of free time I’ve had have gone into developing a prototype built on top of sqlobject and I’ve now got a rough and ready (but fully functional) library:

http://project.knowledgeforge.net/ckan/svn/vdm/branches/sqlobj/

A demo of how it is used is best shown by the tests:

http://project.knowledgeforge.net/ckan/svn/vdm/branches/sqlobj/vdm/dm_test.py

Why be tied to SQLObject: obviously being so directly tied to sqlobject is not such a great thing but I intentionally chose to build on it because so many people will already be writing their domain models using SQLObject.

I’ve updated mkdn2latex the python script which converts markdown to latex (see also the original release announcement). Changes include:

• Support for markdown code blocks and html pre/code blocks generally using latex verbatim
• Verified compatibility with markdown 1.6
• A few minor bugfixes

Following my release of the markdown to latex script I’ve had a few enquiries from people asking about integrating mathematics with markdown generally (e.g. for web output as well as for output to latex). I’d already been using mathematics in markdown and then processing to html before I wrote the mkdn2latex script and in a world where one didn’t need to produce nice pdfs for conferences and journals it would be my preferred format. Anyway here’s a summary of the ways in which you can add mathematics support to basic markdown:

Mathematics in Markdown Howto

There are two possible options for pure web output with mathematics using markdown:

1. Add asciimathml/latexmathml support into the html files in which the markdown output will be inserted (these are javascript files to convert latex like mathematics to mathml on the fly see 1 and 2 — note that i recommend latexmathml as it is closer to latex).

2. Convert to latex and then convert to html use latex2html or similar

For pure html work I’ve used approach (1) up until now. This requires no change to your markdown processor only that you link to the right asciimathml/latexmathml javascript in the resulting html document (you can see an example in this simple wrapper around the basic markdown script)

In both cases you will want to insert math sections into your source markdown file. My convention is that any maths whether in paragraph or out should be enclosed in double dollars as in: \$\$ …. \$\$ (note that the \ should not be there but because latexmathml script is being used on this blog we need to escape one of the $ so that the text actually displays — as opposed to being render as mathematics). This is slightly different from the standard asciimathml/latexmathml conventions which just use a single $). I’ve made the necessary modifications (very minor) to asciimathml and latexmathml and you can find them at:

http://knowledgeforge.net/okftext/svn/trunk/js/

(look in the src subdirectories)

To summarize:

1. Create your markdown documents as normal.

2. To add mathematics just add it as for latex but using $\$ as delimiters. (If you plan to use javascript approach read up on those scripts to see what parts of latex they support). For example this would be fine (again ignore the backslashes):

    A simple markdown file, $$x$$, with some mathematics:
   
   
    \$\$ x^{2} + y^{2} = z^{2} \$\$
   
   
    A new paragraph after a block of mathematics ...
   

3. Then:

   1. EITHER convert to markdown as usual but then insert link to modified latexmathml.js in your html documents (or if using original latexmathml just convert $$ to $ everywhere)
   2. OR convert markdown to latex using my script and then use latex2html

We intend to add annotation/commentarysupport to the open shakespeare web demo either in this release or next. As a first step I’ve been looking to see what (open-source) web-based annotation systems are already out there. Below is a list of what I’ve been able to find so far (if you know of more please post a comment). After examining several of these in some detail the one we’re going to try our properly is marginalia (if you’re interested our current efforts to do this including writing a python wsgi annotation service backend can be found here in the subversion repository).

1. stet: javascript annotation system used for gpl v3 comments system

   • http://en.wikipedia.org/wiki/Stet_(software)
   • Bit of a hack at present and did not seem designed for external reuse (when I last looked the README was fairly emphatic that this was very alpha with little documentation)

2. commentary: javascript based wsgi middleware developed by ian bicking

   • http://pythonpaste.org/commentary/
   • Rather hacked together (apparently he coded it in a week). Had problems getting it working locally and no documentation to help in adaptation. Seems to be unmaintained (demo site is currently down) which is perhaps not surprising given how many other projects Ian has on the go.
   • One nice feature is that you don’t seem to have to mess with the underlying web pages you want to add comments to (this only works if you are sitting on top of another wsgi application)

3. marginalia: javascript library and spec for adding web annotation to pages

   • http://www.geof.net/code/annotation/
   • javascript code seems well factored and understandable and docs are good

4. annotea: W3C project based on RDF

   • http://www.w3.org/2001/Annotea/
   • Been around a long time and now seems to be inactive
   • Server and client support rather lacking. No simple interface based on, e.g., javascript — you have to write a special client yourself — which is a major drawback
   • That said the protocol is well-documented and so writing a client (or a server) shouldn’t be that hard (other than having to mess around with rdf in javascript …)
   • The Schema seems reasonable
   • xpointer based which according to the marginalia site is a problem

UPDATE (2008-06): a new version is available (v1.2): http://www.rufuspollock.org/2008/06/23/markdown2latex-mkdn2latex-12/

Over the last year I’ve written quite a few papers using markdown plus asciimathml. While this is great for web publication (and editing) and gives me lots of styling freedom via css it doesn’t produce output that’s as nice as that produced by latex especially in paginated form (also latex mathematics support is also currently better than that of obtained from asciimathml or latexmathml).

Unable to find any python code that would do what I want I played around for a couple of hours with the python-markdown script until I got something functional. After a few weeks of use which has allowed me to iron out the bugs and making several improvements I feel the script is now ready for public release. Hope people find it useful.

Download

Get it from: http://project.knowledgeforge.net/okftext/svn/trunk/python/mkdn2latex.py

(You can also it check it out using subversion from the same url if you want)

For the script to function you will also need to install the python-markdown module v1.5 (make sure you install it under the name markdown.py).

Usage

The following will print the latex output to the console (standard out):

 $ mkdn2latex.py path-to-markdown-file.mkd

To convert a markdown file straight to a latex output file do:

 $ mkdn2latex.py path-to-markdwon-file.mkd > path-to-output-file.ltx

NB: As provided the script expects mathematics in your markdown file to be delimited with ‘$\$’ (this should be dollar dollar — the slash is there to stop this being rendered as maths in the blog) as opposed to the standard asciimathml delimiters of ‘`’ or ‘$’.

Whenever I’ve had a few spare minutes over the last couple of months I’ve been hacking away on svnrepo, a pythonic API to local subversion repositories and it is now robust enough to warrant a 0.1 release. svnrepo is (and was intended to be) very small, just a single module, that wrapped the python subversion bindings for repository access to make them simpler to use and more object-oriented. At present the module requires subversion >= 1.3 but I’m hoping to scale that dependency back in future releases.

Getting it

The module is Open Source software (MIT-licensed) and you can either:

1. Download it directly from: http://www.rufuspollock.org/code/svnrepo/svnrepo.py

2. Or get it the python package index. If you are using setuptools just do:

   $ easy_install svnrepo

What it looks like

There are unit tests at:

http://www.rufuspollock.org/code/svnrepo/svnrepo_test.py

And they are pretty good at demonstrating how to use the API but just for the sake of demonstration. Assume that you have an existing subversion repository at REPOSPATH.

from svnrepo import *
REPOSPATH = ...
repos = Repository(REPOSPATH)

history = repos.history('/')
for revision in history:
    print history

rev = repos.get_revision() # get the youngest revision
print rev.log # the log message of the revision
print rev.date

# get a node
rootdir = rev.get_node('/')
print rootdir.is_dir()
print rootdir.list_dir()

# create a new revision
newrev = repos.new_revision()
newrev.log = 'My new revision'
newrev.author = 'me'
fs = newrev.file_system
filepath = 'tmp.txt'
newfile = fs.make_file(filepath)

text = 'nothing ever exists entirely alone'
newfile.write(text)

propname = 'copyright'
propval = 'nemo'
newfile.set_property(propname, propval)

newrev.commit()

Having looked around for a while without success for something that would spit out csv files as ascii tables I decided to hack something together. The result is a small python script [csv2ascii.py][]. It is currently fairly crude, for example it just truncates cell text which is too long, but I hope I’ll have some more time to improve it soon.

Example

Suppose you had the following in a file called example.csv:

"YEAR","PH","RPH","RPH_1","LN_RPH","LN_RPH_1","HH","LN_HH"
1971,7.8523,43.9168,42.9594,3.7822,3.7602,16185,9.691843   
1972,10.5047,55.1134,43.9168370988587,4.0093,3.7822,16397,9.704855

Running:

 $ ./csv2ascii.py example.csv

Would result in:

+------+------+------+------+------+------+------+------+
| YEAR |  PH  | RPH  |RPH_1 |LN_RPH|LN_RPH|  HH  |LN_HH |
+------+------+------+------+------+------+------+------+
| 1971 |7.8523|43.916|42.959|3.7822|3.7602|16185 |9.6918|
+------+------+------+------+------+------+------+------+
| 1972 |10.504|55.113|43.916|4.0093|3.7822|16397 |9.7048|
+------+------+------+------+------+------+------+------+