miscellaneous factZ – The online home of Rufus Pollock

Deliverance is a great library that lets you easily re-theme external websites on the fly. Designed as WSGI middleware, it can be easily combined with some proxying to integrate a bunch of websites together

You can use deliverance plus proxying out-of-the-box using the deliverance-proxy command. However, I was interested in using Deliverance as middleware from code. This turned out to be none too trivial to do — all the examples on the internet seemed to focus on using deliverance-proxy or using it in an ini file.

After much wrestling, most notably with odd issues with gzipped (deflated) content I got it working and you can find a demo implementation (see demo.py and README.txt) here:

http://rufuspollock.org/code/deliverance/

I should also mention the following sources which were all of help in my quest:

• http://codespeak.net/svn/z3/deliverance/sandbox/ianb/ploneconf2008/index.txt
• http://www.gawel.org/weblog/en/2008/12/skinning-with-pyquery-and-deliverance
• http://macadames.wordpress.com/2009/05/23/some-deliverance-tips/
• http://www.coactivate.org/projects/deliverance/lists/deliverance-discussion/archive/2009/05/1241444934295/forum_view
• http://www.sixfeetup.com/blog/2009/4/27/deploying-plone-and-zine-together-with-deliverance-using-repoze

Instructions on using sqlalchemy migrate with Pylons, especially to convert an existing pylons project to use sqlalchemy migrate

This is based off several excellent sources including this guide and these threads.

One important point to note is that you are likely to end up with two versions of your model tables: one in yourapp/model and one in yourapp/migration/versions/*.py with the former representing your tables at HEAD and the second containing upgrade/downgrade scripts whose final result is HEAD. This duplication is a bit annoying and I discuss how it can be avoided below.

1. Install sqlalchemy migrate for your project e.g.

  pip -E {your-virtualenv} install sqlalchemy-migrate
  # or
  easy_install sqlalchemy-migrate

NB: latest version of migrate are only compatible with sqlalchemy >= 0.5 (for previous version of sqlalchemy you need migrate <= 0.4.5)

2. Create the migrate repository (i.e. store for upgrade scripts …).

In your project directory

  migrate create myapp/migration/ "MyApp"

Now create a temporary helper script:

  migrate manage dbmanage.py --repository=myapp/migration/ --url={your-sqlalchemy-db-uri}

3. Set up db version control

  python dbmanage.py version_control

Check the current version (should be 0)

  python dbmanage.py version

4. Create a migration script for your existing db

  python dbmanage.py script "Add existing tables"

This will create a script in myapp/migration/versions/001addexisting_tables.py

Copy into that file the definition for all your existing tables (and other database stuff such as constraints) and then create those tables in the upgrade() function (and delete them in downgrade()).

That’s it! (in theory)

Additional Issues

1. Duplication of model/db code

You now have two places for model/db code:

1. Your migration scripts
2. Your actual model

This doesn’t have to be a problem but it is an obvious way for bugs to creep especially when some people start by creating their DB from the model code and others from the migration scripts.

Warning: this method will not work if do stuff in your table creation that is not persisted into the actual DB sql (e.g. column default values based on a function, custom db types …).

One way to avoid the duplication is to have all table creation and alteration confined to your migration scripts and then have your model tables set up directly from the DB using the autoload=True option. The one disadvantage of this is you can’t see the complete DB set up in one places as tables construction may be spread over several migrate scripts. One solution to this is provided by the experimental ‘create_model’ command which dumps the current DB model in the required sqlalchemy table code.

More discussion in this migrate-users thread

Bringing the Migration DB up to date

If you do set up your DB (from new) directly from your model code rather than the migration scripts then this requires that you set up the migration stuff and update the migrate version to the correct number. (I note there is an experimental updatedbto_model command which is supposed to do this for you). You can do this as follows (assuming your migrate repository is at YOURAPP:

      from migrate.versioning.api import version_control, version
      import YOURAPP.migration.versions
      v = version(YOURAPP.migration.__path__[0])
      # log.info( "Setting current version to '%s'" % v )
      # url is your sqlalchemy db url 
      version_control(url, YOURAPP.migration.__path__[0], v)

Extras

• Should wrap upgrade/downgrade in transactions. I found one way to do this here but testing indicated this didn’t work for me (rollback was not working properly when there was an error)

A new version (v1.2) of my python script for converting markdown to latex is now done. markdown2latex (renamed from mkdn2latex) has been extensively refactored to become a proper python-markdown extension. This means it can be used seemlessly alongside plain markdown conversion, as well as independently whether as a module or, in its classic form, from the command line.

In addition for ease of installation it has also been turned into a proper python package and registered on pypi so you can just do:

$ easy_install markdown2latex

Alternatively you can still get it straight from the repository at:

http://knowledgeforge.net/okftext/svn/trunk/python/markdown2latex/

An (ongoing) summary of my experience with some of the utilities available for plotting from a python perspective.

Last updated: 2008-03-06

Ploticus

• (+) Fast, powerful, mature, well-documented
• (-) Not python based

C-based rather than python-based but fast and powerful. There is a (fairly crude) set of python bindings available here: http://www.srcc.lsu.edu/~davids/ploticus_module.html. Alternatively one can just call the ploticus command from a python script.

Matplotlib

• (+) Fairly powerful, mature, well-documented, nice pure python API
• (-) A little slow; requires a backend to be installed (so installation on a server is a problem)
• Could support object-orientation better

PyChart

http://home.gna.org/pychart/

• (+) Pure python, quite simple to use, good documentation
• (-) Not quite as nice looking or as powerful as e.g. ploticus

Biggles

http://biggles.sourceforge.net/

• last updated: 2004-03-08
• looks fine but does not seem to be actively developed any longer

Example

See: http://home.gna.org/pychart/examples/index.html. This is the bar/line example from there:

from pychart import *
theme.get_options()

data = [(10, 20, 30), (20, 65, 33),
    (30, 55, 30), (40, 45, 51),
    (50, 25, 27), (60, 75, 30)]

ar = area.T(size = (150,120),
            y_grid_interval=10,
            x_axis=axis.X(label="X label", label_offset=(0,-7)),
            y_axis=axis.Y(label="Y label"),
            legend = legend.T(), y_range = (0, None))

ar.add_plot(bar_plot.T(label="foo", data=data),
            line_plot.T(label="bar", data=data, ycol=2))
ar.draw()

import readline
readline.write_history_file('my_history.py')

I’ve been thinking about how to have a versioned domain model similar to the way we have versioned filesystems (e.g. subversion) for over two years. Over the last few months whatever bits of free time I’ve had have gone into developing a prototype built on top of sqlobject and I’ve now got a rough and ready (but fully functional) library:

http://project.knowledgeforge.net/ckan/svn/vdm/branches/sqlobj/

A demo of how it is used is best shown by the tests:

http://project.knowledgeforge.net/ckan/svn/vdm/branches/sqlobj/vdm/dm_test.py

Why be tied to SQLObject: obviously being so directly tied to sqlobject is not such a great thing but I intentionally chose to build on it because so many people will already be writing their domain models using SQLObject.

I’ve updated mkdn2latex the python script which converts markdown to latex (see also the original release announcement). Changes include:

• Support for markdown code blocks and html pre/code blocks generally using latex verbatim
• Verified compatibility with markdown 1.6
• A few minor bugfixes

Following my release of the markdown to latex script I’ve had a few enquiries from people asking about integrating mathematics with markdown generally (e.g. for web output as well as for output to latex). I’d already been using mathematics in markdown and then processing to html before I wrote the mkdn2latex script and in a world where one didn’t need to produce nice pdfs for conferences and journals it would be my preferred format. Anyway here’s a summary of the ways in which you can add mathematics support to basic markdown:

Mathematics in Markdown Howto

There are two possible options for pure web output with mathematics using markdown:

1. Add asciimathml/latexmathml support into the html files in which the markdown output will be inserted (these are javascript files to convert latex like mathematics to mathml on the fly see 1 and 2 — note that i recommend latexmathml as it is closer to latex).

2. Convert to latex and then convert to html use latex2html or similar

For pure html work I’ve used approach (1) up until now. This requires no change to your markdown processor only that you link to the right asciimathml/latexmathml javascript in the resulting html document (you can see an example in this simple wrapper around the basic markdown script)

In both cases you will want to insert math sections into your source markdown file. My convention is that any maths whether in paragraph or out should be enclosed in double dollars as in: \$\$ …. \$\$ (note that the \ should not be there but because latexmathml script is being used on this blog we need to escape one of the $ so that the text actually displays — as opposed to being render as mathematics). This is slightly different from the standard asciimathml/latexmathml conventions which just use a single $). I’ve made the necessary modifications (very minor) to asciimathml and latexmathml and you can find them at:

http://knowledgeforge.net/okftext/svn/trunk/js/

(look in the src subdirectories)

To summarize:

1. Create your markdown documents as normal.

2. To add mathematics just add it as for latex but using $\$ as delimiters. (If you plan to use javascript approach read up on those scripts to see what parts of latex they support). For example this would be fine (again ignore the backslashes):

    A simple markdown file, $$x$$, with some mathematics:
   
   
    \$\$ x^{2} + y^{2} = z^{2} \$\$
   
   
    A new paragraph after a block of mathematics ...
   

3. Then:

   1. EITHER convert to markdown as usual but then insert link to modified latexmathml.js in your html documents (or if using original latexmathml just convert $$ to $ everywhere)
   2. OR convert markdown to latex using my script and then use latex2html

We intend to add annotation/commentarysupport to the open shakespeare web demo either in this release or next. As a first step I’ve been looking to see what (open-source) web-based annotation systems are already out there. Below is a list of what I’ve been able to find so far (if you know of more please post a comment). After examining several of these in some detail the one we’re going to try our properly is marginalia (if you’re interested our current efforts to do this including writing a python wsgi annotation service backend can be found here in the subversion repository).

1. stet: javascript annotation system used for gpl v3 comments system

   • http://en.wikipedia.org/wiki/Stet_(software)
   • Bit of a hack at present and did not seem designed for external reuse (when I last looked the README was fairly emphatic that this was very alpha with little documentation)

2. commentary: javascript based wsgi middleware developed by ian bicking

   • http://pythonpaste.org/commentary/
   • Rather hacked together (apparently he coded it in a week). Had problems getting it working locally and no documentation to help in adaptation. Seems to be unmaintained (demo site is currently down) which is perhaps not surprising given how many other projects Ian has on the go.
   • One nice feature is that you don’t seem to have to mess with the underlying web pages you want to add comments to (this only works if you are sitting on top of another wsgi application)

3. marginalia: javascript library and spec for adding web annotation to pages

   • http://www.geof.net/code/annotation/
   • javascript code seems well factored and understandable and docs are good

4. annotea: W3C project based on RDF

   • http://www.w3.org/2001/Annotea/
   • Been around a long time and now seems to be inactive
   • Server and client support rather lacking. No simple interface based on, e.g., javascript — you have to write a special client yourself — which is a major drawback
   • That said the protocol is well-documented and so writing a client (or a server) shouldn’t be that hard (other than having to mess around with rdf in javascript …)
   • The Schema seems reasonable
   • xpointer based which according to the marginalia site is a problem

UPDATE (2008-06): a new version is available (v1.2): http://www.rufuspollock.org/2008/06/23/markdown2latex-mkdn2latex-12/

Over the last year I’ve written quite a few papers using markdown plus asciimathml. While this is great for web publication (and editing) and gives me lots of styling freedom via css it doesn’t produce output that’s as nice as that produced by latex especially in paginated form (also latex mathematics support is also currently better than that of obtained from asciimathml or latexmathml).

Unable to find any python code that would do what I want I played around for a couple of hours with the python-markdown script until I got something functional. After a few weeks of use which has allowed me to iron out the bugs and making several improvements I feel the script is now ready for public release. Hope people find it useful.

Download

Get it from: http://project.knowledgeforge.net/okftext/svn/trunk/python/mkdn2latex.py

(You can also it check it out using subversion from the same url if you want)

For the script to function you will also need to install the python-markdown module v1.5 (make sure you install it under the name markdown.py).

Usage

The following will print the latex output to the console (standard out):

 $ mkdn2latex.py path-to-markdown-file.mkd

To convert a markdown file straight to a latex output file do:

 $ mkdn2latex.py path-to-markdwon-file.mkd > path-to-output-file.ltx

NB: As provided the script expects mathematics in your markdown file to be delimited with ‘$\$’ (this should be dollar dollar — the slash is there to stop this being rendered as maths in the blog) as opposed to the standard asciimathml delimiters of ‘`’ or ‘$’.