miscellaneous factZ – The online home of Rufus Pollock

UPDATE (2008-06): a new version is available (v1.2): http://www.rufuspollock.org/2008/06/23/markdown2latex-mkdn2latex-12/

Over the last year I’ve written quite a few papers using markdown plus asciimathml. While this is great for web publication (and editing) and gives me lots of styling freedom via css it doesn’t produce output that’s as nice as that produced by latex especially in paginated form (also latex mathematics support is also currently better than that of obtained from asciimathml or latexmathml).

Unable to find any python code that would do what I want I played around for a couple of hours with the python-markdown script until I got something functional. After a few weeks of use which has allowed me to iron out the bugs and making several improvements I feel the script is now ready for public release. Hope people find it useful.

Download

Get it from: http://project.knowledgeforge.net/okftext/svn/trunk/python/mkdn2latex.py

(You can also it check it out using subversion from the same url if you want)

For the script to function you will also need to install the python-markdown module v1.5 (make sure you install it under the name markdown.py).

Usage

The following will print the latex output to the console (standard out):

 $ mkdn2latex.py path-to-markdown-file.mkd

To convert a markdown file straight to a latex output file do:

 $ mkdn2latex.py path-to-markdwon-file.mkd > path-to-output-file.ltx

NB: As provided the script expects mathematics in your markdown file to be delimited with ‘$\$’ (this should be dollar dollar — the slash is there to stop this being rendered as maths in the blog) as opposed to the standard asciimathml delimiters of ‘`’ or ‘$’.

Having looked around for a while without success for something that would spit out csv files as ascii tables I decided to hack something together. The result is a small python script csv2ascii.py. It is currently fairly crude, for example it just truncates cell text which is too long, but I hope I’ll have some more time to improve it soon.

Example

Suppose you had the following in a file called example.csv:

"YEAR","PH","RPH","RPH_1","LN_RPH","LN_RPH_1","HH","LN_HH"
1971,7.8523,43.9168,42.9594,3.7822,3.7602,16185,9.691843   
1972,10.5047,55.1134,43.9168370988587,4.0093,3.7822,16397,9.704855

Running:

 $ ./csv2ascii.py example.csv

Would result in:

+------+------+------+------+------+------+------+------+
| YEAR |  PH  | RPH  |RPH_1 |LN_RPH|LN_RPH|  HH  |LN_HH |
+------+------+------+------+------+------+------+------+
| 1971 |7.8523|43.916|42.959|3.7822|3.7602|16185 |9.6918|
+------+------+------+------+------+------+------+------+
| 1972 |10.504|55.113|43.916|4.0093|3.7822|16397 |9.7048|
+------+------+------+------+------+------+------+------+

Anyone who has converted some old wordprocessed documents to plain ascii text will know that wordprocessors love to insert their only special versions for a few of the standard characters such as ‘ and ” (- also comes up pretty frequently). I personally came across while using odt2txt plus openoffice to convert some old .rtf and .doc files to plain text. By default odt2txt writes the files as utf-8 which is fine except there is really no reason these shouldn’t be full on ascii (plus the standard vim distribution on mac osx doesn’t support unicode!). So after some digging around here are the relevant code conversions you will usually need:

\u2018 (curly right single quote) -> '
\u2019 (curly left single quote) -> '
\u201c (curly right double quote) -> "
\u201d (curly left double quote) -> "

Or in python:

 out = <your-unicode-text>
 out = out.replace( u'\u2018', u"'")
 out = out.replace( u'\u2019', u"'")
 out = out.replace( u'\u201c', u'"')
 out = out.replace( u'\u201d', u'"')
 out.encode('ascii')

Archiving for my own benefit the results of yet another 5 minute look for how to do find and replace across multiple files from the command line:

1. Use sed:

     sed -i 's/foo/foo_bar/g'  *.html
   

2. use the old perl hack:

     perl -w -pi~ -e 's/foo/bar/' [files]
   

   Notes: -p: loop, -i edit files in place (backup with extension if supplied), -w enable warnings

3. Install rpl

   • http://freshmeat.net/projects/rpl/
   • port install rpl (mac osx)
   • apt-get install rpl (debian/ubuntu)

Combining either (1) or (2) with find is pretty powerful. E.g. to do a find and replace on all html files in all subdirectories:

     perl -w -pi -e 's/foo/bar/' `find <path> -name '*.html'`

I am a heavy vim user and want to be able to copy and paste from the system clipboard so that, for example, I can edit my posts in vim and then paste them from firefox in here. I should also mention that I prefer to use terminal not gui versions of vim which influences the solutions recommended below.

Mac OSX

Use pbcopy and pbpaste. These are two very useful Mac OSX terminal commands that give access to the system clipboard. To use from vim pipe your selection (using ‘!’) through them, e.g. to copy the whole document to the clipboard:

:%!pbcopy

Debian/Ubuntu

Method 1: similar to Mac OSX trick using xclip (sudo apt-get install xclip) and then pipe through that. Specifically:

:%!xclip -i -selection clipboard

Method 2: If you have xterm-clipboard enabled in your version of vim (not the case in the default version of vim installed, you’ll need vim-full) you can access the clipboard as a register. Selection register is named ‘*’ and clipboard is named ‘+’. So to copy to the clipboard get your selection and then do:

"+y

For more information on vim and the xclipboard do:

:help x11-selection

or see http://vim.sourceforge.net/tips/tip.php?tip_id=71 (2008-01-02: or http://vim.wikia.com/wiki/In_line_copy_and_paste_to_system_clipboard).

Just some random ideas on how to do this culled from elsewhere

Rename *.foo to *.bar:

ls -d *.foo | sed -e 's/.*/mv & &/' | sh

#!/usr/bin/env python
# Convert a whole directory tree of flac files to oggs
# Just a wrapper on a couple of system commands
# Could be made more 'pythonic' by replacing find with os.walk
# Public Domain: copy, redistribute, reuse freely and without restriction
import os
import shutil
import commands

# base directory for flac files
srcPath = '/var/share/music'
# destination directory for ogg files
destPath = '/var/share/ogg'

cmds = [
        "find . -type d -exec mkdir %s/'{}' \\;" % destPath,
        "find . -name '*.flac' -exec oggenc --quiet --quality 4 '{}' \\;",
        "find . -name '*.ogg' -exec mv '{}' %s/'{}' \\;" % destPath,
        ]

def run_cmd(cmd):
    status, output = commands.getstatusoutput(cmd)
    if status:
        print 'Had error running [%s]: %s' % (cmd, output)

def convert_to_ogg(clean=True):
    if clean:
        shutil.rmtree(destPath)
    if not os.path.exists(destPath):
        os.makedirs(destPath)
    os.chdir(srcPath)
    for cmd in cmds:
        run_cmd(cmd)

if __name__ == '__main__':
    convert_to_ogg()

Very annoyingly if you update the WP site url (or move your blog without updating site url) your install will be b0rked and you will have to go mess around with the db. What you need to do (tested on WP 2.0) is:

update wp_options set option_value = '[wordpress -address]' where option_name = 'siteurl';

update wp_options set option_value = '[site -address]' where option_name = 'home';

Here are some scripts along with instructions for migrating a drupal site to wordpress.

README.txt

These instructions are ‘implemented’ in code as a small python script called migrate-drupal-data.py which you can find below.

1. Dump your drupal database. To find out how to do this refer to the manual for your db (for mysql from the command line you use mysqldump and for postgres it is pg_dump)

2. Load the drupal dump into your wordpress db:

     mysql -u [username] -p [password] < [path-to-drupal-dump]
   

3. Edit the convert-drupal-data.sql script replacing ‘weblog_’ with your wordpress prefix

4. Run convert-drupal-data.sql script against your wordpress db:

     mysql -u [username] -p [password] < convert-drupal-data.sql
   

   WARNING: this will delete all existing posts, comments and categories from your wordpress db. If you don’t want to do that edit the script as indicated therein).

5. Finished.

Scripts

convert-drupal-data.sql

-- taken from
-- http://vrypan.net/log/archives/2005/03/10/migrating-from-drupal-to-wordpress/
-- and then improved  

-- these lines will result in the deletion of all existing posts, comments
-- and categories. Comments these out using '--' or /* ... */ if you don't
-- want that to happen 
DELETE FROM weblog_wp_categories ;
DELETE FROM weblog_wp_posts ;
DELETE FROM weblog_wp_post2cat ;
DELETE FROM weblog_wp_comments ;

/*
-- does not work (think it is to do with password issues)
-- also causes problems with auto-increment
delete from weblog_wp_users;

INSERT INTO weblog_wp_users
  (ID, user_login, user_pass, user_nickname, user_email, user_registered)
  SELECT
  uid, name, pass, name, mail, FROM_UNIXTIME(timestamp)
  FROM users;
*/

INSERT into weblog_wp_categories(
  cat_ID,cat_name, category_nicename, category_description, category_parent
  )
  SELECT term_data.tid, name, name, description, parent
  FROM term_data, term_hierarchy
  WHERE term_data.tid=term_hierarchy.tid;

INSERT INTO weblog_wp_posts(
  ID, post_author, post_date, post_content, post_title, post_excerpt,
  post_name, post_modified
  )
  SELECT nid, 1, FROM_UNIXTIME(created), body, title, teaser, concat('OLD',nid), FROM_UNIXTIME(changed)
  FROM node
  WHERE type='blog' OR type='page' OR type='story' OR type='forum' ;

INSERT INTO weblog_wp_post2cat (post_id,category_id)
  SELECT nid,tid
  FROM term_node ;

INSERT INTO weblog_wp_comments (
  comment_post_ID, comment_date, comment_content, comment_parent
  )
  SELECT nid, FROM_UNIXTIME(timestamp), concat(subject,' ', comment), thread
  FROM comments ;

DROP TABLE term_data;
DROP TABLE term_hierarchy;
DROP TABLE node;
DROP TABLE term_node;
DROP TABLE comments;
DROP TABLE users;

migrate-drupal-data.py

#!/usr/bin/env python
import os

# target wordpress db information
user = 'your_user_name'
db = 'your_db_name'

# replace this with the path to your dump from drupal
drupalDbDump = '~/tmp/drupal-db-dump.sql'

# path to drupal conversion sql script
templateConvertDrupalSql = 'convert-drupal-data.sql'

# tmp file used as an intermediary 
convertDrupalData = 'tmp_sql.sql'

# default wordpress prefix used in templateConvertDrupalSql script
# you shouldn't have to change this
templatePrefix = 'weblog_'

# your wordpress prefix
wpPrefix = ''

def prepareConvertSql():
    ff = file(templateConvertDrupalSql)
    tstr = ff.read()
    ff.close()
    tstr = tstr.replace(templatePrefix, wpPrefix)
    ff2 = file(convertDrupalData, 'w')
    ff2.write(tstr)
        ff2.close()

if __name__ == '__main__':
    sqlCmd = 'mysql -u %s -p %s < %s'
    cmdInsertDrupalData = sqlCmd % (user, db, drupalDbDump)
    cmdConvertDrupalData = sqlCmd % (user, db, convertDrupalData)

    # getData()
    prepareConvertSql()
    os.system(cmdInsertDrupalData)
    os.system(cmdConvertDrupalData)

Money Cases

• 1988 First National Bank of Chicago is the victim of $70-million computer heist. [1]
• 1994, summer. Russian crackers siphon $10 million from Citibank and transfer the money to bank accounts around the world. Vladimir Levin, the 30-year-old ringleader, uses his work laptop after hours to transfer the funds to accounts in Finland and Israel. Levin stands trial in the United States and is sentenced to three years in prison. Authorities recover all but $400,000 of the stolen money. [1]

To Investigate

• Kevin Mitnick

Biblio

1. http://www.nationmaster.com/encyclopedia/Timeline-of-hacker-history
2. Bruce Sterling, The Hacker Crackdown (available in a digital edition – just search online)