Saturday, December 15, 2007

Iterating properly in Python

After more than a year of python hacking you take a look at the official Python tutorial and still discover new ways of enhancing your code. Just two ways of saving list constructions on our actual code.

The first one: In our code, we are currently doing all the time something like this:


for key, value in myDictionary.items() :
doWhatEverWit(key,value)

This is not that appropiate because it creates the tuple list and then iterates. But if you do:


for key, value in myDictionary.iteritems() :
doWhatEverWit(key,value)

You just get fetched keys and values without creating any temporary list.

The other thing our code is full of is constructing a list using list comprehensions:


tempList = [expresionUsing(item) for item in collection]
for item in tempList: doWhatEverWith(item)

I knew about list comprehensions and i knew about generator functions. Generator functions are functions can be a iteration source by doing yield instead of return. yield returns a value but keeps the function execution state for the next call. That is a very interesting feature but i never used it because it is harder to create a function that yields than to create the loop itself.

But by reading the tutorial i found a brand new kind of expressions that sum up list comprehensions and generator functions: generator expressions. They have the same syntax than list comprehensions using parethesis instead of brackets, but instead of creating a list they create a source of iteration so that no temp lists are created. Previous code would look like that:


iterable = (expresionUsing(item) for item in collection)
for item in iterable: doWhatEverWith(item)

There are plenty of places in the code we generated last year (WiKo, TestFarm and build and utility scripts in CLAM where we can apply those two ideas. So i am pretty happy on having discovered that.

Thursday, December 6, 2007

Command of the day: svk ls

Why not calling them command-of-the-year as the one and only command-of-the-day i posted until now was posted one year ago? Well, they are intended to be commands that saved me the day like the one i explain today.

On my previous entry, I explained that Pau and me were publishing on sourceforge a python script, WiKo, that we used to edit our latest articles and thesis. Because we used it in such context, latest version have been commited in a private repository we have for articles and the like.

I wanted to move the files to a public repository at sourceforge. Sadly, svnadmin dump just works when you are in the same host as the repository is and we have no regular ssh access to the subversion. We could bother our admin, Jordi Funollet, but he is always very busy and we are always requesting him. (Thanks, Jordi for your support!!)

Luckily i found a nice entry blog on how to use svk to remotely dump a svn repository.

You need to install svk, in debian based:

 sudo apt-get install svk 

Then we should create a mirror repository with svk just containing the revisions concerning the selected path

 svk ls svn+ssh://username@svn.myserver.com/path/to/repo/sub/path/to/wiko 

svk will ask you several questions:

  • Choose a base URI to mirror from (press enter to use the full URI): To filter by path, i recommend to do such filtering on the command line by specifying the full path, so return.
  • Depot path: [//mirror/wiko] Is very important here to change the depot name so you can skip that '//mirror' part, so answer //yourproject beginning with a double slash!!
  • To end up, it asks which revisions do you want, normally the answer is 'a' (all)

Then dump the local svk repository:

 svnadmin dump ~/.svk/local > my-repository-dump 

And now you can import it into sourceforge as normal:

 svnadmin load [the-new-repository]  < my-reposotory-dump 

With sourceforge the process is more complex. You should upload the dumpfile to the shell server. For example, for wiko it was at: sftp://wiko.sourceforge.net/home/groups/w/wi/wiko/

Then you should specify the uploaded dump file as it was a cvs to svn migration. It last a while, after that you can checkout wiko code!

 svn co https://wiko.svn.sourceforge.net/svnroot/wiko/wiko 

I lost the history of the file before a move operation, but at least we recovered some of the history.

Taking a look at the man page you can see that svk is really helpfull for other purposes, but definitely, its ability of dumping a remote repository saved my day :-)

WiKo, the wiki compiler

Pau and me have just released WiKo as sourceforge project. WiKo is a simple but powerful Python script that takes files with wiki content on a given directory and either builds a web, a LaTeX article or a blog.

The script has a very long history before being branded as WiKo. It has its roots on the simple script i did to build most of my websites (Can Voki, Codders, KKEPerians UNLTD...) The script eased keeping static a lot of pages sharing the same layout design. The script was later enhanced to read wiki pages and to build also LaTeX articles. Even I used it for my master thesis. Recently Pau also started to use it for his PhD thesis and he has added great improvements such as LaTeX formulas in HTML output or bibliography linking. And lately we started to use it as internal wiki for some projects we are involved within Barcelona Media. So we thought that such script deserved a sourceforge project, and that's how WiKo born. I hope it can be useful to you all.

WiKo might be useful to you if:

  • You have an static website and you are considering to make it dynamic just to reuse design
  • You have to write articles to be available in both html and pdf
  • You are moving content from web, to LaTeX articles and to blogs and you are feed up of translating markup
  • You love wiki syntax but you hate to edit wikis in web forms
  • You like LaTeX output but you hate (or don't know) its markup
  • You like colaborative environments such subversion and want to use it as base for a wiki
  • You still are considering ikiwiki but you would like it was not written in Perl so you could contribute to it ;-)

Also I am currently working hard on being able to use WiKo to edit this blog. I am almost there so stay tunned.