ipyclam: interactive python console for CLAM networks manipulation

Xavier Serra (not the MTG head, but an homonym student at the UPF) is about to reach an interesting milestone on his Final Career Project related to CLAM. He is developing an interactive Python console for CLAM named ipyclam. It is a text based interface to explore and manipulate CLAM networks. Hopefully this will end being a dock widget within the Network Editor but it is also useful as interactive text console tool and as programming interface, which are the first useful outputs Xavi is about to get. ipyclam comes to cover the need of manipulating complex networks in cases when point and click is just tedious and also that of generating parametrized networks.

Let's make clear that it is not a full wrapper for the CLAM API like Hernan's pyclam, ipyclam just enables high level operations you usually do with Network Editor: instantiating, connecting, configuring and playing modules. Moreover we have intentionally decoupled ipyclam API to any existing CLAM API in order to define the most pythonic, convenient, and clean API we could think. For example the code to build an stereo cable would be:

net = Network()
net.processing1 = net.types.AudioSource
net.processing1.NOuputs = 2
net.processing2 = net.types.AudioSink
net.processing2.NInputs = 2

net.processing1.Output1 > net.processing2.Input1
net.processing1.Output2 > net.processing2.Input2

net.play()

An interesting feature, is that network.code() just returns an string with similar python code, so a network knows how to regenerate itself. This opens the door to python based format for CLAM networks. Bye bye XML!.

Convenience? Yes, please. Connections can be done in batch, broadcasting connections from all the connectors in one processing to all or to a single connection of another:

net.processing1 > net.processing2 # all to all
net.processing1.Output1 > net.processing2 # one to all
net.processing1 > net.processing2.Input1 # all to one

Convenience is good but all things should be doable. Often most convenient solution is not the best suited for all the cases. Accessing configuration parameters and connectors of any kind directly from proccessing is quite convenient but names can collide, to solve that some special attributes are provided to restrict the name scope.

network.processing1._config.NOutputs
network.processing1._inports.InPort1
network.processing1._outports.OutPort1
network.processing1._incontrols.InControl1
network.processing1._outcontrols.OutControl1

Connection names can have spaces, and attributes cannot, so we also provide an alternative subscript syntax.

net.processing1['Output 1']

One of the features I like the more is tab key exploration.

• network.[tab] and you get the list of processings
• network.processing._inports[tab] and you get the list of input ports
• network.processing._config[tab] and you get the list of configuration parameters
• network.processing.[tab] and you get connectors and cofiguration parameters
• network.types.[tab] and you get the list of available processing types

It works for interactive console with tab completion like ipython. We also provide ipyclam_console, an ipython based shell conveniently importing modules and adding a network to start playing with.

For more details on the API just check this document.

The implementation

ipyclam is defined in two layers. The front-end layer, purely written in python, makes the convenient API happen: Exploration, dynamic object attributes, many interfaces to similar operations... This upper layer, relies on the lower, less pythonic and more purpose oriented one, the network proxy layer, which provides access to the state and operations of the actual network with quite fewer entry points.

Xavi defined the front-end layer while evolving, in parallel, a dummy proxy layer simulating network status with basic python structures. That gave us the flexibility to explore and evolve both the frond-end and the proxy interface without bothering about the actual CLAM implementation that would have made such exploration quite rigid.

Two layers approach is also convenient because by defining the ops in the network proxy layer, we are able to reuse the front-end API with any other patching like environment. I am thinking on JACK or environments like ingen.

Status and road map

Xavi just finished most important bits of the API: Inspection, module creation, module type enumeration and module inter-connection. Currently we still rely on a dummy network proxy, but a CLAM based proxy will arrive during next week. Configuration API seems to be controversial and we are holding it until we have something implemented with CLAM.

Once we have a full CLAM based proxy the next big goal is to integrate the console in NetworkEditor. We need a console widget for Qt which is able to execute a python interpret. We found some candidates:

• qconsole which already has a python wrapper
• qtermwidget a more robust console (based on KDE Konsole) but without direct support for python, and
• ipython qt frontend quite experimental and using a weird threading model, but still interesting as ipyclam_console already uses ipython and it is awesome

So either candidate seems to require some work on our side.

Well, I hope you will find this new CLAM feature as amusing as I do. Xavier is really doing quite a good job, congrats and keep it that high.

Numpy arrays to video

I found some examples on how to generate video with python by piping frames to external programs like ffmpeg or mencoder. All those snippets encode each frame as a given image format (png, jpg...) by using PIL, matplotlib.. to get them. Besides the dependency, building figures or encoding images can be quite slow. I finally found a way of piping raw numpy buffers as frames to mencoder. You can use the following code:

import subprocess
import numpy as np

class VideoSink(object) :
        def __init__( self, size, filename="output", rate=10, byteorder="bgra" ) :
                self.size = size
                cmdstring  = ('mencoder',
                        '/dev/stdin',
                        '-demuxer', 'rawvideo',
                        '-rawvideo', 'w=%i:h=%i'%size[::-1]+":fps=%i:format=%s"%(rate,byteorder),
                        '-o', filename+'.avi',
                        '-ovc', 'lavc',
                        )
                self.p = subprocess.Popen(cmdstring, stdin=subprocess.PIPE, shell=False)

        def run(self, image) :
                assert image.shape == self.size
#               image.tofile(self.p.stdin) # should be faster but it is indeed slower
                self.p.stdin.write(image.tostring())
        def close(self) :
                self.p.stdin.close()

It is a tenth faster than the PIL based one. And there is no fair comparison to the matplotlib one. I got it working with mencoder but I could not figure out how to make it with ffmpeg.

You can find that code with an usage example there. Being based on other public domain snippets consider it also public domain.

I am using that class to save the output of my Freenect based project. More on that on next entries.

Libraries becoming plugins, plugins becoming libraries

Let me post about some of the latest changes on the CLAM build system.

The CLAM project has been distributing two kinds of shared libraries: the main library modules (core, audioio and processing) and a set of plugins which are optional (spacialization, osc, guitareffects...). Plugins are very convenient since programs like the NetworkEditor can be extended without recompile and enable third party extensions. They are so convenient that we were seriously considering splitting processing and auidioio as plugins.

But the use of content of a clam plugin is limited to abstract Processing interface. No other symbols are visible from other components. That has a serious impact on the flexibility to define plugin boundaries. For example, if a plugin defines a new data token type, any processings managing that token type should be in that same plugin. To use symbols of one plugin from another implies installing headers, provide soname, linker name and so on. That is, building a module library.

So, if plugins want to be libraries and libraries want to be plugins, let them all be both. Each module will be compiled as library (libclam_mymodule.so.X.Y.Z), with soname (libclam_mymodule.so.X.Y), linker link (libclam_mymodule.so), headers (include/CLAM/mymodule/*), pc file... and it will also provide a plugin library (libclam_mymodule_plugin.so) which has no symbol by itself but is linked against the module library. When a program loads such a plugin, all the processings in the module library become available via the abstract interface. On the other side, if program or a module needs to use explicitly the symbols of another module, it just has to include the headers and link against the module library.

I just applied the changes to the plugins and seems to work nicely. Adapting the main modules will be harder because the old sconstruct file, but it is a matter of days and we can split them. I moved a lot of common SCons code to the clam.py scons tool. There is a cute environment method called ClamModule which generates a clam module including the module library, soname, linker name, plugin, pkg-config, and install targets. I also added some build system freebies for the modules and applications:

• All the intermediate generated code apart in a 'generated' directory
  
• Non verbose command line
  
• Colour command line
  
• Enhanced scanning methods with black lists

Well, do not rely too much on the current clam scons tool API. Changes are still flowing through the subversion. Of course, warn me if I broke something. ;-)

Boolean Controls in CLAM

After a long refactoring to get typed controls into clam without breaking anything, we already have them. Kudos for this achievement go also to Francisco (who started the whole thing as its GSoC project), Hernan, Nael and Pau.

Now controls are defined just like ports with a type. So for example, you can define an out control being the type OutControl<MyType>. As a side effect, control callbacks came in natural way. Instead of using a different class (formerly InControlCallback<MyType, HostProcessingType>), now you just have to pass a processing method to the control constructor. Templates do the magic too, but that is hidden from the processing programmer API which is cool. See, some example bellow.

So, once we enabled typed controls working, now is time to have more than just float controls. The main problem now is that in order to get some useful new control type you have to modify the network editor and the prototyper to make them useful.

Although the most demanding needs are for enum and integer controls, I started with simpler bool controls. My goal is you don't have to modify the NetworkEditor or the Prototyper to introduce a new control type. That is, a plugin could add new control senders, control displays, default connected processings (double clicking on a control) and binders for the prototyper (to locate ui elements to link to processing controls). So i wanted a new simple use case not being float to explore a feasible API.

Below you can see a network with a BoolControlSender (widget hxx/cxx, processing hxx,cxx) and a BinaryCounter (hxx,cxx) outputing into two BoolControlDisplay (widget hxx/cxx, processing hxx,cxx). Maybe is a little late for using CLAM for your Christmas lights, i guess ;-)



Currently, instead of bools we use floats considering a threshold for being true or false. A new ControlGate processing provides a transition by doing such translation:

So now is time to look to the code I had to add and see what can be enhanced to ease adding new control types:

• One of the things I didn't liked during the implementation is having to add an entry into a long list of 'if' statements in ProcessingBoxEmbededWidget.cxx. This is clamming for a refactoring into a Factory.
• Also the menu entry filling (to connect to new processing) and the default create-and-connect action become a list of if clauses with the type as parameter.
• Both, prototyper binders and embedded widgets had to duplicate the control sending code. That could be generalized into a binder object that both use.

So, it is clear that there is room for a lot of enhancement and it looks like those enhancements could also be applied to ports as well :-)

Command of the day: git svn (Using git over a subversion repository)

This may be the end of a long quest for a tool to do 'offline commits on subversion'. That is, being able to work with small commits when you are not able to commit because you are off-line (on-flight or on-train hacking) or you are not a commiter yet (menthorship).

The fact is that, we can benefit of some nice features of git without having to migrate from subversion. I found a very interesting blog entry explaining that. I compiled my own notes for CLAM:

Install the required packages:
sudo apt-get install git-svn

Configure your user:
git config --global user.name 'David García Garzón'
git config --global user.email 'yo@alli.com'

Create a git clone of the svn:
git svn clone http://dgarcia@clam-project.org/clam --stdlayout -r HEAD
Notice that i skiped the 'trunk' part of the url we normally use. '--stdlayout' is to interpret trunk/ branches/ tags/ as such. '-rHEAD' avoids cloning the whole history which is faster.

Work with the repository as with a normal git one, avoiding remote operations (push/pull). Git can fustrate SVN users with a lack of shortcuts (ci, di, up...). You can define them at will:
git config --global alias.ci "commit -a"
git config --global alias.di diff
git config --global alias.stat status
git config --global alias.up "svn rebase"
git config --global alias.rci "svn dcommit"

To update your git repository to server changes:
git svn rebase
If you have uncommited local changes you have to 'stash' them first.

To apply all the local commits to subversion, once rebased:
git svn dcommit


To generate, instead, an incremental set of patches:
git format-patch -M -C -s --inline --stdout remotes/trunk > mypatch.mbox

Note that this is not a regular patch file but an mbox file containing several mails, one per commit, each with the patch attached and some aditional information such as the commit message.

Those commits can be sent to the menthor and applied to git sandbox with svn write access.
git am mypatch.mbox

An then the menthor can commit using previous command 'dcommit'. That easy... or not so.

Some other highlights i would do on git: Nice programmer oriented subcomands such as git bisect, to help bug hunting by doing a dicotomic search on revisions, or git stash, to discard temporarily uncommited changes and recovering them later, or 'git grep', which executes grep ignoring git control files and generated files. Tools like StGit and guilt that allows to edit the order of the local commits considering them a serie of patches similarly to quilt.

VST plugins with Qt user interface

I recently did an spike on what we need to make VST plugins first class CLAM citizens. CLAM allows to visually build JACK and PortAudio based applications with Qt interfaces as well as GUI-less VST and LADSPA plugins. The more flashy feature of VST is user interfaces that are mostly built using VSTGUI. We are using Qt as interface for JACK and Portaudio based apps because we are using the nice features of Qt toolkit to dynamically bind the UI elements and the underlaying processing. Moreover, Qt styling features enables shinning designer-made interfaces. Why not being able to reuse the same interface for VST and JACK? That has been a long standing TODO in CLAM so now is time to address it.

In summary, we fully solved croscompiling vst's from linux and we even started using qt interfaces as vst gui. In that last point, there still is a lot of work to do, but the basic question on whether you can use qt to edit a vst plugin is now out of any doubt.

To make the spike simpler, and in order not to collide with other CLAM developers, currently working on it, i just left apart all the CLAM wrapping part, just addressing vst crosscompiling and Qt with the sdk examples.

Cross compilation was pretty easy. This time I found lot more documentation on mingw and even scons. Just by adding the crossmingw scons tool we are already using for the apps and i managed to get Linux cross-compiled plugins running on Wine.

Adding a regular vstgui user interface is just a matter of compiling vstgui sources along with the example editor that comes in the sdk.

Once there, we should address Qt. VSTGUI is just a full graphical toolkit implementing the 'editor interface' plus a toolkit with some provided widgets and, i guess, a way of automating the binding of controls to processing. So what we need for qt is to implement the AEffEditor interface using the qt toolkit instead. The first problem is about the graphical loop. You have to create a QApplication and calling qApp::processEvents() on the editor's idle method so that qt widgets get responsive. The problem then is that, if you don't provide a QWidget as parent to your interface, it becomes a top level window ignoring the host provided window that still appears as an empty one.

VST host provides such window as a native Windows handle. How do you create a widget on an existing window handle? Months ago trolls redirected me to a commercial solution. Not such a 'solution' for us, a FLOSS project. So i was digging in windows qt source code for a hack when i found the answer just at the public and multiplatform QWidget api. QWidget::create works like a charm. The following simple class is a native window wrapper you can use as a regular QWidget.

class QVstWindow : public QWidget
{
        Q_OBJECT
public:
        QVstWindow(WId handle) {create(handle);}
        QVstWindow::~QVstWindow() {}
};

Still there are some issues: focus handling, reopening, drag&drop... But the basic mouse clicking and resizing works

Once i got that, loading a designer ui file was very easy.

As I said there are still many caveats to solve. A matter of playing with it and refining things. Here is a list of TODO's:

• Communicate controls from and to the interface
• Handle focus and other events properly
• Build a CLAM network wrapper which reensembles more the one for LADSPA
• Wiki documentation on how to build your own plugin
• One button plugin generator like the one we have for LADSPA ;-)

I feel that there is more people around other projects interested in using Qt for VST plugins so this is also a call for collaborative research on pending issues, at least the generic ones. Contact us on the CLAM development list or for a broader audience in the Linux Audio Developers list.

Command of the day: interdiff

Often, when working in code projects, I have to keep some local changes uncommited because I am offline or because i am doing some experiment and i am not sure that it will be successful. In those cases you end up doing a big commit with has very low grain to rollback any regressive change. Sure new distributed VCS allow offline commits but we are working with subversion. So I was glad when i recently discovered how to use interdiff command.

If you plan to make a set of offline changes, on each point you would commit, just take an 'svn diff' on a file. Given those accomulative diffs respect the BASE revision, interdiff knows how to extract the incremental ones from one state to the next one so. When you are back online, you revert all changes, apply and commit each patch separately.

Other interesting commands to manage patches (available in the patchutils package):

• combinediff: the reverse, joins two patches together
• recountdiff: very useful when you want to reapply a patch but the state of the code has changed.
• flipdiff: exchanges the order of two pages
• filterdiff: remove the changes affecting a set of files from an existing patch

No excuses for coarse commits after on-flight coding to Parma!

Refactoring script for CLAM networks

Now that the CLAM NetworkEditor has become such a convenient tool to define audio processing systems, we started to use it massively instead of C++ code to integrate processings. Even thought, we started to find that whenever we change a configuration structure, a port/control name, or even a processing class name, the networks xml had to be edited by hand because they didn't load. That problem led us to avoid such kind of changes, which is not a sane option. 'Embrace change', agile gurus said, and so we did.

We have developed a python tool to support network refactoring. It can be used, both as a python module, or as a command line tool, to batch modify CLAM network files using high level operations such as:

• Renaming a processing class name
  
• Renaming a processing name
  
• Renaming processing ports or controls
  
• Renaming a processing configuration parameter
  
• Removing/adding/reorder configuration parameters
  
• Setting configuration parameters values
  

The script just does XPath navigation and DOM manipulations in order to know which pieces need to be changed. Each high level command is just a couple of lines in python.

We are starting to use it to:

• Adapt to changes in the C++ code
  
• Change network configuration parameters in batch
  
• Provide migration scripts for user's networks
  

About the last point, we plan the next release to provide network migration scripts containing a command set such as:

  
ensureVersion 1.3
renameClass OutControlSender NewClassName
renameConnector AudioMixer inport "Input 0" "New Port Name 0"
renameConfig AudioMixer NumberOfInPorts NInputs
setConfigByType AudioMixer NInputs 14
upgrade 1.3.2

Still some short term TODO's:

• Include the clam version in the network xml so that the ensureVersion and upgrade commands work.
  
• Integrating also Qt Designer UI files in the refactorings
  
• Add some other commands as they are needed
  

Happy CLAM networks refactoring!

Managing Audio Back2Back Tests

So long since last post, and a lot of things to explain (GSoC results, GSoC Mentor Submit, QtDevDays, CLAM network refactoring script, typed controls...). But, let's start explaining some work we did on a Back-to-back system we recently deployed for CLAM and our 3D acoustic project in Barcelona Media.

Back-to-back testing background

You, extreme programmer, might want to have unit tests (white box testing) for every single line of code you write. But, sometimes, this is a hard thing to achieve. For example, canonical test cases for audio processing algorithms that exercise a single piece of code are very hard to find. You might also want to take control of a piece of untested code in order to refactor it without introducing new bugs. In all those cases back-to-back tests are your most powerful tool.

Back-to-back tests (B2B) are black box tests that compare the output of a reference version of an algorithm with the output of an evolved version, given the same set of inputs. When a back-to-back test fails, it means that something changed but normally it doesn't give you any more information than that. If the change was expected to alter the output, you must revalidate the new output again and make it the new reference. But if the alteration was not expected, you should either roll-back the change or fix it.

In back-to-back tests there is no truth to be asserted. You just rely on the fact that the last version was OK. If b2b tests get red because an expected change of behaviour but you don't validate the new results, you will loose any control on posterior changes. So, is very important to keep them green or validating any new correct result. Because of that, B2B tests are very helpful to be used in combination of a continuous integration system such as TestFarm, that can point you to the guilty commit even if further commits have been done.

CLAM's OfflinePlayer is very convenient to do back2back testing of CLAM networks. It runs them off-line specifying some input and outputs wave files. Automate the check by subtracting the output with a reference file and checking the level against a threshold, and you have a back-to-back test.

But still maintaining the outputs up-to-date is hard. So, we have developed a python module named audiob2b.py that makes defining and maintaining b2b test on audio very easy.

Defining a b2b test suite

A test suite is defined by defining back-to-back data path, and a list of test cases, each one defining a name, a command line and a set of outputs to be checked:

#!/usr/bin/env python
# back2back.py
 from audiob2b import runBack2BackProgram
 data_path="b2b/mysuite"
 back2BackTests = [
  ("testCase1",
   "OfflinePlayer mynetwork.clamnetwork b2b/mysuite/inputs/input1.wav -o output1.wav output2.wav"
   , [
    "output1.wav",
    "output2.wav",
   ]),
  # any other testcases there
 ]
 runBack2BackProgram(data_path, sys.argv, back2BackTests)

Notice that this example uses OfflinePlayer but, as you write the full command line, you are not just limited to that. Indeed for 3D acoustics algorithms we are testing other programs that also generate wave files.

Back-to-back work flow

When you run the test suite the first time (./back2back.py without parameters) there is no reference files (expectation) and you will get a red. Current outputs will be copied into the data path like that:
b2b/mysuite/testCase1_output1_result.wav
b2b/mysuite/testCase1_output2_result.wav
...

After validating that the outputs are OK, you can accept a test case by issuing:
$ ./back2back.py --validate testCase1
The files will be moved as:
b2b/mysuite/testCase1_output1_expected.wav
b2b/mysuite/testCase1_output2_expected.wav
...

And the next time you run the tests, they will be green. At this point you can add and commit the 'expected' files on the data repository.

Whenever the output is altered in a sensible way and you get a red, you will have again the '_result' files and also some '_diff' files so that you can easily check the difference. All those files will be cleaned as soon you validate them or you get back the old results.
So the main benefit of that is that the expectation files management is almost automated so it is easier to maintain them in green.

Supporting architecture differences

Often the same algorithm provides slightly different values depending on the architecture you are running on, mostly because different precision (ie. 32 vs. 64 bits) or different implementations of the floating point functions.

Having back-to-back tests changing all the time depending on which platform you run them is not something desirable. The audiob2b module generate platform dependant expectations by validating them with the --arch flag. Platform dependant expectations are used instead the regular ones just if the ones for the current platform are found.

Future

The near future of the tool is just being used. We should extend the set of controlled networks and processing modules in CLAM. So I would like to invite other CLAM contributors to add more back2back's. Place your suite data in 'clam-test-data/b2b/'. We should decide where the suite definitions themselves should be placed. Maybe somewhere in CLAM/test but it won't be fair because dependencies on NetworkEditor and maybe in plugins.

Also a feature that would extend the kind of code we control with back-to-back, would be supporting file types other than wave files such as plain text files, or XML files (some kind smarter than just plain text). Any ideas? Comments?

Exterminating signals and slots

I recently posted at the clam devel wiki some advices, conventions, traps and tips on programing with Qt within CLAM addressed to the GSoC students. The most painful trap is the one of gratuitous signals and slots usage. Signals and slots is a very powerful way of designing independent components that comunicate each other with low coupling. Is that powerful that it is very tempting for novices to use them everywhere and this can turn very harmful for you.

If a connected slot doesn't exist, you just get an error console on run-time, checks on slots are that soft, and this is bad. Also the signal slot resolution is more expensive than just a method call. But the main reason to avoid them is that they make the code very hard to follow. When you see a signal 'emit' you have to find where such signal is connected in order to find which are the objects and slots that will be called. Multiply this process by nearly 200 signal emisions that were done at the vmqt library Annotator is using and you'll understand why such components although very smartly designed in structure, they are very hard to maintain and use.

You should emit signals just when you don't know which is the receiver of an event. If you can guess the receiver, there is no use for signals. On the other side, if you cannot access the emitter code, or you don't want to couple it, then there is room for a 'connect'.

A bad smell for noticing that you are over using sigslots is having in a class you are writting a connection like this:

connect(this, signal, knownWidget, slot)

and later, maybe in a different method:

emit signal()

When that known, and later forgotten, object is the only one you are connecting to the slot you might want to just keep a reference to it and just call the slot instead of emitting the signal.

knownWidget->slot()

which is faster, compile time checked and much more traceable.

Given that bad smell locator at hand i decided to do a fast review of vmqt module. vmqt is the successor of the many Visualization Modules we had in CLAM, just annother definitive VM rewrite. I tried to use it several times, but it is very hard to have a global view of what it does because all the program flow is driven by signal connections. No joke: 180 signal emissions and a similar number of 'connect's in 30 classes.

After a first review, 110 signals emisions have been avoided just by having a pointer to the Plot2D at the Renderers (the objects that represent drawing layers of the plot such as the lines, the playhead, the grid...). Having a pointer to it, Renderers can do a direct call to the Plot2D slots (now regular functions) instead of emitting a signal whose only receiver is the plot.

70 'emit's are still wandering around. Some of them communicate the plot with the wplot (a widget containing the plot and other elements such as the sliders, the rulers...) so they are used to syncronize. Those are likely to be removed in a similar way than for renderers.

The rest are specific renderer signals that are connected and propagated by the specific wplot. Those are harder to remove and indeed they are very convenient to keep.

As the extermination goes on, one can better see how to really take profit of the nice vmqt module structure and which aspects can be enhanced.

A configuration idiom for Python scripts

For our Python scripts we have used a fair number of idioms and variations such as importing a config.py file with the parameters as module global vars or importing the content of a dictionary. A different one we used lately is very useful and simple:

# This should be your program using the config

import os
class config :
    #Here we define the default parameters
    paramA = "value A"
    paramC = "value C"
    class category1 :
        paramC="value sub C"

    #Here we load new values from files, if they exist
    for configFile in [
        "/usr/share/myprog/defaultconfig",
        "/etc/myprog.conf",
        "~/.myprog/config",
        "myconfig.conf",
    ] :
        if not os.access(configFile,os.R_OK) : continue
        execfile(configFile)

# That's the config params usage. Pretty!
print config.paramA
print config.paramB
print config.paramC
print config.category1.paramC
print "This is a template: %(paramC)s" % config.category1.__dict__

If you write in the config file this:

# myconfig.conf
paramA="new value A"
paramB="new value B"
category1.paramC="new subCvalue"
paramB+=" extension " + paramA

you'll get this

new value A
new value B extension new value A
value C
new subCvalue
This is a template: new subCvalue

The config file is read as you were adding code at the same indentation level you have on the 'execfile' call.

Notice the advantages:

• Your config file looks like var assignations


• You can use inner classes to build up categories


• You can have a list of configuration locations with different precedence


• You can include almost whatever python code


• You can do templating with the params getting a dictionary like config.__dict__ or config.category1.__dict__


• You can put config checking code after loading.
  

Be carefull on unreliable contexts:

• Malicious config files can include almost whatever python code


• Config syntax errors crash the program (i guess that can be solved)


• Config files may add any new attribute, category, or method you didn't have

But if you are just managing your own utility scripts like us, that idiom is fantastic.

Command of the day: hotspot2calltree

Today, the command of the day is hotspot2calltree.

I do like a lot using kcachegrind to tune my C++ code. You can use KCacheGrind to navigate through the actual function calls in a given execution of your program and seeing very graphically where the time is spent.

Today i needed to optimize some python code but that's not C++ code. No problem. Add in your code this:

import hotshot
prof = hotshot.Profile("profile.hotspot")
prof.runcall(myFunction)
prof.close()

And then, at the shell:

sudo apt-get install kcachegrind-converters
hotspot2calltree -o profile.callgrind profile.hotspot
kcachegrind profile.callgrind

And now you get a nice kcachegrind profile you can navigate on.

Preparing for GSoC 2008

GSoC 2008 is already here! We are preparing our submision for CLAM as organization and I hope we are as lucky as last year. For GSoC 2007 we got 6 fervent students who pushed CLAM a big step forward. We still don't know whether we will be selected as organization or not. We haven't even filled the submision data. But it is time to trigger some resorts. So, what to do now?

If you are an experienced CLAM developer, please consider becoming a mentor. The more mentors the more students we can cope with.

If you are a user, is the time to push your favourite feature into the GSoC project proposals.

If you are an student wanting to be part of the program, I advice you to get involved with the project from now as we will consider early involvement a big plus for eligibility.

If you are Xavi, Pau or myself, then you should fill CLAM submision instead of blogging ;-)

I love summer.

Comand of the day: kig

Each task has a tool. With graphics and figures too. If i want to edit a photograph or to perform some nice effects on an existing image, that's a task for Gimp. If I need some cute artwork for icons, banners, web... I prefer to vector with my so loved Inkscape. When such drawings are not so artistical and need some related and formal figures and diagrams, i normally consider dia, limited but correct. If i want to plot a complex graph i let the task to graphviz's dot and i just declare the vertexs. If I want to plot some program data output, python plus matplotlib is the faster way to process and render it. Automating execution to visualization process is very convenient. Some time ago I used gnuplot for that but python is more flexible towards data formats. Often what you need is to explore such data, interactive visualization can be revealing with tools such as qtiplot.

But my problem today wasn't none of the above but drawing some geometric diagrams (angles, vectors, tangents...) to illustrate some trigonometric equations. I was thinking on QCad but Kig did perfectly. Kig is about doing geometrical manipulation: intersections, angle transportation, angle mesurements, solidarized elements, python scripting...



I was to integrate kig png exporting into WiKo figure generation but i found two show-stoper problems: Command line options for batch exporting seems not to work and there is no way to control the viewport but by resizing the windows and controlling the zoom. While the concept of limited size canvas exists, I didn't find a way to resize it :-( Anyway is worth to dig in such a tool even doing patches to have batch exporting working.

Impressions on Django (II): Development environment

On my previous post, I explained some basic ideas on how Django works. The post explained, mostly the programming model which is a clever implementation of the classical Model-View-Controller pattern. Not that new. But compared to other web development environments I worked with (PHP, Zope/Plone or plain mod_python), Django is a clear improvement because is both simple and pragmatical.

The programming model is something that has a direct impact on how you design the product, but there are other factors that determine how easy a developer can develop within a platform. In this post I arge about such factors, how Django afects the development workflow.

Editing and controlling source code

Compared to Zope, another python web application environment I've been fighting against, Django is a clear step forward to control your work.

First of all, the developer has the control of the source. Files are stored on the file system! It may seem an obvious assertion, but, if you have been using Zope you will know why I am so happy with that: Zope forces you to store all the code into a large binary file which acts as virtual filesystem. You cannot use a regular version control system such subversion to control your changes. ...and worst, it forces you to input code in web forms!! Argh!! Django ends with such a nightmare. Files are back to the file system and to your preferred editor.

Modularity

Still Django applications are modular in the same sense that Zope: you can combine several applications (modules) on a single site. For example, the administration interface is an application itself that can be used to edit any model of any installed application. You can disable it or enable it as a whole or for specific models. There is also an authentication application that can be used to transversally control the access to your application features. Every other application can use the authentication application to obtain information about validated users and to limit the access to certain features depending on the user profile. Session management and per-session storage, RSS feeds, SiteMap files, data mining interface... they all come for free, provided as applications. Besides the included applications, a nice application repository is available.

Debug cycle and exploration

Debugging your code is also very straight forward. While developing with Django, you are deploying a development Web Server not related to Apache, that you start from the console, and it is not visible from the outside. This enables each developer having her own instance of the web site. No administration passwords are needed and reloading is easier. Every message you print you'll get it in the console so it is ideal for debugging by tracing compared to using plain mod_python which runs on the apache server and you are to monitor the apache error log. Also, when some code fails with an exception, Django constructs a very handy web page with a lot of information: the back-trace enriched with code context, parameters and local variables inspection, all the environment values, the request values... All that available in a collapsible error web page. Lately I've been tempted of importing a failing python code into Django just to get such a big amount of information.

Also by running './manage.py shell' you get a python shell executed in the same conditions your code will be. Indeed the 'manage.py' script has very useful subcommands to infer model definitions from existing databases, to see database definitions from models, to import and export test data fixtures...

The next post

So yes, Django is a mod_python with gears and a Zope without the fat. However, while I don't regret choosing Django, it is not that perfect. On my next post, the last one on those first impressions on Django series, I will explain the problems we found while using the framework. The limitations we found and how it could be (is being) improved.

Well, maybe not the next post. Some other topics have appeared while writing this series of posts and they are likely to require an entry.

Impressions on Django (I)

During the last months, I've been involved in a project on (text) information retrieval. The project required to develop a web interface over a python core and we chose Django as framework. Other option was using plain mod_python as we did for EfficiencyGuardian. But some time ago, in the context of the SIMAC project, Xavier Oliver, who was the responsible of implementing BOCA, CLAM Annotator's collaborative back-end, used Django and he was very enthusiastic on how rapid he had all the web working. So I wanted to give it a try for our current work. Here I am posting my first impressions as newbie django user. What Django has to offer?

Database abstraction

The most important part of Django is a persistent object model which maps python objects into database entities giving you a nice abstraction on the database layer. By defining such model classes, you get an object oriented programming interface to query and change related database tables, and even navigating and joining through relations as they were regular connected objects.

I normally dislike too transparent interfaces when they deal with efficiency sensible things such as database access. But Django states very clearly when and how such access is done allowing you to control it but using a high level object oriented idiom.

Administration interface

You can enable an existing administration module for your site. That is a web interface to create, edit and remove your model objects. You can control the way they are edited by adding extra properties to the model classes attributes.

Attributes have more types than real SQL types. Fields can be, for example, URL's, emails, telephone numbers or zip codes, and Django gives you for free custom validation and custom administration interfaces for them.

The administration interface also considers table relations providing, for instance, web interfaces for choosing related objects by foreign key and buttons to add a new related objects.

Application logic

Often, as in our case, the administration interface is pretty close to what the final application will be. But direct manipulation of a model is not enough for most web applications. Normally you need some additional application logic: Which functionalities are presented to the user, and which is the user dialog with the system to perform such functionalities.

Three elements are combined to build up such application logic in Django: URL mappings, views and page templates. URL mappings map regular expressions of requesting URL into calls to python functions. Such pỳthon functions are the views which perform the needed actions on the system and construct an output web page. Views usually inject python data into an HTML skeleton, the page template, to generate the response web page.

Django provides some convenience views to create, update, delete and listing model objects, also supporting common features such as pagination, date based browsing, validation and destructive action confirmation.

The next entry

Here i explained the basics of Django execution model. On the next entry i'll write some impressions on Django as development environment compared to other environments i used for web development such as Zope, mod_python and vanilla PHP.

Qt 3 and Qt4 relicensed as GPL v3

I read in Thiago Maceira's blog that Trolls have relicensed Qt3 and Qt4 as GPL v3.

I still had no time to analyze how this will affect clam. Relicensing CLAM has proven to be a hard political and burocratic problem, but a transition from GPL v2 to GPL v3 is easier since we have the 'or later' notice, so we have the door open to that controversial upgrade. Definitely, trolls' move to v3 is something that will boost the adoption scene.

A nice lateral consequence of this affects directly to one CLAM application, SMSTools, which still uses Qt3, and not Qt4 (althought Zack Welch started a temptative Qt4 port). Until now, Qt3 didn't enjoyed the same nice dual licensing Qt4 has. Former Qt3 licensing was a modified GPL that had problems on using Qt3 for non-unix platforms such as windows. So with Qt3 relicensing we can now freely distribute SMSTools precompiled binaries for Windows.

Update: I read in the official announcement that they didn't droped GPLv2, they just added a third license to the existing dual licensing so you can use one of three licensing schemes: non-free, GPLv2 and GPLv3. Definitely they want to make developers live easier, not just by building excellent API's.

BiBTeX Tooltip in html WiKo output

I was busy yesterday on providing better BiBTeX support for the HTML output in WiKo. I was adding bibliography tooltips. They were an original suggestion from Pau Arumi, and i agree with him that they will be very handy for reading an article in electronic format. The former html bibliography page is still available: just click instead of hovering.

To have it working, you should download the latest wiko version, install 'python-bibtex' debian/ubuntu package and appending to your css style sheet the latest statements of this css concerning 'bibref' class.

Then, just place any bibtex files on the same folder the wiki files are and refer a bibliography entry in the wiki files as '@cite:SomeBibtexId'. By 'WiKonpiling' you'll get both normal LaTeX bibliography and this html equivalent.

Take a look on the final feel at this chapter of my master thesis.

Iterating properly in Python

After more than a year of python hacking you take a look at the official Python tutorial and still discover new ways of enhancing your code. Just two ways of saving list constructions on our actual code.

The first one: In our code, we are currently doing all the time something like this:

for key, value in myDictionary.items() :
        doWhatEverWit(key,value)

This is not that appropiate because it creates the tuple list and then iterates. But if you do:

for key, value in myDictionary.iteritems() :
        doWhatEverWit(key,value)

You just get fetched keys and values without creating any temporary list.

The other thing our code is full of is constructing a list using list comprehensions:

tempList = [expresionUsing(item) for item in collection]
for item in tempList: doWhatEverWith(item)

I knew about list comprehensions and i knew about generator functions. Generator functions are functions can be a iteration source by doing yield instead of return. yield returns a value but keeps the function execution state for the next call. That is a very interesting feature but i never used it because it is harder to create a function that yields than to create the loop itself.

But by reading the tutorial i found a brand new kind of expressions that sum up list comprehensions and generator functions: generator expressions. They have the same syntax than list comprehensions using parethesis instead of brackets, but instead of creating a list they create a source of iteration so that no temp lists are created. Previous code would look like that:

iterable = (expresionUsing(item) for item in collection)
for item in iterable: doWhatEverWith(item)

There are plenty of places in the code we generated last year (WiKo, TestFarm and build and utility scripts in CLAM where we can apply those two ideas. So i am pretty happy on having discovered that.

Command of the day: svk ls

Why not calling them command-of-the-year as the one and only command-of-the-day i posted until now was posted one year ago? Well, they are intended to be commands that saved me the day like the one i explain today.

On my previous entry, I explained that Pau and me were publishing on sourceforge a python script, WiKo, that we used to edit our latest articles and thesis. Because we used it in such context, latest version have been commited in a private repository we have for articles and the like.

I wanted to move the files to a public repository at sourceforge. Sadly, svnadmin dump just works when you are in the same host as the repository is and we have no regular ssh access to the subversion. We could bother our admin, Jordi Funollet, but he is always very busy and we are always requesting him. (Thanks, Jordi for your support!!)

Luckily i found a nice entry blog on how to use svk to remotely dump a svn repository.

You need to install svk, in debian based:

 sudo apt-get install svk 

Then we should create a mirror repository with svk just containing the revisions concerning the selected path

 svk ls svn+ssh://username@svn.myserver.com/path/to/repo/sub/path/to/wiko 

svk will ask you several questions:

• Choose a base URI to mirror from (press enter to use the full URI): To filter by path, i recommend to do such filtering on the command line by specifying the full path, so return.
• Depot path: [//mirror/wiko] Is very important here to change the depot name so you can skip that '//mirror' part, so answer //yourproject beginning with a double slash!!
• To end up, it asks which revisions do you want, normally the answer is 'a' (all)

Then dump the local svk repository:

 svnadmin dump ~/.svk/local > my-repository-dump 

And now you can import it into sourceforge as normal:

 svnadmin load [the-new-repository]  < my-reposotory-dump 

With sourceforge the process is more complex. You should upload the dumpfile to the shell server. For example, for wiko it was at: sftp://wiko.sourceforge.net/home/groups/w/wi/wiko/

Then you should specify the uploaded dump file as it was a cvs to svn migration. It last a while, after that you can checkout wiko code!

 svn co https://wiko.svn.sourceforge.net/svnroot/wiko/wiko 

I lost the history of the file before a move operation, but at least we recovered some of the history.

Taking a look at the man page you can see that svk is really helpfull for other purposes, but definitely, its ability of dumping a remote repository saved my day :-)