CLAM binders, ipyclam and PyQt

I just finished a pair programming session with Xavier Serra. He is going to defend his Final Career Project on Friday and we are preparing some cool demos involving binding ipyclam and pyqt. It has been an exciting session because as we debunked one by one the show stoppers that we were finding, we realized that the potential of the tool mix is wider than we thought.

What we can do now is to build a network with ipyclam, building an interface by combining pyqt and ui files and binding them to that network, all that either in live by using the interactive shell or using a written script. This formula adds the flexibility of scripting to CLAM prototyping system, raising the ceiling of what you can do without raising too much the learning threshold.

For example, let's play a file in a loop with ipyclam:

import ipyclam
n=ipyclam.Network(ipyclam.Clam_NetworkProxy())
n.file = n.types.MonoAudioFileReader
n.out = n.types.AudioSink
n.file.Loop = 1
n.file.SourceFile = "jaume-voice.mp3"
n.file > n.out

n.backend="PortAudio"
n.play()

Then you can instantiate an Oscilloscope, bind it and play:

from PyQt4 import QtCore, QtGui
a=QtGui.QApplication([])

w=n.createWidget("Oscilloscope")
w.show()
w.setProperty("clamOutPort", "file.Samples Read")

n.stop()
n.bindUi(w)
n.play()

You can also load an UI file and bind it simultaneously..

w2 = n.loadUi("myui.ui")
w2.show()
n.stop()
n.bindUi(w2)
n.play()

Adding a tonal analysis and some related views

n.stop()
n.tonal = n.types.TonalAnalysis
n.file > n.tonal
w3 = n.createWidget("CLAM::VM::KeySpace")
w3.setProperty("clamOutPort", "tonal.Chord Correlation")
n.bindUi(w3)
w3.show()
n.play()

The enabler of the mix has been the new binder architecture in CLAM. Until 1.4, the Prototyper used what we call 'binders' to relate processing elements with user interface elements. Each binder concentrates in a given kind of binding: oscilloscopes, control sliders, transport buttons...

For the upcomming CLAM 1.5 the binder interface (CLAM::QtBinder) has been redefined and moved out from the Prototyper to the qtmonitors module. So now you can use them in any CLAM application. The bindings are now based on Qt dynamic properties which provide more flexibility than former QObject name mangling.

Currently there are several binders implemented:
* Action/Button -> launch a configurator dialog on some processing
* Action/Button -> launch a open dialog for a MonoFileReader
* Checkable -> Send a bool, or a bistable float control
* Slider -> maped float/int control
* Any Monitor -> outPort to monitor
* ControlSurface -> Send pair of controls
* Slider -> ProgressControl or a MonoFileReader or similar to control

Now you can extend those binder by means of plugins as we extended processings. You can use Qt dynamic properties in the interface elements to specify how the binding is done.

The abstract interface provides the static method QtBinder::bindAllBinders() which is the one called by python bindUi. This looks for every ui element in the QObject hierarchy, looks for every registered binder and if they match it is applied.

To implement your own handle you just have to rewrite the bool handles(QObject*) method that returns true if the object managed by the binder (type, name, presence of properties...) and the bool bind(QObject * uiElement, Network & network, QStringList & errors) which does the actual binding. To register the binder, you just have to instanciate one as static variable of a cxx file in your library.

All that is still being building up but its potential seems clear, so i hope that this new way of claming will be useful to you all.

ipyclam: interactive python console for CLAM networks manipulation

Xavier Serra (not the MTG head, but an homonym student at the UPF) is about to reach an interesting milestone on his Final Career Project related to CLAM. He is developing an interactive Python console for CLAM named ipyclam. It is a text based interface to explore and manipulate CLAM networks. Hopefully this will end being a dock widget within the Network Editor but it is also useful as interactive text console tool and as programming interface, which are the first useful outputs Xavi is about to get. ipyclam comes to cover the need of manipulating complex networks in cases when point and click is just tedious and also that of generating parametrized networks.

Let's make clear that it is not a full wrapper for the CLAM API like Hernan's pyclam, ipyclam just enables high level operations you usually do with Network Editor: instantiating, connecting, configuring and playing modules. Moreover we have intentionally decoupled ipyclam API to any existing CLAM API in order to define the most pythonic, convenient, and clean API we could think. For example the code to build an stereo cable would be:

net = Network()
net.processing1 = net.types.AudioSource
net.processing1.NOuputs = 2
net.processing2 = net.types.AudioSink
net.processing2.NInputs = 2

net.processing1.Output1 > net.processing2.Input1
net.processing1.Output2 > net.processing2.Input2

net.play()

An interesting feature, is that network.code() just returns an string with similar python code, so a network knows how to regenerate itself. This opens the door to python based format for CLAM networks. Bye bye XML!.

Convenience? Yes, please. Connections can be done in batch, broadcasting connections from all the connectors in one processing to all or to a single connection of another:

net.processing1 > net.processing2 # all to all
net.processing1.Output1 > net.processing2 # one to all
net.processing1 > net.processing2.Input1 # all to one

Convenience is good but all things should be doable. Often most convenient solution is not the best suited for all the cases. Accessing configuration parameters and connectors of any kind directly from proccessing is quite convenient but names can collide, to solve that some special attributes are provided to restrict the name scope.

network.processing1._config.NOutputs
network.processing1._inports.InPort1
network.processing1._outports.OutPort1
network.processing1._incontrols.InControl1
network.processing1._outcontrols.OutControl1

Connection names can have spaces, and attributes cannot, so we also provide an alternative subscript syntax.

net.processing1['Output 1']

One of the features I like the more is tab key exploration.

• network.[tab] and you get the list of processings
• network.processing._inports[tab] and you get the list of input ports
• network.processing._config[tab] and you get the list of configuration parameters
• network.processing.[tab] and you get connectors and cofiguration parameters
• network.types.[tab] and you get the list of available processing types

It works for interactive console with tab completion like ipython. We also provide ipyclam_console, an ipython based shell conveniently importing modules and adding a network to start playing with.

For more details on the API just check this document.

The implementation

ipyclam is defined in two layers. The front-end layer, purely written in python, makes the convenient API happen: Exploration, dynamic object attributes, many interfaces to similar operations... This upper layer, relies on the lower, less pythonic and more purpose oriented one, the network proxy layer, which provides access to the state and operations of the actual network with quite fewer entry points.

Xavi defined the front-end layer while evolving, in parallel, a dummy proxy layer simulating network status with basic python structures. That gave us the flexibility to explore and evolve both the frond-end and the proxy interface without bothering about the actual CLAM implementation that would have made such exploration quite rigid.

Two layers approach is also convenient because by defining the ops in the network proxy layer, we are able to reuse the front-end API with any other patching like environment. I am thinking on JACK or environments like ingen.

Status and road map

Xavi just finished most important bits of the API: Inspection, module creation, module type enumeration and module inter-connection. Currently we still rely on a dummy network proxy, but a CLAM based proxy will arrive during next week. Configuration API seems to be controversial and we are holding it until we have something implemented with CLAM.

Once we have a full CLAM based proxy the next big goal is to integrate the console in NetworkEditor. We need a console widget for Qt which is able to execute a python interpret. We found some candidates:

• qconsole which already has a python wrapper
• qtermwidget a more robust console (based on KDE Konsole) but without direct support for python, and
• ipython qt frontend quite experimental and using a weird threading model, but still interesting as ipyclam_console already uses ipython and it is awesome

So either candidate seems to require some work on our side.

Well, I hope you will find this new CLAM feature as amusing as I do. Xavier is really doing quite a good job, congrats and keep it that high.

Libraries becoming plugins, plugins becoming libraries

Let me post about some of the latest changes on the CLAM build system.

The CLAM project has been distributing two kinds of shared libraries: the main library modules (core, audioio and processing) and a set of plugins which are optional (spacialization, osc, guitareffects...). Plugins are very convenient since programs like the NetworkEditor can be extended without recompile and enable third party extensions. They are so convenient that we were seriously considering splitting processing and auidioio as plugins.

But the use of content of a clam plugin is limited to abstract Processing interface. No other symbols are visible from other components. That has a serious impact on the flexibility to define plugin boundaries. For example, if a plugin defines a new data token type, any processings managing that token type should be in that same plugin. To use symbols of one plugin from another implies installing headers, provide soname, linker name and so on. That is, building a module library.

So, if plugins want to be libraries and libraries want to be plugins, let them all be both. Each module will be compiled as library (libclam_mymodule.so.X.Y.Z), with soname (libclam_mymodule.so.X.Y), linker link (libclam_mymodule.so), headers (include/CLAM/mymodule/*), pc file... and it will also provide a plugin library (libclam_mymodule_plugin.so) which has no symbol by itself but is linked against the module library. When a program loads such a plugin, all the processings in the module library become available via the abstract interface. On the other side, if program or a module needs to use explicitly the symbols of another module, it just has to include the headers and link against the module library.

I just applied the changes to the plugins and seems to work nicely. Adapting the main modules will be harder because the old sconstruct file, but it is a matter of days and we can split them. I moved a lot of common SCons code to the clam.py scons tool. There is a cute environment method called ClamModule which generates a clam module including the module library, soname, linker name, plugin, pkg-config, and install targets. I also added some build system freebies for the modules and applications:

• All the intermediate generated code apart in a 'generated' directory
  
• Non verbose command line
  
• Colour command line
  
• Enhanced scanning methods with black lists

Well, do not rely too much on the current clam scons tool API. Changes are still flowing through the subversion. Of course, warn me if I broke something. ;-)

Boolean Controls in CLAM

After a long refactoring to get typed controls into clam without breaking anything, we already have them. Kudos for this achievement go also to Francisco (who started the whole thing as its GSoC project), Hernan, Nael and Pau.

Now controls are defined just like ports with a type. So for example, you can define an out control being the type OutControl<MyType>. As a side effect, control callbacks came in natural way. Instead of using a different class (formerly InControlCallback<MyType, HostProcessingType>), now you just have to pass a processing method to the control constructor. Templates do the magic too, but that is hidden from the processing programmer API which is cool. See, some example bellow.

So, once we enabled typed controls working, now is time to have more than just float controls. The main problem now is that in order to get some useful new control type you have to modify the network editor and the prototyper to make them useful.

Although the most demanding needs are for enum and integer controls, I started with simpler bool controls. My goal is you don't have to modify the NetworkEditor or the Prototyper to introduce a new control type. That is, a plugin could add new control senders, control displays, default connected processings (double clicking on a control) and binders for the prototyper (to locate ui elements to link to processing controls). So i wanted a new simple use case not being float to explore a feasible API.

Below you can see a network with a BoolControlSender (widget hxx/cxx, processing hxx,cxx) and a BinaryCounter (hxx,cxx) outputing into two BoolControlDisplay (widget hxx/cxx, processing hxx,cxx). Maybe is a little late for using CLAM for your Christmas lights, i guess ;-)



Currently, instead of bools we use floats considering a threshold for being true or false. A new ControlGate processing provides a transition by doing such translation:

So now is time to look to the code I had to add and see what can be enhanced to ease adding new control types:

• One of the things I didn't liked during the implementation is having to add an entry into a long list of 'if' statements in ProcessingBoxEmbededWidget.cxx. This is clamming for a refactoring into a Factory.
• Also the menu entry filling (to connect to new processing) and the default create-and-connect action become a list of if clauses with the type as parameter.
• Both, prototyper binders and embedded widgets had to duplicate the control sending code. That could be generalized into a binder object that both use.

So, it is clear that there is room for a lot of enhancement and it looks like those enhancements could also be applied to ports as well :-)

VST plugins with Qt user interface

I recently did an spike on what we need to make VST plugins first class CLAM citizens. CLAM allows to visually build JACK and PortAudio based applications with Qt interfaces as well as GUI-less VST and LADSPA plugins. The more flashy feature of VST is user interfaces that are mostly built using VSTGUI. We are using Qt as interface for JACK and Portaudio based apps because we are using the nice features of Qt toolkit to dynamically bind the UI elements and the underlaying processing. Moreover, Qt styling features enables shinning designer-made interfaces. Why not being able to reuse the same interface for VST and JACK? That has been a long standing TODO in CLAM so now is time to address it.

In summary, we fully solved croscompiling vst's from linux and we even started using qt interfaces as vst gui. In that last point, there still is a lot of work to do, but the basic question on whether you can use qt to edit a vst plugin is now out of any doubt.

To make the spike simpler, and in order not to collide with other CLAM developers, currently working on it, i just left apart all the CLAM wrapping part, just addressing vst crosscompiling and Qt with the sdk examples.

Cross compilation was pretty easy. This time I found lot more documentation on mingw and even scons. Just by adding the crossmingw scons tool we are already using for the apps and i managed to get Linux cross-compiled plugins running on Wine.

Adding a regular vstgui user interface is just a matter of compiling vstgui sources along with the example editor that comes in the sdk.

Once there, we should address Qt. VSTGUI is just a full graphical toolkit implementing the 'editor interface' plus a toolkit with some provided widgets and, i guess, a way of automating the binding of controls to processing. So what we need for qt is to implement the AEffEditor interface using the qt toolkit instead. The first problem is about the graphical loop. You have to create a QApplication and calling qApp::processEvents() on the editor's idle method so that qt widgets get responsive. The problem then is that, if you don't provide a QWidget as parent to your interface, it becomes a top level window ignoring the host provided window that still appears as an empty one.

VST host provides such window as a native Windows handle. How do you create a widget on an existing window handle? Months ago trolls redirected me to a commercial solution. Not such a 'solution' for us, a FLOSS project. So i was digging in windows qt source code for a hack when i found the answer just at the public and multiplatform QWidget api. QWidget::create works like a charm. The following simple class is a native window wrapper you can use as a regular QWidget.

class QVstWindow : public QWidget
{
        Q_OBJECT
public:
        QVstWindow(WId handle) {create(handle);}
        QVstWindow::~QVstWindow() {}
};

Still there are some issues: focus handling, reopening, drag&drop... But the basic mouse clicking and resizing works

Once i got that, loading a designer ui file was very easy.

As I said there are still many caveats to solve. A matter of playing with it and refining things. Here is a list of TODO's:

• Communicate controls from and to the interface
• Handle focus and other events properly
• Build a CLAM network wrapper which reensembles more the one for LADSPA
• Wiki documentation on how to build your own plugin
• One button plugin generator like the one we have for LADSPA ;-)

I feel that there is more people around other projects interested in using Qt for VST plugins so this is also a call for collaborative research on pending issues, at least the generic ones. Contact us on the CLAM development list or for a broader audience in the Linux Audio Developers list.

Refactoring script for CLAM networks

Now that the CLAM NetworkEditor has become such a convenient tool to define audio processing systems, we started to use it massively instead of C++ code to integrate processings. Even thought, we started to find that whenever we change a configuration structure, a port/control name, or even a processing class name, the networks xml had to be edited by hand because they didn't load. That problem led us to avoid such kind of changes, which is not a sane option. 'Embrace change', agile gurus said, and so we did.

We have developed a python tool to support network refactoring. It can be used, both as a python module, or as a command line tool, to batch modify CLAM network files using high level operations such as:

• Renaming a processing class name
  
• Renaming a processing name
  
• Renaming processing ports or controls
  
• Renaming a processing configuration parameter
  
• Removing/adding/reorder configuration parameters
  
• Setting configuration parameters values
  

The script just does XPath navigation and DOM manipulations in order to know which pieces need to be changed. Each high level command is just a couple of lines in python.

We are starting to use it to:

• Adapt to changes in the C++ code
  
• Change network configuration parameters in batch
  
• Provide migration scripts for user's networks
  

About the last point, we plan the next release to provide network migration scripts containing a command set such as:

  
ensureVersion 1.3
renameClass OutControlSender NewClassName
renameConnector AudioMixer inport "Input 0" "New Port Name 0"
renameConfig AudioMixer NumberOfInPorts NInputs
setConfigByType AudioMixer NInputs 14
upgrade 1.3.2

Still some short term TODO's:

• Include the clam version in the network xml so that the ensureVersion and upgrade commands work.
  
• Integrating also Qt Designer UI files in the refactorings
  
• Add some other commands as they are needed
  

Happy CLAM networks refactoring!

Managing Audio Back2Back Tests

So long since last post, and a lot of things to explain (GSoC results, GSoC Mentor Submit, QtDevDays, CLAM network refactoring script, typed controls...). But, let's start explaining some work we did on a Back-to-back system we recently deployed for CLAM and our 3D acoustic project in Barcelona Media.

Back-to-back testing background

You, extreme programmer, might want to have unit tests (white box testing) for every single line of code you write. But, sometimes, this is a hard thing to achieve. For example, canonical test cases for audio processing algorithms that exercise a single piece of code are very hard to find. You might also want to take control of a piece of untested code in order to refactor it without introducing new bugs. In all those cases back-to-back tests are your most powerful tool.

Back-to-back tests (B2B) are black box tests that compare the output of a reference version of an algorithm with the output of an evolved version, given the same set of inputs. When a back-to-back test fails, it means that something changed but normally it doesn't give you any more information than that. If the change was expected to alter the output, you must revalidate the new output again and make it the new reference. But if the alteration was not expected, you should either roll-back the change or fix it.

In back-to-back tests there is no truth to be asserted. You just rely on the fact that the last version was OK. If b2b tests get red because an expected change of behaviour but you don't validate the new results, you will loose any control on posterior changes. So, is very important to keep them green or validating any new correct result. Because of that, B2B tests are very helpful to be used in combination of a continuous integration system such as TestFarm, that can point you to the guilty commit even if further commits have been done.

CLAM's OfflinePlayer is very convenient to do back2back testing of CLAM networks. It runs them off-line specifying some input and outputs wave files. Automate the check by subtracting the output with a reference file and checking the level against a threshold, and you have a back-to-back test.

But still maintaining the outputs up-to-date is hard. So, we have developed a python module named audiob2b.py that makes defining and maintaining b2b test on audio very easy.

Defining a b2b test suite

A test suite is defined by defining back-to-back data path, and a list of test cases, each one defining a name, a command line and a set of outputs to be checked:

#!/usr/bin/env python
# back2back.py
 from audiob2b import runBack2BackProgram
 data_path="b2b/mysuite"
 back2BackTests = [
  ("testCase1",
   "OfflinePlayer mynetwork.clamnetwork b2b/mysuite/inputs/input1.wav -o output1.wav output2.wav"
   , [
    "output1.wav",
    "output2.wav",
   ]),
  # any other testcases there
 ]
 runBack2BackProgram(data_path, sys.argv, back2BackTests)

Notice that this example uses OfflinePlayer but, as you write the full command line, you are not just limited to that. Indeed for 3D acoustics algorithms we are testing other programs that also generate wave files.

Back-to-back work flow

When you run the test suite the first time (./back2back.py without parameters) there is no reference files (expectation) and you will get a red. Current outputs will be copied into the data path like that:
b2b/mysuite/testCase1_output1_result.wav
b2b/mysuite/testCase1_output2_result.wav
...

After validating that the outputs are OK, you can accept a test case by issuing:
$ ./back2back.py --validate testCase1
The files will be moved as:
b2b/mysuite/testCase1_output1_expected.wav
b2b/mysuite/testCase1_output2_expected.wav
...

And the next time you run the tests, they will be green. At this point you can add and commit the 'expected' files on the data repository.

Whenever the output is altered in a sensible way and you get a red, you will have again the '_result' files and also some '_diff' files so that you can easily check the difference. All those files will be cleaned as soon you validate them or you get back the old results.
So the main benefit of that is that the expectation files management is almost automated so it is easier to maintain them in green.

Supporting architecture differences

Often the same algorithm provides slightly different values depending on the architecture you are running on, mostly because different precision (ie. 32 vs. 64 bits) or different implementations of the floating point functions.

Having back-to-back tests changing all the time depending on which platform you run them is not something desirable. The audiob2b module generate platform dependant expectations by validating them with the --arch flag. Platform dependant expectations are used instead the regular ones just if the ones for the current platform are found.

Future

The near future of the tool is just being used. We should extend the set of controlled networks and processing modules in CLAM. So I would like to invite other CLAM contributors to add more back2back's. Place your suite data in 'clam-test-data/b2b/'. We should decide where the suite definitions themselves should be placed. Maybe somewhere in CLAM/test but it won't be fair because dependencies on NetworkEditor and maybe in plugins.

Also a feature that would extend the kind of code we control with back-to-back, would be supporting file types other than wave files such as plain text files, or XML files (some kind smarter than just plain text). Any ideas? Comments?

Exterminating signals and slots

I recently posted at the clam devel wiki some advices, conventions, traps and tips on programing with Qt within CLAM addressed to the GSoC students. The most painful trap is the one of gratuitous signals and slots usage. Signals and slots is a very powerful way of designing independent components that comunicate each other with low coupling. Is that powerful that it is very tempting for novices to use them everywhere and this can turn very harmful for you.

If a connected slot doesn't exist, you just get an error console on run-time, checks on slots are that soft, and this is bad. Also the signal slot resolution is more expensive than just a method call. But the main reason to avoid them is that they make the code very hard to follow. When you see a signal 'emit' you have to find where such signal is connected in order to find which are the objects and slots that will be called. Multiply this process by nearly 200 signal emisions that were done at the vmqt library Annotator is using and you'll understand why such components although very smartly designed in structure, they are very hard to maintain and use.

You should emit signals just when you don't know which is the receiver of an event. If you can guess the receiver, there is no use for signals. On the other side, if you cannot access the emitter code, or you don't want to couple it, then there is room for a 'connect'.

A bad smell for noticing that you are over using sigslots is having in a class you are writting a connection like this:

connect(this, signal, knownWidget, slot)

and later, maybe in a different method:

emit signal()

When that known, and later forgotten, object is the only one you are connecting to the slot you might want to just keep a reference to it and just call the slot instead of emitting the signal.

knownWidget->slot()

which is faster, compile time checked and much more traceable.

Given that bad smell locator at hand i decided to do a fast review of vmqt module. vmqt is the successor of the many Visualization Modules we had in CLAM, just annother definitive VM rewrite. I tried to use it several times, but it is very hard to have a global view of what it does because all the program flow is driven by signal connections. No joke: 180 signal emissions and a similar number of 'connect's in 30 classes.

After a first review, 110 signals emisions have been avoided just by having a pointer to the Plot2D at the Renderers (the objects that represent drawing layers of the plot such as the lines, the playhead, the grid...). Having a pointer to it, Renderers can do a direct call to the Plot2D slots (now regular functions) instead of emitting a signal whose only receiver is the plot.

70 'emit's are still wandering around. Some of them communicate the plot with the wplot (a widget containing the plot and other elements such as the sliders, the rulers...) so they are used to syncronize. Those are likely to be removed in a similar way than for renderers.

The rest are specific renderer signals that are connected and propagated by the specific wplot. Those are harder to remove and indeed they are very convenient to keep.

As the extermination goes on, one can better see how to really take profit of the nice vmqt module structure and which aspects can be enhanced.

Preparing for GSoC 2008

GSoC 2008 is already here! We are preparing our submision for CLAM as organization and I hope we are as lucky as last year. For GSoC 2007 we got 6 fervent students who pushed CLAM a big step forward. We still don't know whether we will be selected as organization or not. We haven't even filled the submision data. But it is time to trigger some resorts. So, what to do now?

If you are an experienced CLAM developer, please consider becoming a mentor. The more mentors the more students we can cope with.

If you are a user, is the time to push your favourite feature into the GSoC project proposals.

If you are an student wanting to be part of the program, I advice you to get involved with the project from now as we will consider early involvement a big plus for eligibility.

If you are Xavi, Pau or myself, then you should fill CLAM submision instead of blogging ;-)

I love summer.

Qt 3 and Qt4 relicensed as GPL v3

I read in Thiago Maceira's blog that Trolls have relicensed Qt3 and Qt4 as GPL v3.

I still had no time to analyze how this will affect clam. Relicensing CLAM has proven to be a hard political and burocratic problem, but a transition from GPL v2 to GPL v3 is easier since we have the 'or later' notice, so we have the door open to that controversial upgrade. Definitely, trolls' move to v3 is something that will boost the adoption scene.

A nice lateral consequence of this affects directly to one CLAM application, SMSTools, which still uses Qt3, and not Qt4 (althought Zack Welch started a temptative Qt4 port). Until now, Qt3 didn't enjoyed the same nice dual licensing Qt4 has. Former Qt3 licensing was a modified GPL that had problems on using Qt3 for non-unix platforms such as windows. So with Qt3 relicensing we can now freely distribute SMSTools precompiled binaries for Windows.

Update: I read in the official announcement that they didn't droped GPLv2, they just added a third license to the existing dual licensing so you can use one of three licensing schemes: non-free, GPLv2 and GPLv3. Definitely they want to make developers live easier, not just by building excellent API's.

Trolltech Developers Day

On my last entry I talked about the trip to San Francisco. Besides the GSoC mentor submit, we also joined the Trolltech developers days. The slides and examples of the Trolltech event are available online. It is worth to take a look at them although slides are not as nice as having some trolls explaining them in live.

We really learned a lot of such event: Andreas Aardal Hansen, a very smart hacker, gave us a nice lesson on how we could exploit the Graphics View framework to turn CLAM into a cooler environment. The framework has been massively adopted in just one year since it was published, and the reason is that it is just impressive how easy you can do complex and cute things on a snap. The reason why we didn't used it is because we did the full rewrite of the NetworkEditor just before Qt 4.2 which included such framework. Another rewrite at that time would have been too hard.

We also could have a little talk with Thierry Bastian who is responsible of turning KDE's Phonon multimedia interface into a regular cross platform module in Qt. Unfortunately we were still travelling the first day and we missed his talk on Phonon.

Another nice session I assisted was the one presented by Girish Ramakrishnan on Qt style sheet support. Just by taking a look at the latest demos you can realize that Qt application can be very cool and even use such style sheets to let designers enhancing the look and feel of your applications without even programming.

While I was on the style sheet presentation Pau was on the Model View talk. I thought that it was about the old Smalltalk MVC model we failed to generalize in CLAM so many times and in so many flawors. So I thought it was a mistake on the troll's side to generalized it so much, but I just took a look at the presentation, and maybe is worth a try. It seems that they did a smart twist to the pattern to make it worth to work with.

I also nagged Jason Barron, at the support team, with some bugs we found on latest Qt version and some question on how to implement some things for CLAM. I still have to get in contact with him again to get some solution for some problems.

As I said we learned a lot, and we are eager to apply such knowledge to CLAM. A month later we have just applied such knowledge to fix some QtOpenGL bugs but we are planning at some time to port the NetworkCanvas to QGraphicsView.

GSoC 2007 mentor submit

A couple of weeks ago, Pau and me were at San Francisco for the Google Summer of Code Mentor Submit. At the submit we joined some interesting sessions on how to improve the next GSoC, but also we met a lot of other hackers of free software projects all around the world: Mixxx, Inkscape, KDE, Amarok, Pidgin, Scumvm, Haiku, Crystal Space, Ogre, Moin Moin, Drupal, VideoLan, XMMS2, audacious... Well, as you can see in this picture we were pretty much people:



I knew about most of all other projects, but, sadly, nearly no one knew about CLAM. :-( The good news is that most of them get pretty interested as they saw CLAM capabilities on audio processing and application prototyping. We need independently packaged end user tools!!

I also was glad to confirm the collaboration between projects that one could figure out being competitors:

• People working on desktops (KDE, Gnome...) are in fact in a very close collaboration.
• Wiki implementations (Moin Moin, Wikimedia...) meet there
• Also several 3D engines (Crystal Space, Ogre...) met there
• Comunity portals such as Drupal, Joomla, Plone...

They all share common challenges and solutions and they used the submit to join efforts.

Related to common issues and efforts, we took a very productive meeting on multiplatform development where some multiplatform projects, including us, shared experiences. We explained how we faced such problem at CLAM: the use of portable 3rd party libraries, crosscompiling from Linux to other platforms, and the use of testfarm to rapidly detect multiplatform issues. Some projects, such VideoLan had automated third party compilation scripts a step we have not automated at all. We realized that a lot of people was also solving such problems and we compromised to share experiences and results. For example, we talked about having a single repository of precompiled binaries of third party libraries. We also realized about our not-so-open-source mind we most of us realized that we were not reporting back patches we did to external libraries. We compromised to send them to upstream or making them available in a common place for the other project using them.

Simplifying Spectral processing in CLAM

Current CLAM Spectrum implementation has four different internal representations: Complex array, polar array, separated magnitude and phase buffers, and separated magnitude and phase break point functions (point controled envelopes). Conversion and synchronization among such representations were handled by the Spectrum class itself and this was intended to be transparent. But processing algorithms had to know which representations were already present to trade off between doing conversion and applying different versions of a given algorithm depending on the representation.

For instance, spectrum product in polar representation is cheaper by itself than complex product, but if you have to convert incoming or outcomming spectrums it migth be not so cheap. What about having two different representations for each operand? Which one should be converted? Output format could be a discriminant but do the module has information to know which is the convenient output representation? Such decissions make the implemention of processings using spectrums really complex.

Besides that, algorithms also had to consider different cases for different scales (linear or dB). So the complexity of processing objects having to deal with spectra is very high. A lot of code dealt with querying the presence of a given representation or synchronizing them.

So, we decided to simplify all that by having several kind of spectrum for each representation and doing any representation change explicit by a converter processing. We are currently considering just two spectrum classes: MagPhaseSpectrum and ComplexSpectrum. We added also converters to the current Spectrum to enable progressive migration. The implementation of FFT and IFFT was highly simplified as we don't need to deal with conversions and prototypes configuration, just to input or output complex spectra.

This is a setup of a system which perform complex and polar products of two spectra:



The spectral product has been simplified. Now we have a product for each kind of spectrum in contrast with the previous situation where we had a single module containing implementations for every combinations of inputs.

Code is really simpler to understand and write this way. After having all that we could do a profiling on comparing which is best, conversions and cheap product or no conversions and expensive product. The option without conversions won but the situation can change on the future depending on the optimizations we could do on the conversion. Having the processing modules factored that way we could change them and doing the benchmark that easy.

In the process of takin such decission we noticed that some elementary CLAM elements such as the AudioSource, AudioSink and specially AudioMixer were taking too much CPU.



After fixing that we got a clear profile that highlights which is the proper strategy for spectrum product for our concrete case. Callgrind is a really nice tool:



So I hope we can move soon to that new way of doing spectral processing in CLAM but still a lot of code must be updated.

CLAM Plugins

One of the most interesting features of the upcomming CLAM release will be a plugin system that will allow users to add their own processings as a plugin to any CLAM based application.
Just by placing a library on a given folder.



We provide some cute tutorials on writting simple processings from scratch, and a SConstruct file that will compile and install everything in a given folder as a plugin.

It has been more easy than we thought as we were frightened by Visual 'mecatxis' and we just dropped it. A single class did the work. There is still some work to do on documenting the registration of the configuration dialogs for the processings and making the Processing Tree of the network editor to populate automatically from the factory content.

The former can be done very quick by adding the Qt4 configurator as a CLAM library, but we want to provide a more general solution not requiring users linking against Qt4. The later (processing tree population) is something Andreas Calvo is working with. And he is doing nice progresses.

Deprecating Windows as development platform

In this entry i will explain how we got ride of Windows as development platform for the CLAM project still providing Windows binaries by using the tandem mingw32 (for Linux) and Wine. We dealt with crosscompilation of most of CLAM dependencies such as Portaudio, Asio, libsndfile, liboggvorbis, pthreads, fftw3, libmad, id3lib, XercesC, libxml++, NSIS and Py2Exe.




After a very dissapointing and fustrating effort from Pau and me of trying to have a reproducible build environment for CLAM based on Visual Express 2005, we decided that the whole thing was foolish and decided give mingw a try.

I am proud of not having a Windows box available at home ;-) but that doesn't mean i cannot contribute to that part of the development so the last days i did an spike on having mingw crosscompiling CLAM from Linux! We succeeded to compile it all, the dependencies, CLAM and the applications. Because some dll conflicts when installing that could not go into the 1.1.0 version of CLAM. After a gutsy update we could compile it all,
so expect rewamped windows binaries for the next release... and maybe development binaries. You can already give a try to the latest svn snapshots.


The key point has been using Wine to install with their own windows installers some libraries and tools such as Qt4, GTK (for pkg-config, libxml++ and dependencies), and NSIS to build windows installers from Linux!

Wine and mingw are a great tandem to compile and test the binaries. Also SConstruct has been a nice tool to hack a quick build environment for third party libraries when the one provided (normally based on autotools) was too messy to get on with.

We kept a reproducible log of the mingw crosscompilation at the clam wiki. This includes how to get all the dependencies working, which may be usefull for you own project. CLAM dependencies we addressed were: Portaudio, Asio, libsndfile, liboggvorbis, pthreads, fftw3, libmad, id3lib, XercesC, libxml++, NSIS and Py2Exe.

Enhancing QSynth/Rosegarden/SonicVisualizer knobs for CLAM

I am still expanding the widgets toolbox for CLAM.



PK Widgets, the ones in my previos blog entry, are really nice but they don't integrate well with other widgets because the background and the fixed size. So you have to do a mostly PK interface or not. Most standard Qt widgets look nice with a nice theme such plastique, but for some reason QDial is the ugly duck of the collection... well, according Chris Cannam, also QLCD. I agree with him. QDial looks like old CDE style in every style you choose:



Besides the look, QDial behaviour is far from optimal. Clicking on any point jumps the pointer which is a bad thing if you want to progressively control a parameter. Also when you move the knob beyond one of the extremes you may switch sharply to the other extreme. Using such a mischievous control in a live performance could be a disaster. So let's look for asthetic and behavioral alternatives.

Going back to the look, I just liked a lot the knobs in QSynth a program by Rui Nuno Capella.



The knobs indeed were taken from Rosegarden and Pedro Lopez Cabanillas enhanced the look by adding some nice but expensive gradients. The ones that attracted me.

I took those widgets with such a long trail and I extended them.
I ported them to Qt4 using QGradients which are faster, and i did some visual improvements: shadows, bumped pointer, configurable colors, angular and linear modes of mouse handling... I added them to the CLAM Widgets plugin and added some configurability so you can change the colors from the designer. And that's how they look now:

PK Widgets integration in CLAM

And last but not least, the other important change i added this Easter to CLAM has been integrating Patrick Kidd's PkWidgets. This was a really dormant to-do. A working implemtation was on the 'draft' cvs from 2004.

On 2004, Patrick amazed the LAD community with his nice looking PKSampler. I must confess that I never got PKSampler sounding because a bunch of missing dependencies on Debian/Ubuntu. But, a part from the audio work, the interface was really gorgeous. Something that was not so common on the Linux Audio scene.

The nice look was mostly due to a clever combination of free technologies: Qt, Python and Povray. Povray rendered 3D widgets with a realistic look, Qt the interface foundation and the resource system and Python to glue it all.
A clever trick allows to load povray images as frames of an interactive animation to render a widget.

At that time I was starting what is now the Prototyper and I was already messing with designer plugins, i could not stand and I ported some of the Patrick widgets to C++ in order to include them in a Designer plugin.

I exchanged some mails with him but the code was left on CLAM draft repository... until Saturday. I updated it to Qt4, added some configuration features, merged some of the new pixmaps and i integrated them on the CLAMWidgets plugin. The result, now you can prototype with CLAM interfaces such that one:

Educational Vowel Synth and ControlSurface Widget

Some months ago I had the idea of doing a Vowel Synthesizer program which could help children to understand why the vowel triangle has some sense and it also could be used to train foreign language vowels.



The core of the idea is that axes correspond mostly to the position of the first two formants F1 and F2. So by a simple mapping, given a point on the triangle you could synthesize any vowel even those in the middle. Also by analyzing incoming voice from a microphone, the system could place a point in the triangle that identifies how far one is from the intended vowel.



I did a first prototype which synthesized something far from human but at least you can identify the vowels when you place the proper frequencies. As i had a proof of concept i stopped there.



This easter i improved the prototype by adding a new control widget: The ControlSurface, which control two parameters by moving a single point. It is really more handy than having two sliders and it is perfect for parameter exploration, not just for the Vowel Synthesizer.

It will also force a change on the way the CLAM Prototyper address the binding of controls as it controls two parameters and our current system binds a widget to a single inlet.

Realtime MFCC and LPC analysis

As I said on the previous blog entry, too much things done in CLAM this Easter. We fully dropped Fltk visualization module from the main libraries. Zach Welch (one of the active new commers) ported Voice2Midi to Qt4 and we moved a subset of the Qt3 visualization module to the SMSTools and dropped the rest. (Zach prepared a set of patches to port it to Qt4 but let's wait until the rewrite). All those drops mean that CLAM size has been reduced to a half of its former size: less mainteinance and faster entry for the new developers.

A few examples stopped being functional because they used Fltk Plots. One of them the LPC_example. It compiled by disabling visualization rendered the example useless. I decided to convert it into a Network Editor example.

After some hacking i had this piece of network working.



After some designer and prototyper hacking i got this interface:



Trying to understand the meaning of the LPC I also added a new output the spectral envelope:



So it captures the spectral envelope which is gonna be very useful for the vowel synth.

Then a user asked for realtime MFCC. As we already had the processings but using portless interface, and LPC visualization could be reused for MFCC, i also took this task. The result:



Mixing it all in a single analysis interface



Vowel formants are even more clear in MelSpectrum that in MelCepstrum or the Spectrum it self. Another idea to be used for the Educational Vowel Synth.

Realtime Voice Gender Change

This Easter has been very productive for the CLAM framework. Holidays and a bunch of new developers that have aproximated the team during these weeks have accelerated a lot the development which had already been accelerated during last months. As consequence, a lot of dormant to-do's has been addressed and most of them deserve a blog entry (pixmap widgets, realtime LPC/MFCC, Qt3/Fltk drop, Qt4 ports, new visualization widgets, new control widgets...). Just take a look at the CLAM-devel mailing list archives or the development screenshots and you'll see a im not exagerating. So let's start from the beginning.

One of the nicer things I worked on and which is going to be the star on CLAM demos for sure, is the voice Gender Change effect on the NetworkEditor. Since Xavier implemented the offline version, it has been arround for a while on the SMSTools turning Elvis voice into Elvira. It was also on the NetworkEditor but crashing or not even sounding.

So, I am a demo whore, let's make people smile. Let's make it realtime. Pau already did some work as he ported almost all the SMS transformation. I just had to fix some of the internal processings, the Pitch shift, and the SpectralShapeShift. Most of the bugs were about detecting non-harmonic parts and figure out which is the best strategy in those cases. In some case this was detected but the action relied on input to output copies implicit on the offline version but not on the realtime one.

What i was not able to solve is the CombFilter. It added a really extrident noise to the residual part. So I disabled it and it does not sound as perfet but... tada! i started listening my sister voice from the speakers instead of mine when i talked the mic, which is specially frightening, most of you know why.

That's the look the network had after having the system completed.



And a first attemp Prototyper interface for an stand alone application:



As soon as it is available as nightly snapshots try it. It is worth to listen. My first experiments i used it to learn how to do a mixt choir in Ardour with just my own voice.