CLAM binders, ipyclam and PyQt

I just finished a pair programming session with Xavier Serra. He is going to defend his Final Career Project on Friday and we are preparing some cool demos involving binding ipyclam and pyqt. It has been an exciting session because as we debunked one by one the show stoppers that we were finding, we realized that the potential of the tool mix is wider than we thought.

What we can do now is to build a network with ipyclam, building an interface by combining pyqt and ui files and binding them to that network, all that either in live by using the interactive shell or using a written script. This formula adds the flexibility of scripting to CLAM prototyping system, raising the ceiling of what you can do without raising too much the learning threshold.

For example, let's play a file in a loop with ipyclam:

import ipyclam
n=ipyclam.Network(ipyclam.Clam_NetworkProxy())
n.file = n.types.MonoAudioFileReader
n.out = n.types.AudioSink
n.file.Loop = 1
n.file.SourceFile = "jaume-voice.mp3"
n.file > n.out

n.backend="PortAudio"
n.play()

Then you can instantiate an Oscilloscope, bind it and play:

from PyQt4 import QtCore, QtGui
a=QtGui.QApplication([])

w=n.createWidget("Oscilloscope")
w.show()
w.setProperty("clamOutPort", "file.Samples Read")

n.stop()
n.bindUi(w)
n.play()

You can also load an UI file and bind it simultaneously..

w2 = n.loadUi("myui.ui")
w2.show()
n.stop()
n.bindUi(w2)
n.play()

Adding a tonal analysis and some related views

n.stop()
n.tonal = n.types.TonalAnalysis
n.file > n.tonal
w3 = n.createWidget("CLAM::VM::KeySpace")
w3.setProperty("clamOutPort", "tonal.Chord Correlation")
n.bindUi(w3)
w3.show()
n.play()

The enabler of the mix has been the new binder architecture in CLAM. Until 1.4, the Prototyper used what we call 'binders' to relate processing elements with user interface elements. Each binder concentrates in a given kind of binding: oscilloscopes, control sliders, transport buttons...

For the upcomming CLAM 1.5 the binder interface (CLAM::QtBinder) has been redefined and moved out from the Prototyper to the qtmonitors module. So now you can use them in any CLAM application. The bindings are now based on Qt dynamic properties which provide more flexibility than former QObject name mangling.

Currently there are several binders implemented:
* Action/Button -> launch a configurator dialog on some processing
* Action/Button -> launch a open dialog for a MonoFileReader
* Checkable -> Send a bool, or a bistable float control
* Slider -> maped float/int control
* Any Monitor -> outPort to monitor
* ControlSurface -> Send pair of controls
* Slider -> ProgressControl or a MonoFileReader or similar to control

Now you can extend those binder by means of plugins as we extended processings. You can use Qt dynamic properties in the interface elements to specify how the binding is done.

The abstract interface provides the static method QtBinder::bindAllBinders() which is the one called by python bindUi. This looks for every ui element in the QObject hierarchy, looks for every registered binder and if they match it is applied.

To implement your own handle you just have to rewrite the bool handles(QObject*) method that returns true if the object managed by the binder (type, name, presence of properties...) and the bool bind(QObject * uiElement, Network & network, QStringList & errors) which does the actual binding. To register the binder, you just have to instanciate one as static variable of a cxx file in your library.

All that is still being building up but its potential seems clear, so i hope that this new way of claming will be useful to you all.

ipyclam: interactive python console for CLAM networks manipulation

Xavier Serra (not the MTG head, but an homonym student at the UPF) is about to reach an interesting milestone on his Final Career Project related to CLAM. He is developing an interactive Python console for CLAM named ipyclam. It is a text based interface to explore and manipulate CLAM networks. Hopefully this will end being a dock widget within the Network Editor but it is also useful as interactive text console tool and as programming interface, which are the first useful outputs Xavi is about to get. ipyclam comes to cover the need of manipulating complex networks in cases when point and click is just tedious and also that of generating parametrized networks.

Let's make clear that it is not a full wrapper for the CLAM API like Hernan's pyclam, ipyclam just enables high level operations you usually do with Network Editor: instantiating, connecting, configuring and playing modules. Moreover we have intentionally decoupled ipyclam API to any existing CLAM API in order to define the most pythonic, convenient, and clean API we could think. For example the code to build an stereo cable would be:

net = Network()
net.processing1 = net.types.AudioSource
net.processing1.NOuputs = 2
net.processing2 = net.types.AudioSink
net.processing2.NInputs = 2

net.processing1.Output1 > net.processing2.Input1
net.processing1.Output2 > net.processing2.Input2

net.play()

An interesting feature, is that network.code() just returns an string with similar python code, so a network knows how to regenerate itself. This opens the door to python based format for CLAM networks. Bye bye XML!.

Convenience? Yes, please. Connections can be done in batch, broadcasting connections from all the connectors in one processing to all or to a single connection of another:

net.processing1 > net.processing2 # all to all
net.processing1.Output1 > net.processing2 # one to all
net.processing1 > net.processing2.Input1 # all to one

Convenience is good but all things should be doable. Often most convenient solution is not the best suited for all the cases. Accessing configuration parameters and connectors of any kind directly from proccessing is quite convenient but names can collide, to solve that some special attributes are provided to restrict the name scope.

network.processing1._config.NOutputs
network.processing1._inports.InPort1
network.processing1._outports.OutPort1
network.processing1._incontrols.InControl1
network.processing1._outcontrols.OutControl1

Connection names can have spaces, and attributes cannot, so we also provide an alternative subscript syntax.

net.processing1['Output 1']

One of the features I like the more is tab key exploration.

• network.[tab] and you get the list of processings
• network.processing._inports[tab] and you get the list of input ports
• network.processing._config[tab] and you get the list of configuration parameters
• network.processing.[tab] and you get connectors and cofiguration parameters
• network.types.[tab] and you get the list of available processing types

It works for interactive console with tab completion like ipython. We also provide ipyclam_console, an ipython based shell conveniently importing modules and adding a network to start playing with.

For more details on the API just check this document.

The implementation

ipyclam is defined in two layers. The front-end layer, purely written in python, makes the convenient API happen: Exploration, dynamic object attributes, many interfaces to similar operations... This upper layer, relies on the lower, less pythonic and more purpose oriented one, the network proxy layer, which provides access to the state and operations of the actual network with quite fewer entry points.

Xavi defined the front-end layer while evolving, in parallel, a dummy proxy layer simulating network status with basic python structures. That gave us the flexibility to explore and evolve both the frond-end and the proxy interface without bothering about the actual CLAM implementation that would have made such exploration quite rigid.

Two layers approach is also convenient because by defining the ops in the network proxy layer, we are able to reuse the front-end API with any other patching like environment. I am thinking on JACK or environments like ingen.

Status and road map

Xavi just finished most important bits of the API: Inspection, module creation, module type enumeration and module inter-connection. Currently we still rely on a dummy network proxy, but a CLAM based proxy will arrive during next week. Configuration API seems to be controversial and we are holding it until we have something implemented with CLAM.

Once we have a full CLAM based proxy the next big goal is to integrate the console in NetworkEditor. We need a console widget for Qt which is able to execute a python interpret. We found some candidates:

• qconsole which already has a python wrapper
• qtermwidget a more robust console (based on KDE Konsole) but without direct support for python, and
• ipython qt frontend quite experimental and using a weird threading model, but still interesting as ipyclam_console already uses ipython and it is awesome

So either candidate seems to require some work on our side.

Well, I hope you will find this new CLAM feature as amusing as I do. Xavier is really doing quite a good job, congrats and keep it that high.

Numpy arrays to video

I found some examples on how to generate video with python by piping frames to external programs like ffmpeg or mencoder. All those snippets encode each frame as a given image format (png, jpg...) by using PIL, matplotlib.. to get them. Besides the dependency, building figures or encoding images can be quite slow. I finally found a way of piping raw numpy buffers as frames to mencoder. You can use the following code:

import subprocess
import numpy as np

class VideoSink(object) :
        def __init__( self, size, filename="output", rate=10, byteorder="bgra" ) :
                self.size = size
                cmdstring  = ('mencoder',
                        '/dev/stdin',
                        '-demuxer', 'rawvideo',
                        '-rawvideo', 'w=%i:h=%i'%size[::-1]+":fps=%i:format=%s"%(rate,byteorder),
                        '-o', filename+'.avi',
                        '-ovc', 'lavc',
                        )
                self.p = subprocess.Popen(cmdstring, stdin=subprocess.PIPE, shell=False)

        def run(self, image) :
                assert image.shape == self.size
#               image.tofile(self.p.stdin) # should be faster but it is indeed slower
                self.p.stdin.write(image.tostring())
        def close(self) :
                self.p.stdin.close()

It is a tenth faster than the PIL based one. And there is no fair comparison to the matplotlib one. I got it working with mencoder but I could not figure out how to make it with ffmpeg.

You can find that code with an usage example there. Being based on other public domain snippets consider it also public domain.

I am using that class to save the output of my Freenect based project. More on that on next entries.