Workshop Report

Report on the Workshop
„Intermedia composing with EyesWeb“
Coordination, research: Laura Mello
Teacher: György Pongracz
In cooperation with Klangnetz for the Festival HörenSehen2.0
Partnerships: Staalplaat, Klangzeitort, TU Berlin Audiokommunikation

16. – 25.07.09
Staalplaat
Flughafenstr.38
Berlin


Workshop Flow
- 4 days Introduction to the multimodal Environment EyesWeb .
- 8 days individual/group workshop: participants or groups of participants worked on their projects with György´s assistance.
- 1 day presentation of results.


Motion capture, motion tracking and movement analysis. What is it?
An usb camera is connected to a computer with installed software EyesWeb and another Audiosoftware like e.g. Ableton Live, Reason or Reaktor.
In front of the camera there is a „main character“, someone or something that is moving whithin the picture in front of a background, i.e., there is some contrast between main character and background.
These movements are captured and analyzed by EyesWeb. The analysis is strongly based on Rudolf Laban´s Theory of Effort. Analysis results can be shared by Applications via XML, HTTP, RS232, TCP, UDP, OSC, DMX, MIDI

There are three main libraries for analysing movement:
1. Motion Analysis
2. Space Analysis
3. Trajectory Analysis

For example, in this case, György´s body middle point movements are captured and mapped to a filter in Reaktor through MIDI.






The engineer György Pongracz uses to work with dancers. We wanted to try out other possibilities of EyesWeb. Using the same algorithms made for human beings we tried out with other moving beings, like for example this friendly camel catched up at a pet zoo in Berlin.




The camel didn´t take our invitation to come to Staalplaat, but he allowed us to use his image. So we filmed him at the pet zoo Hasenheide in Berlin and exported it as an AVI format, with resolution 352 x 288 pixels to be read by EyesWeb. It means the camera is not being used real time in this case, what actually doesn´t make much of a difference for EyesWeb, because the analysis feature is the one that uses the greatest amount of processing capacity. The cameras that work the best with EyesWeb are WebCameras, with small resolution and bad quality.

The sound is being generated by the movements of the camel, connected through MIDI to Ableton Live. His movements on the horizontal (x axis) determine the pitch of the synthesizer and the movements on the vertical axis, the speed of a sound file (with human voice). I´ve recorded some moments of the empty scenery, without the camel, so that this moving element could be identified by EyesWeb as the object to be tracked. When the cute small black goat enters the scene EyesWeb understands it as if both of the animals were only one, so that you get kind of random MIDI values.

Another interesting project was using EyesWeb tracking hand movements, together with a Wii-Mote controller as trigger for pre-produced loops. In fact, this environment is almost like an instrument: one hand is tracked by EyesWeb and the other hand controls the Wii-Mote.




EyesWeb open source modular environment
As the name already says, EyesWeb is made on module blocks that you can interconnect to build your own patches. Each module can also be reprogrammed, as it is open source.
Reading the EyesWeb Reference Manual we find the expressions „multimodal“ and „cross-modal“. It refers to the way EyesWeb is dealing with different datatypes. One feature is the ability to transform automatically datatypes of different, but compatible domains. They give as an example an FFT block that can work both on matrices and on audio streaming.
Another feature is the timestamping of datatypes. Each datatype is associated with a set of timestamp which let the kernel to synchronize them according to different needs.

But what is actually a “Block”?
From the user´s point of view, the Blocks are the most important objects in EyesWeb. Blocks represent an operation, from very simple to very complex, that can be performed on data. It means, blocks do something to an input information, what could be a file or a live signal, the blocks manipulate it in some way.

Source: http://www.infomus.org/EywMain.html