SeoNang

After skeletonization of the incoming video, the linesegments are passed on to a number of other modules one of which is the audio synthesis engine. This is a very direct and immediate interactive system that uses the raw line segment data to create aural feedback to the participant. This is achieved by convolving some granualted samples with a spectrum generated by the skeletons.

First, the synthesis algorithm bins line segment points according to their horizontal position into 3 overlapping groups (here red, green and blue) and then places a value in that bin based on vertical position. Points higher in the skeleton are placed further to the right in the spectrum image, corresponding to higher pitches. Points further to the right are lower pitches. This spectrum matrix is then convolved through multiplication in the frequency domain with granulated samples, creating aural feedback for the participant. When the participant raises his arms, higher pitches emerge. When he crouches, lower pitches are created. Panning of the audio is based on the 3 bins. Red is left, green is center, blue is right. Thus, when the participant moves from side to side in the image, the sound will follow.

<Point to Spectrum Filter>

The Point to Spectrum Filter divides points into 3 groups horizontally. Locations from 0 to 160 are red, 80 to 2240 are green and 160 to 320 are blue. Red points correspond to the Left Channel, green to Center and blue to Right. Points are also categorized by vertical position. The higher in spatial position a point is, the more to the right of the spectrum matrix the point will be placed, corresponding to a higher pitch in the FFT convolution.


Project Description	Movies & Photos	System Design	Credits
Installation Plan	System Schematic	Visual	Audio

Audio Synthesis
		After skeletonization of the incoming video, the linesegments are passed on to a number of other modules one of which is the audio synthesis engine. This is a very direct and immediate interactive system that uses the raw line segment data to create aural feedback to the participant. This is achieved by convolving some granualted samples with a spectrum generated by the skeletons. First, the synthesis algorithm bins line segment points according to their horizontal position into 3 overlapping groups (here red, green and blue) and then places a value in that bin based on vertical position. Points higher in the skeleton are placed further to the right in the spectrum image, corresponding to higher pitches. Points further to the right are lower pitches. This spectrum matrix is then convolved through multiplication in the frequency domain with granulated samples, creating aural feedback for the participant. When the participant raises his arms, higher pitches emerge. When he crouches, lower pitches are created. Panning of the audio is based on the 3 bins. Red is left, green is center, blue is right. Thus, when the participant moves from side to side in the image, the sound will follow. <Point to Spectrum Filter> The Point to Spectrum Filter divides points into 3 groups horizontally. Locations from 0 to 160 are red, 80 to 2240 are green and 160 to 320 are blue. Red points correspond to the Left Channel, green to Center and blue to Right. Points are also categorized by vertical position. The higher in spatial position a point is, the more to the right of the spectrum matrix the point will be placed, corresponding to a higher pitch in the FFT convolution.