Tuesday, January 18, 2011

scoreLight synesthesia laser tracking

"scoreLight" is a prototype musical instrument capable of generating sound in real time from the lines of doodles as well as from the contours of three-dimensional objects nearby (hands, dancer's silhouette, architectural details, etc). There is no camera nor projector: a laser spot explores the shape as a pick-up head would search for sound over the surface of a vinyl record - with the significant difference that the groove is generated by the contours of the drawing itself.

Sound is produced and modulated according to the curvature of the lines being followed, their angle with respect to the vertical as well as their color and contrast. Sound is also spatialized; panning is controlled by the relative position of the tracking spots, their speed and acceleration. "scoreLight" implements gesture, shape and color-to-sound artificial synesthesia; abrupt changes in the direction of the lines produce trigger discrete sounds (percussion, glitches), thus creating a rhythmic base (the length of a closed path determines the overall tempo).

Elliot Woods, Kinect-Augmented Reality, as Projection Mapping Meets Depth Sensing

Kinect Hadouken from Elliot Woods on Vimeo.

Elliot Woods writes with an extraordinary proof of concept: it couples the depth-sensing capabilities of Microsoft’s Kinect with projection mapping to effectively “scan” a 3D scene. It’s almost Holodeck good, from the looks of the potential here.
Kinect hack + projection mapping = augmented reality +hadoukens): Using the kinect camera, we scan a 3D scene in realtime. Using a video projector, we project onto a 3D scene in realtime.By combining these, we can reproject onto geometry to directly overlay image data onto our surroundings which is contextual to their shape and position.
As seen in the video, we can create a virtual light source which casts light onto the surrounding surfaces as a real light source would.
At Kimchi and Chips we are developing new tools so we can create new experiences today. We share these new techniques and tools through open source code, installations and workshops.

More on the Kimchi and Chips blog

Updated: The duo is Kimchi and Chips as it’s a girl + guy, Seoul + Manchester team. I’m on it.

This video nicely shows some of the process of making this work: