Kinect Yoga

Max / Kinect

For my final “MEME” (Multimedia & Electronic Music Experiments) project at Brown, I built an interactive installation written in Max using the Xbox Kinect sensor. Yoga poses are shown on a projector for the user to match. The user is given musical feedback based on the position of their body: each pose corresponds to an individual soundscape, and different tracks within these soundscapes are brought in as each body part enters the correct position in space. When the user holds the correct position for 15 seconds (or when the user points at the Kinect), the next pose appears on the projector.

Soundscapes

For each of 7 yoga poses, I composed a 4-track soundscape. Each track was mapped to a body part — the user’s head or a particular hand, foot, or knee.

The poses are: Tree, Eagle, Triangle, Half-Moon, Warrior I, Warrior II, Warrior III.

Triangle Pose

Warrior I Pose

How it works

Max (aka MAX/MSP/Jitter) is a versatile, visual programming language used for processing audio/video and was a major focus of my coursework at Brown.

I used the Kinect-Via-Synapse interface (demo video / documentation) to send body position data to Max in real time.

Synapse waits for the user to initialize tracking by standing in this pose, which is shown to the user on a projector:

After initializing, the first yoga pose for the user to match is shown on the projector.

This screenshot shows the portion of the Max patch (a program in Max is called a “patch”) receiving body positions from the Kinect via Synapse:

At the very top, the patch tells Synapse which joints to track (e.g. ‘rightfoot_trackjointpos 1’).

In the bottom half, ‘udpreceive 12345’ receives the real-time data from the Kinect (Synapse sends that data on port 12345), then initializes variables (e.g. ’send leftfootX)’ that can be used as inputs in other parts of the patch. The torso is tracked within space, and then all other body parts are tracked in relation to the torso.

Upon initialization, “headratio” is set — this variable is used to adjust the target areas for each body part based on the user’s height. See rightmost section of the screenshot below:

Below is the section of the patch for the right foot:

Above, pose #3 (triangle pose) is selected, and the target area on the right reflects where the right foot should be in relation to the torso for this pose. The crosshairs are the XY coordinates of the user’s right foot. As the right foot enters the target area, the right foot track (vibes, in this case) fades in. The volume increases as the right foot gets closer to the center of the target area. As you can see in this screenshot, the crosshairs are about halfway towards the center of the target area, and the volume is turned up about halfway.

To start, sound only comes from the front speakers. As soon as all tracked body parts for the selected pose are within their target areas, the soundscape gradually fades into the speakers behind the user as well, and the volume slowly crescendos over 15 seconds. After the pose is held for the full 15 seconds, the next pose is automatically selected and shown on the projector. The user may also go to the next pose manually by pointing forward for a few seconds (this is based on the right_hand z-axis).

The result is an immersive, challenging, yet relaxing experience.