Pong is one of the earliest arcade video games, and its success can be attributed to its simplicity, as it requires little explanation, it is easy to learn and fun to play. In contrast, the Piano favors sequential operations, to control musical parameters, through the use of keys, buttons and knobs.
While attending a Penny Stamps talk at the Michigan Theater, one of the slides reminded me of a very complex version of Pong, the arcade video games. Also, an organist performs before every talk and I wanted to explore ways to connect the two things.
In this project, I attempt to design a gestural interface, which extends the simplicity of Pong, in order to create music in more fluid and malleable ways, which is both engaging and enjoyable. Pong is a combination of kinesthetic, audio and visual interaction, and the audio feedback that is provides, varies dynamically with the corresponding event. The interface constitutes an Inertial Measurement Unit (IMU), which sends motion data to the Graphical User Interface (GUI), to control the Pong Paddle. Musical notes are generated when the ball hits the paddle.
The setup for Piano-Pong consists of an inertial motion capture unit coupled with a graphical interface, and an audio output. To navigate the complexities of calibration and low level signal processing, required for a stand-alone IMU, this project uses a standard iPod Touch with a built in inertial measurement unit.
A key aspects of designing good gestural interfaces, is choosing the right frame of reference.A mobile interface would mean a hand centered frame of reference as opposed to a body centered or a world centered one. Movement in the hand-centered frame of reference, calls for fine controlled gestures. Also, when using gestures coupled with the phone interface, may cause the user to loose direct eye contact with the screen. The gestures for Piano-Pong might be better better suited for arm/body-centered actions, to issue commands to the paddle. Hence, I made the conscious decision to use the iPod only as a hand held controller that captures hand/arm movement and sends the data to a program that lives on a personal computer.
The graphical user interface for Piano-Pong is coded using the processing programming language. The interface itself is fairly simple, and consists of a paddle and a ball. The paddle can move anywhere on the XY plane. The background image for the GUI is that of Josef Hofmann playing solos in the Metropolitan Opera House. At the instance the ball strikes the paddle, the corresponding key is highlighted to make clear the connection between playing Pong and also the Piano.
There was almost no learning required to understand the interface or the interactions. Participants preferred and performed better on a larger display (wall projected), when compared to a laptop screen. This may be attributed to the type of gestures used in this interaction, which work better with arm movement, as opposed to wrist or palm movement. Smaller screens required a larger scaling factor to be applied to the gyroscope data. A second observation was the way in which the paddle was controlled. Participants initially only moved along the bottom of the screen, on the X axis, using a side to side motion. It was almost an accidental discovery to learn that the pitch varied along the Y axis, and then they moved the paddle along both axes. With movement along both X and Y axes, there was a tendency amongst participants to remain close to the upper bounds of the screen as it was faster to hit the ball. But, based on the audio feedback, they understood that a variation in the notes generated, can only be achieved by spreading their movement along entire width of the screen. There was a shift in their point of view, from game-play, in which the intent is only to hit the ball, to producing musical notes, which required a higher variation in paddle movement. This dual nature of the interaction was interesting to observe, as participants attempted to toggle between the two modes. One participant mentioned that if there was a scoring system, he would have focused solely on game-play, with audio serving as a feedback on the action. There would not have been a direct intent towards producing music