The Iamascope is emerging as a multimedia performance instrument from its roots as an interactive artwork. The Iamascope is captivating for an audience. It provides an avenue for expression of intimacy and control by the choreographer or performer. The two performances described here took place at Opera Totale 4, in Venice, and an experimental fusion of the Iamascope and Ato Cont Ato at the Petrobras Virtual Reality Exhibition in Rio de Janeiro. In these two performances, the Iamascope's potential began to be realized.
The
Iamascope is an interactive multimedia artwork. The Iamascope combines
computer video, graphics, vision, and audio technology enabling performers
to create striking imagery and sound. The result is an aesthetically uplifting
interactive experience. At an installation, the user takes the place of
a colourful piece of floating glass inside a computer generated kaleidoscope,
and simultaneously views the kaleidoscopic image of themselves on a huge
screen in real time. By applying image processing to the kaleidoscopic
image, the performer's body movements directly control music in a beautiful
dance of symmetry with the image. The image processing uses simple intensity
differences over time which are calculated in real-time. The responsive
nature of the whole system allows users to have an intimate, engaging,
satisfying, multimedia experience. The pictures on this page show participants
in the Iamascope and some of the images they created:
A block diagram of the Iamascope is shown in figure 1 (postscript, 37K). For input, the Iamascope uses a single video camera whose output is distributed to a video board with a drain to texture memory and the image processor computer. Imagery output from the Iamascope is displayed on a large video display (typically, 170 inch diagonal rear projection video). Audio output from the Iamascope is played though stereo speakers beside the large video display. In our current implementation, the video image from the camera is placed in texture memory and then the appropriate part of the video image (currently a ``pie'' slice also referred to as a segment) is selected to form the original image (O) which is used to create the desired reflections (O'). A multi-polygonal circle is drawn, upon which the alternating textures (original or reflected) are drawn. The necessary reflections for the Iamascope are simulated with texture hardware providing frame rates of 30 frames per second. This frame rate provides low-latency, high bandwidth control of the kaleidoscopic image, supporting a sense of intimacy with the Iamascope. The video image is copied into the memory of the vision-to-music computer. The image processing part of the vision-to-music sub-system extracts the exact same pie slice (O) from the whole video image as that used to create the kaleidoscope imagery. By doing this, only movements which cause kaleidoscope effects will cause musical effects.
A picture of a person using the Iamascope is shown above. Other types
of reflections can be made in the Iamascope, including 3 mirror kaleidoscopic
images, 3 mirrored images wrapped around a ball and many others. The figures
shown on this page illustrate some of the beautiful imagery possible with
the Iamascope.
The kaleidoscope sub-system and the vision-to-music sub-system are all written with OpenGL in C with a Tcl front end and a Tk based interface. The kaleidoscope sub-system runs on an SGI O2. The vision-to-music sub-system also runs on an SGI O2(or an Indy) at 15fps with full resolution (at half resolution it runs at 30 fps). The systems are set up in a client/server relationship and communicate using a TCP/IP connection with a Tcl protocol layer built on top. The vision-to-music sub-system is the server and the kaleidoscope sub-system is the client. Using the bi-directional communication channel, any changes to the settings of one sub-system synchronize with the other.
As the lights came up, Naomi Takano appeared
in black; only her face, hands and feet were visible. The huge screen behind
her buzzed with black static. She lept across the Iamascope causing a flash
of brightness; two harmonious notes resounded with the flash. She discovered
the Iamascope. She looked back to the enourmous screen, perplexed, then
moved into it again, aloof as a cat. Gradually, she discovered the flower
patterns of her hands and the beauty of her repeated feet. By the end of
the first movement of the piece, she had embodied the Iamascope. It was
now a part of her. 
She
stripped off her black clothes, limb by limb, revealing colourful garments
underneath. The large circular image of the kaleidoscope exploded with
the new psychadelic colour. Her dance filled the theatre with her magnificent
reflection. She danced both by herself and with the image she created.
Sometimes the image looked like an enormous flower, sometimes a moving
wheel; or her face appeared, a half smile becoming full, then disappearing.
The harmonic music, also played by her movements, engulfed the theatre,
as the audience was spell bound by the moving symmetric images, her teasing
dance, and the majestic sounds. She, too, began to watch herself. Gradually,
we watched her become taken over by the prismatic beauty of her own reflection.
The Iamascope embodied her. She had submitted to its beauty and now she
moved for it.
Now, the Iamascope embodied her. The image changed so that she was inside an enormous spinning ball, looking out at her kaleidoscopic image dancing into infinity. All she could do was dance, watch, listen and feel. The music assumed a dissonant life of its own, though still under her influence. She was ecstatic with her submission to herself. In a rising crescendo of movement and sound she recovered control, and once again held the reigns. The Iamascope responded, and she was back outside the large spinning ball. A calm came over her as the music drifted onto a shore of solitude. She was with herself and the Iamascope was itself again. She lay down to rest, the music died, and the Iamascope went dark.
Opera Totale 4 brought together some of the latest developments in the integration of music, video, gesture and technology for artisitic expression. The Iamascope illustrated a new direction for performance with a multimedia instrument. The main theme explored with the Iamascope was the duality of embodiment by the Iamascope of a person, and embodiment by a person of the Iamascope. Typically, as an installation artwork, the participant enters the Iamascope and experiences the emotions that come from having fine control of the kaleidoscope image and from seeing the beauty they create. When the participant is embodied by the Iamascope, the image is still of themself and gives a powerful, restful and satifying feeling. It is not threatening because they can still see themself, and so it becomes easy to submit to its beauty.
During Opera Totale 4 some of the different possible relationships between machines and people were explored through the use of the Iamascope as a performance instrument. Much as a musician plays a musical instrument, the performer, Naomi, was an imagician playing the image instrument. The audience empathized with her as she experienced the pleasure of discovery, mastery, control, and of being controlled. As a responsive instrument and machine, the Iamascope provides a way to explore our relationships with machines both as participants and as observers.
Another experiment used the Iamascope to explore the relationship between machines and ourselves at the Mostra Petrobras de Realidade Virtual Exhibition in Rio de Janeiro.
An impromptu performance was held in the Mostra Petrobras de Realidade Virtual Exhibition on June 3, 1998 in Rio de Janeiro, converging two exciting works of art and technology. The convergence marks a turning point in the integration of technology with dance, video and music. The two works brought together were AtoContAto and the Iamascope.
AtoContAto is a multimedia performance. AtoContAto, Portuguese for "act and contact", "act with contact" or "act with touch" seeks to make human gestures closer to sound as the dancer establishes close contact with music. Through a machine interface, the performer senses, touches and integrates music with dance. AtoContAto relies on a new gesture interface: tap shoes. Pizo-electric sensors are attached inside the taps, in the region underneath the toes and heel, with a cable terminated at that point. The total number of force sensors is limited to two per foot. The electronics were designed so that a MIDI control signal can be generated by the taps when dancing. This can be plugged into several MIDI devices. In AtoContAto, sonic messages are controlled by the dancer. During the performance, the dancer dances with video images of herself and is transformed into a musical instrument, or ``humanmatic device''.
The Iamascope and AtoContAto are complimentary. In the Iamascope, the performer dances with an abstract image of herself. She plays intimately with her own abstract image and becomes transformed in the graphical instrument. This is complementary to AtoContAto's musical instrument, the tap showes. The accompanying Iamascope music engulfs the performer and is controlled by her , yet remains outside her new machine/human boundary. The music is beautiful, but disembodied. The imagery is beautiful, and embodied. In AtoContAto, the instrumented tap shoes transform the dancer into a musical instrument. She embodies the taps and finely controls music and rhythm with her feet. With the Iamascope, the instrument and dancer come together. In the fusion of the Iamascope with AtoContAto images, body, music, feet and mind extend into one another.
For the impromptu performance, the dancer entered the Iamascope space wearing the instrumented tap shoes. She moved and her Iamascope image moved. She danced and the Iamascope music followed. The rhythm and the percussion sounds from the tap shoes mixed with the euphonic sounds controlled by the dancer's movements inside the Iamascope. The tap shoes did not provide a 100% match between the tapping movements and the sounds. This variability may have heightened the sense of wonder. An assistant changed the key sequence for the disembodied Iamascope music, overriding the usual automatic cycling. He also changed the percussion instrument controlled by the tap shoes to match the performance. In this experimental performance, only one sensor per shoe was working because the computer's audio port was being used to sample the sensors; thus, only left and right audio channels were available.
Here's a picture of Christiane in the AtoContAto/Iamascope fusion experiment:
This performance delighted the unprepared audience. The imagery flashed, the dancer engaged them and the rhythm intrigued them. For an experiment in convergence it was a great success.
This chance meeting of two complimentary pieces suggests rich possibilities for blending artistic and technological expressions. We plan to continue to develop and integrate the two pieces. Some of the immediate technological issues to be considered are these:
Artistically, there remains an enormous amount of exploration to do. Consider:
In summary, the fusion of these two works in the impromptu experiment at the Mostra Petrobras de Realidade Virtual Exhibiton created a new musical and image based instrument. The performer, or imagician, embodied the instruments and expanded her body both musically and visually.
The two experimemts, Opera Totale 4 and the fusion of the Iamascope and AtoContAto, show the promise of a new direction for performance using the Iamascope. The Iamascope allows a performer to both embody the image and to be embodied by the image. As a musician emodies a musical instrument, so too does an imagician embody an image instrument, the Iamascope.
Iamascope: A Graphical Musical Instrument, Fels, S. and Mase, K., accepted for publication in Computers and Graphics, 1998 (CGA98.ps.gz, postscript, gzipped, 4M)
MusiKalscope: A Graphical Musical Instrument, Fels, S., Nishimoto, K., and Mase, K., IEEE Multimedia Magazine, Vol. 5, No. 3, Jul/Sep, pp. 26--35, 1998. (IEEE.multimedia.1998.ps.gz, postscript, gzipped, 5.7M)
http://www.mic.atr.co.jp/~fels
Dept. of Electrical and Computer Eng., University of British Columbia, Vancouver, BC
Sidney received his Ph. D. and M.Sc. in Computer Science at the University of Toronto in 1994 and 1990 respectively. He received his B.A.Sc. in Electrical Engineering at the University of Waterloo in 1988 where he also studied cognitive psychology. His research interests are in human-computer interaction and neural networks. His works include Glove-TalkII, Glove-Talk, Iamascope, and MusiKalscope. Glove-TalkII was a system where a person could use the system to speak with their hands. The device was built as an virtual artificial vocal tract. The person using the system wears special gloves and uses a foot pedal. These devices control a model of a vocal tract so that the person can "play" speech much as muscian plays music. His collaborative work on sound sculpting is an extension of this idea to create musical instruments. The Iamascope is a interactive artwork which explores the relationship between people and machines. In Iamascope the participant takes the place of the coloured piece of glass inside the kaleidoscope. Their movements cause a symphony of imagery and music to engulf them.
My latest work with people and machines has been developed around the theme of embodiment. I see four different types of relationships between machines and people. The types of relationships are:
In each case the aesthetic derives from a different cause. In case 1, the aesthetic derives from the response the person receives from the object, whereas in case 2, the aesthetic derives from the actual control of the object. In case 3, it is self reflection that drives the aesthetic. In case 4, the person is allowing themselves to be controlled by the machine. Where does the aesthic come from in this situation? Submission and belonging are partly to cause. In the Iamascope, I explore these issues and the intimacy necessary to derive feeling in the different types of relationships. As an installation, participants can experience different types of relationships. In the experiments described here, we have used the Iamascope as a performance instrument in an attempt to express the emotion attached with our machines.
http://www.mic.atr.co.jp/~mase
ATR MI&C Research Laboratories, Kyoto, Japan
Kenji Mase obtained his BS degree in EE, MS degree and Ph.D in Information Engineering from Nagoya University, Japan in 1979, 1981, 1992 respectively. He was a visiting researcher at the MIT Media Laboratory in 1988-1989. He is the head of Department 2 of ATR MIC Research.
Naomi Takano has learned modern dance under Fujii Koh who has received the Purple Ribbon of Honor which is the most prestigious Royal Award in Japan given to a person of artistic achievement in 1995. Takano is Fujii's daughter and she has received a lot of prizes, including the First Prize in the Creative Dane Concours which is the most authoritative concours in Japan. She has danced in many Fujii's works like "Hokusai, Now", "Story of a Sleeping God", "Mummies in the desert". She established a reputation as a choreographer through her remarkable pieces "The Hero" , "The Sisters", and "Song of a Drunkard Angel".
Interdisciplinary Nucleaus for Sound Studies (NICS)- Univesity of Campinas (UNICAMP), Brazil
Composer and researcher who works with Algorithmic Composition and Sound Synthesis. He had his PhD degree in Music Composition (1993) at the University of Nottingham, England. He also studied Computer Music at the Institute of Sonology, The Netherlands (1991-92). His master degree in Applied Mathematics (1989), Bachelors of Music (1988) and Bachelors of Mathematics (1984) were at the University of Campinas (Unicamp), Brazil. Coming back to Brazil in 1993, he has been creating an interdisciplinary research group on Computer Music at the Interdisciplinary Nucleus for Sound Studies (NICS-UNICAMP). His works are related to instrumental, electronic music and interactive pieces for dance. His studies focus the relationship between human gestures and sounds, development of software and hardware for real time composition/performance applications. He also created the Gesture Interface Lab (LIGA) at NICS (1996) with that objective. His research pieces and artworks have been presented in several congresses and concert halls at USA, The Netherlands, Germany, England and Italy. He participates annually of the Brazilian Symposium on Computer Music (SBC&M), the Brazilian and International Congress on Cognitive Sciences (EBICC) and the Brazilian Congress on Computer Graphics and Image Processing (SIBGRAPI). At SIBGRAPI'95 he presented Trilhos Sonoros da Ferrovia, a multimedia performance involving electronic music, interactive gloves and dancers with fiber-optical vests. AtoContAto is a developing new interactive performance for music, dance and computer graphics.
Working with dance and music has become my main creative focus. Since 1992, I have developed a series of compositions searching for an integration between human gestures and music. It is an artistic attitude based on the idea that electronic based artworks should move to qualitative achievements to improve not only technologic perspectives of life but also the human interaction with this new reality. As a composer, I believe music, in its expressive side, touches the deepest contends of human expression. On the other hand, music is a very technical art and it is hard, to anybody to active high standard of musical performance. AtoContAto searches for an integration among human gestures, dance, music and computer graphics to diminish this gap between expression and technique. The title in Portuguese well defines the work: "Ato" play or action;" "Contato" contact; "Com toto" With touch. Thus my proposal is to use tap dance to explore and connect different materials (sound "and images) through an interactive process based on dynamics, gestures and expressiveness of the dance in real time.
Automation Institute - Technological Center for Informatics (CTI), Brazil
Artemis Moroni is a researcher in Computer Graphics at CTI , Centro TecnolÛgico para lnform·tica (Technological Center for lnformatics), a Brazilian federal research center in Campinas, S“o Paulo. Her master degree in Computer Science and Bachelors Computer Science were at the University of Campinas (Unicamp), Brazil. She has acted in the areas of Computer Graphics and Art & Technology since 1988. She has participated in the 20th, International Biennial of S“o Paulo, in 1989, as coordinator of the *.* Group, presenting the interactive installation with sensors, Fractal Art, which contained fractal images, sculpture and music. In this installation, the variation of the light due to the movements of the people was used as a parameter in the fractal algorithm applied to the music generation, in real time. Also, as coordinator of the *.* Group, she has participated in the 21st International Biennial of S“o Paulo, 1991, with the installation Foreseen Variations, where was presented a robot choreography. In 1992 the *.* Group presented the installation Aurora Beings at SESC PompÈia, in S“o Paulo. In this event an environment was created for children act as artists and engineers, programming choreographies for a robot, using pre-defined movements and music pieces. She worked as Brazilian Advisor in ISEA 93 - International Symposium on Electronic Arts, presenting the pannel "Three Interactive Installations, Three Moments", em Minneapolis, Minnesotta, USA. She realized the lnternet event "The Electronic Carnival', for ISEA'94, in Helsinki, Finland. She presented the workshop "Robot's Drawings" in Graphicom'95, St. Petersburg, Russia. Also in 95, she was President of the Art Committee of SIBGRAPI'95, the Brazilian Symposium of Computer Graphics and Image Processing. She participated in the electronic forum Memesis, in the event Ars Electronica, in Linz, Austria, in 1996, with the contribution "Holons X Memes". This participation has resulted in the installation Memetics, which was presented in BrasÌlia, Brazil, in the event Olho Lafino (Lafin Eye). In 1997 she was again the President of the SIBGRAPI'97 Art Committee.
The proposal AtoContAto evolved for me from a previous work, the Foreseen Variations. In this project we created a robot choreography, where a dancer realized a performance with a Puma robot. Since that now we have a mobile robot in my institute, we thought about creating a choreography exploring the interactivity between the robot and the dancer. The robot has sensors for detecting variations of light, Exchanging ideas about how to explore the aspects of the sensors and mobility of the robot, arose the possibility of putting sensors in a tap shoe to manipulate music and images... and we continued in this track! As a result for my research, AtoContAto provided a place, in the context of places that embody contextual information interactions and relationships. This includes physical properties that allow entities to express behavior that can be used by other entities, posing quesfions such as: How is the context shaping behavior? How does the environment permit or disallow the expression of properties of a single entity? How do the cooperative or combined behaviors of the resources - human or machine - create this emergent system? How do the work express different kinds of relationships and interactions within this environment?
Tap Dance Studio "Christiane Matallo", Brazil
Her Bachelors in Dance was at the University of Campinas (Unicamp). Brazil. She studied on Alvin Ailey Broadway Dance Center and staged in Steps, Wood Peackers, Van Porter, Savion Glover, Barbara Duffy, Germaine Saisberg, (Jazz) A. C. Koiiern Walsh and Burman Kunikova academies since 1990 until now. She owns a Dance Academy in Campinas in which a generation of young students are formed and a research on new experimental techniques on dance is performed. She has a strong appeal and talent for tap dance. Her main goal is to mix classical techniques, modern dance, tap dance and a strong jazz idiomatic in Brazilian culture. In 1988, as a choreographer, she won the ENDA - Encontro Nacional de Dan(;a (National Dance Meeting) with the piece Sol e Conhaque (Sun and Cognac), in the category amateur top dancing, She also won in this year the SESC MÛbil, in the category professional jazz. In 1989 she won the ENDA with the piece A TV manipula vocÍ? (Does the TV manipulate you?), in the categories amateur tap dancing and afro-jazz. In 1990 she won the ENDA with the piece Berlim, vamos danÁar em cima do muro... (Berlin, let's dance on the wall... ) in the category amateur top dancing. In 1991 she won the ENDA in the contemporary ballet category. She has participated in several ballet festivals. She has promoted two annual festival of her own studio. In 1996 and 1998 she has promoted a month tour at Broadway academies with the students (about fourteen) of her advanced group. In 1997 she started her research on interactive performance of AtoContAto using a new gesture interface: a pair of tap shoes, where pizo-electric sensors underneath the performer shoes control sonic transformations.
Dance is essential for my life. On parallel, I developed an intime relationship with music. My main goal is to open a wide profile and to mix classical techniques, modern, tap dance and a strong jazz idiomatic in Brazilian culture. I intend to be a human icon on the stage, thus my interest in new means of body expression. This is the reason for I am making research on using tap techniques to build a human/machine interface.