A walk-through experience of Aboriginal dance in the Welcome area of the Gallery of First Australians (GFA) at the new National Museum of Australia (NMA).
The Welcome is based on a cycle of six dances that reflect cultures from different parts of the country – a men’s dance, a women’s dance, a fishing dance, a drumming dance, a rainbow serpent story, and an urban dance. The six dancers are projected life-size onto the walls of the gallery – three on each side. Images from the GFA collection of paintings by aboriginal children are integrated with the dance and music. The image of a snake sculpture from the GFA collections circles the room in the period between dances.
The audience interface responds to footsteps in real-time from 32 square metres of vibration sensitive carpet. Six network synchronised PC workstations render interactive 3d graphics and surround sound effects over six data projectors and thirty speakers mounted in the gallery.
Barrass S. (2007) An Immersive Interactive Experience of Contemporary Aboriginal Dance at the National Museum of Australia, in Ros Bandt, Michelle Duffy and Dolly MacKinnon (eds), Hearing Places: Sound, Place, Time, Culture, Cambridge Scholars Publishing, pp. 111-125, ISBN 1-84718-255-0; ISBN 13: 9781847182555
The Air-Teapot challenges our reliance on sight to understand the world we live in and rebels against the visual dominance of computer interfaces. By tapping with the teaspoon, we can feel and hear a teapot in the space in front of us, but there is nothing to be seen but a teapot stand. This is a teapot with character that convinces you there’s more to the world than meets the eye.
Where better to foment a revolution in a teapot than Boston!
Here it is on show in the Teapot Exhibition at Siggraph 2006.
Barrass, Stephen (2006). Haptic-Audio Narrative: From Physical Simulation to Imaginative Stimulation. Lecture Notes in Computer Science. 4129, 157-165.
An Experimenta Media Arts Commission
They say that seeing is believing but this invisible installation has us questioning whether we can trust our eyes. Picking up the swizzle stick in The Cocktail Party Effect, we can feel both a cocktail glass and hear the tinkling of the ice cubes in what appears to be an empty space. Barrass and collaborator Chris Gunn have used state of the art haptic (touch) technology to create a sense-bending work that confuses visual orientation and challenges our reliance upon sight to understand the world we live in. Revealing secrets and sharing memories from its juicy past, this is a cocktail glass with character that convinces you there’s more to the world than meets the eye.
This paper describes the design and development of an interactive narrative for the ‘Experimenta Vanishing Point’ media arts exhibition in 2005. The Cocktail Party Effect tells the story of the imminent extinction of Great Apes in the wild using touch and sound in the absence of visual elements. The narrative is driven by haptic-audio exploration of a virtual cocktail glass which functions as a heterodiegetic narrator, and the traversal of cut-up conversations that make up the story within. The interface was developed through a series of prototypes that explored the perception and mental imagery of a haptic-audio simulation of the invisible glass. These experiments also developed narrative functions of the haptic-audio interface beyond conventional iconic metonyms to include grammatical and dramatic special effects. Observations during the exhibition show promising narrative engagement with the piece but identify problems with the clarity of the sounds, and a conflict between the narrator and the story content.
This is a video of the Mensx4 rowing finals at the Sydney Olympics in 2000. A slight improvement in technique could have made a big difference for Australia or Germany 
COST-SID Workshop on Sonification.
During this workshop our group chose to work on the case study of sonification of rowing.
The case study was provided by Nina Schaffert who was awarded the Best Poster Prize at ICAD 09.
Schaffert N. Mattes K. and Effenberg A.O. (2009), A Sound Design for the Purposes of Movement Optimisation in Elite Sport (Using the Example of Rowing), Proceedings of the 15th International Conference on Auditory Display, Copenhagen, Denmark, May 18-22, 2009. Download
Roderick Murray-Smith, John Williamson, Anton Schertenlieb, Stephen Barrass (camera), Nina Schaffert, Reiner Gehret, Hannah Buhl, Georg Spehr
Sin-ification – (Reiner Gerhet, Nina Schaffert)
This video shows the pilot study decribed in Nina’s poster in which a sine tone is used to sonify the acceleration of the skiff. From the video it is very clear that there is more information about rowing in the sonified data than meets the eye. The athletes and the coaches also found this sonification to be very promising.
The minimisation of the forward and back reversals, heard as the oars enter and leave the water, would improve the technique of the rowers and the speed of the boat.
The target users are elite rowers who may have expectations of pleasantness, enjoyability and musicalness of the sonification. For this reason our working group chose to explore approaches to the aesthetic dimensions of sonifications of the rowing dataset.
Xylophone MIDI-fication – (Anton Schertenlieb, Nina Schaffert, Stephen Barrass)
Would it make a difference to hear the sonification played on a more familiar musical instrument ? In this example the data is played on a Xylophone. This sonification was made by loading the data into the xSonify tool in order to map it to a MIDI Xylophone. It could just as easily be a piano, harpsichord, trombone or any other instrument. Does the choice of instrument effect the acceptance, understanding or perception of the data in any way ? The Xylophone was chosen because it has a short envelope duration suited to rapid playback. Since we were not able to render the MIDI to an audio file it was played out the headphone and recorded with Audacity on another laptop (please excuse some background chatter that was also recorded during the process !)
Ear-cleaning Sonification (Hanna Buhl and Georg Spehr)
Acoustic Ecology is a movement that seeks to encourage the appreciation and preservation of natural soundscapes. Ear-cleaning is a technique developed in acoustic ecology for learning to attend and observe the soundscape. This attention to the sonic environment may be very helpful in sports activities such as rowing where the eyes are already busy. In experiments it was found that when rowers wore ear-plugs their performance was reduced. The skiff is full of sounds produced by movement through the water, the oars, the rowlocks, the seats and the physical exertions of the rowers. The rowers also listen to the other skiffs in a race. This raises the possibility that the rowers could be taught to listen to auditory cues about synchronicity, rowing motion, and acceleration that are natural consequence of rowing.
Here is a recording made inside a rowing skiff.
Ecological/Metaphorical Sonification (John Williamson, Roderick Murray-Smith)
The soundscape in a rowing skiff includes a range of natural sounds that can provide a basis for auditory metaphors in a sonification design. This approach is similar to the Auditory Icons based on everyday sounds that have been developed for auditory interfaces. In this sonification computational models of splashing, wooden and metal impacts communicate the context is rowing. Impacts on these models are triggered by events in the data such as turning points and thresholds in the acceleration and the rate of change of acceleration. The sonification was produced in a unified Python programming environment with python(x,y) for data processing and pysndobj for sound synthesis and sequencing.
Techno Music Probe (Stephen Barrass)
This is an exploration of the concept of music-alisation in the techno genre. The software Albatat Muse was used to generate the soundtrack from movements in the video. This is a probe, rather than a prototype. The only link between the soundtrack and the video is the syncing of the music generated at 152 bpm with the movement of the oars at 38 (x 4 = 152) strokes per minute. The changes in the pitch corresponding with the drive and recovery phases, together with the sync of the percussion, raise interesting possibilities for a musically based sonification of the data.
Music-ification prototype (Stephen Barrass)
This is a prototype inspired by the Techno-music probe. The sonification was produced by loading the data into Sonification Sandbox. This first required the data to be adjusted with Excel so that all values were positive. Once the data was mapped to MIDI the output file was then loaded into Garageband. This file was then rendered using the FM Percussive synth, and the Metal Pad. Unfortuately the tempo of the MIDI data was not saved in the file transfer of the MIDI file from Sandbox, and Garageband only allows 240 bpm which is not fast enough. This meant the rendered audio had to be tempo adjusted 12x using Audacity. A narrative element was introduced by adjusting the mix over the course so that a change in the music conveys the final push to the finish.
PAF Streaming (Reiner Gerhet and Stephen Barrass)
This sonification is an extension of the PD patch used to produce the original Sin-ification. The oscillator was replaced with a Phase Aligned Formant (PAF) synthesiser. The acceleration maps to the pitch (as before) and the derivative of acceleration maps to both the width and centroid of a formant. This causes moments of rapid change in acceleration (or jerking) to emerge as a distinctive figure against the smoother bass ground. This is an example of Stream-based Sonification in which auditory figure / grounds emerge from relations in data, rather than from pre-determined top down “channels”.
Through the workshop we learned of the work of Chris Henkelmann who has been exploring ways to improve the aesthetic quality of realtime sporting feedback.
You can find his examples of PAF, tristimulus and other synthesis techniques applied to rowing data in the examples.zip at the bottom of this page.
The variation in these examples demonstrate the immensity of the space of sonifications that are possible for the same dataset. A blank canvas and the broad palette of synthesis algorithms raises the question of how to go about designing a sonification. The sonification design needs to balance functionality and aesthetics in a brief that includes users, tasks, and data, and must also consider learnability, context, aesthetics, emotive aspects. This is where a comprehensive method for designing sonifications can guide the string of decisions at many different levels that are required for a sonification that works in contexts beyond the scientific laboratory or the media arts gallery.
Sonic Branding (Hanna Buhl, Georg Spehr, Stephen Barrass)
There is also reseach on the sonification of rowing in other countries, such as Sweden. We discussed the possibility that the sonification could also include some reference to the national identity of the rowing team. Could the national anthem be part of the sonic branding ? We look forward to the Sonified Olympics where rowing teams from Germany, Sweden and Australia have sonifications in their training regime. May the best sonification win 
sketch for a scarf that amplifies interpersonal electricity
In Paris it is very fashionable to wear a large colourful scarf. It is also very fashionable to greet friends with a bisous kiss on each cheek. L’Escarpe a’ Bisous is a Parisian style scarf that raises the wearers awareness of interpersonal electricity during this greeting.
Materials: Scarf, taxtons, efferent nerve wires.
Size: 2m x 0.5m
The Scarf was curated by Sarah Kettley for a New Media Scotland event WeWearWearables.
taxton sewn in with nerve wires
paris style scarf tie
bisous
Mark in the Bathroom is a transformation of Anny in the Kitchen that shows the plasticity of meaning in sound, and an extension of asynchronous and contrapuntal film sound to theatre performance.
Anny in the Kitchen was an audio sketch for a female actor playing sounds with kitchen implements composed for the COST-SID Workshop on Theatre and Sonic Interaction Design. On the day before the workshop we faced the situation that 1) the audio sketch was to be used directly as a soundtrack for the theatre performance, and 2) that the only available actor was male. This led to a round of brainstorming in which the actions of a man in the bathroom in the morning were substituted for the original sounds designed to be produced by the actions of a woman in the kitchen in the evening.
The workshop on Recycling Auditory Displays at ICAD 08 was organised by Christian Frauenberger and Stephen Barrass for ICAD 08 at IRCAM in Paris.
This year we presented the results at ICAD 09 in Copenhagen.
Barrass S and Frauenberger C (2009) A Communal Map of Design in Auditory Display, in Proceedings of the International Conference on Auditory Display, 18-21 May, Copenhagen.icad09-paper.pdf
Abstract
The workshop on Recycling Auditory Displays at ICAD 2008 aimed to capture knowledge about the design of auditory displays from the participants in a manner that would be easy to understand and reuse. The participants introduced themselves by providing examples of a good and a bad sound design. These examples raised issues of culture, identity, aesthetics and context that are more usually associated with product sound design than auditory display. Based on these discussions the themes Users, Applications, Techniques, and Environments were chosen to focus the further development of ideas. A mindmapping session was used to collect over 150 entries under these themes, and more than 30 references. An additional Others theme was needed for ideas that did not fit neatly into the existing categories. The information that has been collected shows that most research in auditory display falls under the themes of Applications and Techniques. The information under the themes of Users and Others shows the overlap with related disciplines such as auditory neuroscience, product design, sound arts, semiotics, and interface design. The Environment theme raised the need for future research to include contextual issues. The outcome of the workshop has been to produce a collaborative understanding of the current state of design knowledge in the Auditory Display community, and to identify future directions for research into the design of Auditory Displays.
Statue of Lur players in the main square of Copenhagen
This sonification was designed for the Sonification contest at ICAD 2009 in Copenhagen.
Here is a 3 minute excerpt from the 20 minute piece generated from a Yeast gene. This excerpt contains a segment of junk followed by non-junk.
This sonification explores the intersection of functional and aesthetic aspects of sonification. The functional design uses an emergent motif to signify the start codon. The aesthetic design includes the choice of the Lur (a Viking instrument), and the introduction of pauses for musical effect modelled on a piece of Lur music composed and played by Odd Sylvarnes Lund.
- The Lur is a large curved horn the size and shape of a mammoth’s tusk. As you can see from the famous statue in the main square in Copenhagen, the Vikings played two together in left and right pairs.
Non-junk (intron) regions of DNA start with the sequence ATG (start codon), and end with one of TAG, TAA, or TGA (stop codons). The notes for each DNA element were chosen so that the sequence ATG is a downward fanfare. This fanfare is played 3 times at the start so you can listen for it in the rest of the piece. However you must pay attention because it only occurs once on the bondary between the extron and the intron. As C does not appear in start or stop codons it has been given a very low note that falls outside the motifs formed by the other elements.
- Made with samples from the Lur composition, Python for data processing, csound for audio rendering, and Audacity for post-production.
A method can provide creative inspiration and theoretical guidance or the designer faced with a blank slate and wide open space of possibilities.
This method for designing sonifications has threads drawn from user-centred design, sound art, film sound, semiotics, perceptual psychology and auditory scene analysis. This mesh of methods addresses affective, schema and perceptual aspects of sonic experience.
Its a meshodology
* Urban Sound Planning | Stadt(Klang)Planung
* Functional Sounds | Funktionale Klänge
* Sound Branding | Akustische Markenkommunikation
Here is a summary in an interactive MindMap where you can click on the links to unfold branches and hear examples.
Listening to the Mind Listening Poster
Listening to the Mind Listening (LML) explored whether sonifications can be more than just “noise” in terms of perceived information and musical experience. The project generated an unprecedented body of 27 multichannel sonifications of the same dataset by 38 composers. The design of each sonification was explicitly documented, and there are 88 analytical reviews of the works. The public concert presenting 10 of these sonifications at the Sydney Opera House Studio drew a capacity audience.
Clck here to listen to the pieces on the Listening to the Mind Listening Concert site.
The inspiration for the concert came from a conversation with a nueroscientist working with MEG recordings of babies in the womb. These recordings are very noisy and she wondered whether by listening to them we might hear “little tunes” of mental activity through the noise. For more background on the development and production of the concert see this interview on the ABC Radio series All in the Mind.
We found that there were generally 4 main stages in the design of the sonifications that were submitted to the concert. Although all produced from the same data the pieces ranged widely in musical genres. There were significant differences between reviews of the sonifications depending on whether the reviewer was a sonification researcher, music composer, nueroscientist or general listener. These analyses are described in this publication …
Barrass, S., Whitelaw, M., and Bailes, F. (2006). Listening to the mind listening: An analysis of sonification reviews, designs and correpondences. Leonardo Music Journal, 16:13-19.
The concert was produced at the Sydney Opera House Studio using a Lake DSP processor to provide high quality spatial audio rendering through a custom 16.2 dome of speakers arranged to mimic the positions of EEG electrodes on the scalp by Guillaume Potard. Composers were offered several options for spatial formatting, allowing them to either use the original scalp electrode positions, or arbitrarily reconfigure the original dataset within the speaker array. The technical details are described further in this paper….
Barrass S, Whitelaw M, and Potard G. (2006). Listening to the Mind Listening. Media International Australia incorporating Culture and Policy, Special Issue on Practice-based Research, 2006(118):60-67.
