Talk2Me - The Art of Augmenting Conversations

INTRODUCTION

I presented the majority of the paper below as a short paper at ACM Multimedia 2006 (ACM MM2006) in a poster session. I am sending in this paper to the Graduate Student Symposium at 6th Creativity & Cognition Conference 2007, as I think it describes the project well, will not be part of the published outputs in this context, and as such I trust doesn’t breach publishing conventions. Attending ACM MM 006 and other conference/s, presenting my work, engaging with other works and in discussions both in the workshops and as part of the sessions has proved an invaluable experience. I found myself to be part of a larger research community, and noted researchers from different disciplines grappling with similar or tangential questions to my own. However, many of the researchers presenting have long-finished their PhD’s and the majority come from a more technical background. While I find this engagement extremely useful, I feel it would be very beneficial to get feedback on the actual PhD itself from this graduate symposium with its larger supportive community designed to address the issues particular to graduate research. I am specifically keen to get feedback from a community that focuses on promoting creative engagement. This for me is the very essence of my actual PhD research. In my research I discuss engagement, and in particular within interactive installation or physical computing environments. My focus however, is on engagement that is not primarily functional, but promotes instead a playful engagement, and encourages joyful and non-useful responses—a creative engagement that is too-often forgotten when working with technologies.

1. WHAT IS IT?

Talk2Me is an interactive installation housed in a large portable light-permeable dome. The audience interacts—via microphones, SMS texting, and remotely—with characters (bots) in a text-based virtual environment (a MOO). A MOO is an electronic space many can log into, and type to communicate and build objects, such as rooms, recording devices, or keyword-responding characters like the talking bots in this particular MOO. Two versions of this MOO work, The Word: The Wall, have been exhibited previously as an online performance. The work I discuss in this paper is a later version, exhibited in February 2006, named Talk2Me. This version is housed in a large dome space, but still runs the original MOO database. New ways in which a participant-audience can interact with the bots in this MOO are being explored in successive iterations. Talk2Me is a continually developing work in progress that has had one exhibition showing in the early stages of its development. In the current version the participant-audience members speak into one of the three suspended microphones in the space, or they may SMS a message to a mobile number. The speech and/ or the SMS text is then converted to text within the MOO database. In turn the bots’ responses, (they respond to designated keywords with a text output), and the SMS and online inputs are converted to audio and heard through one of the three suspended headphones in the space. The input/outputs are also displayed as text on one of the three screens in the dome environment as well as online through a client login interface developed for this work. The SMS and online inputted texts are also added to the database, so not only are they spoken and displayed at the time, but they also become future random outputs of the work. In this way the participantaudience members act in co-authoring the work. The bots respond to keywords from the audience, sometimes to each other, and sometimes randomly. The participant-audience can also respond and engage with each other in all of the above ways, as well as by simply talking to each other in the physical space.

The exchange is not one of information, rather of play among all the involved parties. Talk2Me works with error and imprecision, privileges humour and absurdity, and embellishes the resulting play and synchronicity. There is joy in the pleasure of language, “the turning of a phrase” [1] in an environment designed to engage its participants, and create its rules on the fly. The work has its own ‘underbelly’, shown in the responses given both by the bots and by the environment itself. While the MOO presents as friendly enough, there are unexpected sub-texts, and both abstract and bewildering responses, with underlying themes that emerge over time. The different bots have their own agendas, so one participants’ ‘conversation’ can be vastly different to that of another. An experience for one may be with the Eliza ‘bot’ engaging in a conversation of constant interrogation; another may interact with COG, the robot dog in this MOO, who requires constant patting; whilst yet another may find themselves discussing hairiness, calamine lotion, or yoghurt with the random output bot named Wall.

2. EXAMPLE OF INTERACTIONS

This is an example of some of the outputs from Talk2Me, Feb
2006, Brisbane. The legend is: Black text for Audience Speech; Dark
Grey Text for Bot Response to the Speech; Light Grey for the Random
Wall Bot Outputs
talk
talking and talking and talking till the talking was done
backed up: saved or sinking
In the land of the MOO, verbs really do act as agents, as in they 'do'
things...
hardly
there were leaves high in the air and the smell of rain
there was hardly time to be thinking, let alone waiting
do you think we can resist forever?
we were winding and winding until it was wound
delusion and delirium swimming in textual sea
it's a day in which my hair is too tight u know
generous
generosity abounded
if the house is dark does it mean it is empty
that was a hairy hell of a notion
a hyphenated lucidity

3. THE WORK

Many of the design considerations for the work Talk2Me developed from the initial choice of a dome shape to house the work. The words ‘domestic’ and ‘dome’ stem from the same root domus (the house or home). The dome also carries connotations of the Western monumental religious space of the duomo, or house of god, and of the non-Western, transient, and nomadic space of the ‘yurt’. It was this play of opposites—between familiarity and unfamiliarity, between a sense of space at once domestic and ‘other’, as well as my own experiences in dome architectures that drew me to this shape. I wanted a space without hard edges or sharp corners, somewhere that was easy to move in, and sympathetic to human form. I looked for dome structures and portable structures and ended up with the light-weight dome shown in Figure 1. When I first received the dome it was smaller than I had envisaged, more house-sized, more homely, less exhibition-sized. I had to rethink my design process of the work to ‘cope’ with this altered sense of space and actual available space.

Given I wanted to create people-friendly spaces, this has worked well. I have created a homely environment with familiar enough technologies, such as speech-to-text, which many participants are likely to already have experienced in some form. However I have factored in some disjointedness or difficulty in the actual interaction. This is not intended to be difficulty that prevents interaction from taking place, instead, the ‘difficulty’ is designed to increase with the depth of the interaction. The familiar becomes unfamiliar or is used in unexpected ways. The word ‘tent’ stems from tendere, meaning to stretch, and in this sense, to engage. As Csikszentmihalyi argues, to maintain optimal engagement, tasks need to be within the realm of the possible, but must still stretch the participant. [2]

In Talk2Me the actual tools for interaction do not differ widely from those used in locative-experience works, and the artifacts are similar to those used in site-specific works. A point of difference is the addition of a container that envelopes the participants and the experience. The contained interactive environment of the dome structure acts as an auditorium and a projection space, operating as an autonomous environment, independent of its surrounds, and limits intervention from outside noise, light, and external events. Feedback from the work’s first exhibition suggests people feel safe within the space, and the dome is traditionally viewed throughout Eastern and Western cultures as a safe, spiritual place. I was surprised by the recurrence of the word ‘safe’ in this feedback, as I have never considered public gallery space, or for that matter any pre-existing installation work of mine to be ‘unsafe’. The work was shown in a very large open space and people reported they took comfort in entering a smaller room, and in so doing also retreating from the larger room, often returning several times in one visit. The most frequent feedback I received was surprise at the content in what was assumed to be a ‘technology-driven’ work. The voices were found to be agreeable and seductive to listen to, enticing people to stay longer. The most observable ‘oddity’ was that people felt very comfortable to shout loudly in the dome even though the walls are paper-thin. I think the tent-ness of it really worked. People relate readily to portable nomad-type spaces, and are happy to set up and bunk down and try out new things in a camping-type setting.

4. ARTISTIC MOTIVATION

My motivation is to translate a compelling and real-time ‘interactive writing’, nose-to-screen work, and reinterpret it into a physical interactive installation without losing the original spontaneity and engagement of the participants. One aim is to maintain the priority for the participants to be involved in content development, as happens in the screen environment. Another is to extend the interactive environment in ways that will entice and include a ‘casual’ audience, as well as those who already ‘know’ how to do it. This means the engagement and interactive ‘tasks’ and levels within the work need to cater for the needs of participants new to interactive technologies, and for those of experienced players.

In order to do this I needed to develop a familiar-enough environment for people to move around in and interact with each other and the technologies in an easy enough manner, yet also introduce elements of uncertainty to keep the activities, responses and interactions slightly edgy. I needed the audience to be active and to try things, but not feel as if they were on display or performing for others whilst doing this. I wanted to develop an environment that people felt at ease to be playful in: to join in a different kind of ‘joust’. [3]

The MOO environment on which this installation is based had developed over time to become a kind of micro society or system with its own modes of behaviour and rules: rules that did not become clear until somebody broke them. [4] The originating interactive writers in the MOO were Anna Cicognani (Italian architect and originator of the MOO environment for educational purposes); Zina Kaye (Polish/ English journalist); and myself, Ann Morrison (New Zealand artist, at the time studying Computer Science, and director and conceiver of the work). We deliberately played with the poetics of language, discussing our everyday lives in non-everyday language. We took great pleasure in jumping rapidly between, and extending, developing themes, and in building on shared meanings as though we were forming a foreign or pidgin language. This direction was not pre-determined, rather it evolved naturally among the players.

In this physical conversion of the original MOO, it is my intention that the participants’ experience unravel in a similar pleasurable manner. The title Talk2Me calls the participants to action—to speak in return. To respond in a ‘game of language’ is to make the next ‘move’ or a countermove to what then becomes the first move. [5] Co-authoring is important so participants can makemoves that are more than reactions, where they can take pleasure in being inventive with their countermoves. A ‘good’ move alters the other player, displaces and disorients them because it is unexpected. A ‘good’ game is full of unexpected moves, with continually heightening levels of displacement. Speech acts within this context in particular, fit within the domain of joust rather than communication. [6] To make the next move is to join in the tryst, to take part in the joust. Speakers, unbeknownst to them, enter into a contract of moves.

5. RELATED WORK

The original text-based interactive writing MOO that Talk2Me is based upon has been previously exhibited. The work—titled The Word: The Wall—constantly doubles over the pleasurable use of everyday language with the functional use of coding language. The instigating members both write creatively and program code and this collapsing of the usual distinction between the two modes of language use was one of the driving forces behind the work. The Word: The Wall and Talk2Me foreground and perform Foucault’s assertion that words are active agents [7]—shifts in language in the MOO not only change what can be said (the conversation), but also modify the space in which it can be said (the environment). Language in the MOO space is used as both a set of constraints, and enablers, that guide the activities promoted as well as building augmenting environments.

Contemporary works by Susan Hiller and Simon Penny display a similar attention to language and immersion, and offer innovative analogies for machinistic embodiment.

In Susan Hiller’s work, Whispering Voices (Biennale of Sydney 2002) the sound-scape can be heard as one merged ‘noise form’ (babble) or isolated as disparate sources, with personal narratives revealed. For the audience Hiller’s installation proves an immersive experience, as they drift through a field of other people’s discussions. In Talk2Me, the constant murmurings of the bot’s conversations within the physical dome environment act to immerse the audience in a comparable fashion.

In Simon Penny’s early work Petit Mal, Penny opposes the idea of promoting the Cartesian ideal of the mind-body split or of replicating anthropomorphic or zoomorphic behaviours within a robot. Petit Mal is an attempt to build a robot whose behaviour arises from the dynamics of its ‘physicality’. Moreover, the robot’s reactions to people give the impression of a machine intelligence that is unique to its physical and electronic nature. Petit Mal is not reducible to, or modeled upon, an automaton or simulation of some biological system.

In a similar manner the robots in Talk2Me do not pretend to be human, emulate human voices, or even begin to attempt to hold human conversations. Their embodiment and personas are derived from code and language, and their behaviour is a play on such. For example, within Talk2Me the phrase “the walls have ears” has multiple functions: it refers not only to an every-day world saying, and an actual event that occurred in the MOO, but it also refers to three functions of the MOO. In the MOO, a recorder is placed in every room, conversation and activity at each interface can be logged, and all activities are added into the database as a standard function. As such, the bots continually reference their own actual world in their ‘conversations’. As participants we encounter another kind of interlocutor in this bot-inhabited mini-society.

6. INTERDISCIPLINARY METHOD

‘Critical Technical Practices’ (CTP) is a term coined by Philip Agre as a way to discuss technology not simply as a means in itself, but also as a way to “reflectively explore the assumptions and attitudes that underpin ideas about technology”. [8] Constant changes in technologies also mean we need to keep updating the methods used for critique. An emerging community of researchers working within the inter-discipline of CTP comprises computer scientists, engineers, AI researchers, artists, and cultural theorists: people such as Phoebe Senger, Philip Agre and Simon Penny. For many, CTP necessitate re-thinking the ways of doing computer science, with an increased focus on the social and political ramifications of this technology. Penny sees this re-focusing happening also within the art world, and as Penny rightly points out there are some pitfalls in this approach. Art production as direct implementation of theory or social political rhetoric, no matter how sincere the intent, produces art that is ‘underwhelming in its presence’—literal re-representations that undermine the potential impact of the political concerns. To quote a saying from the MOO space of Talk2Me: ‘Dis-functionality should be avoided everywhere except in the artwork’. An artist may take the liberty in art to explore more fully what, in life, it may be more productive to avoid. Considering art as a critical technical practice calls for the artist to foreground different or altered ways of experiencing or interpreting what our by-necessity pragmatic lives prevent us from pursuing.

CTP also hold values in common with experience design. Experience Design’s main exponent, Nathan Shedroff, discusses how considering all attributes that make an experience unique (including the senses and spatial dimensions) “is an approach to creating successful experiences for people in any medium”. [9] Similarly CTP hold a deep investment in the material practice of designing and building technical works, valuing the phenomenological experience of the ‘thing-ness’ of the objects and the objects’ own interactions with each other’s thing-ness. [10]

Penny argues that knowledge, techniques, and methods from the technical sciences, the humanities, and the arts need to be combined in new ways. He describes these as foreign disciplinary languages that need to be understood in their own contexts. [11] To work in an inter-disciplinary manner these ‘languages’ deserve more than a ‘copy and paste’ of eligible bits into the artist or practitioner’s own master discourse.

There are many tensions in working in an inter-disciplinary manner. Electronic/new media artists based in the academy are often caught in a double bind between Art and Information Technology. While they may build systems or art works that incorporate some form of critical engagement with technology, their concerns are formulated from within the discipline of conceptual art practice, and they are often ‘housed’ in the academy within the discipline of Information Technology (certainly true of this artist). In my case, it is not principles of IT, but post-object art, performance art and site-specific art, that form some of the core understandings from which my work is then built. Electronic/new media artists often find themselves working within or around the academy and answerable to funding bodies and disciplines that privilege their own trajectories. For example, an installation might be referred to and funded as a case study within an IT school, yet exhibited in the art world, where a whole different set of priorities and specialized languages and conversations are in play. The makers of these works are answerable in comprehensive ways to multi-disciplinary knowledges, which they must-needs adequately grasp and reflect in order to successfully navigate pre-requisites for competitive funding and exhibition. This may give onto productive tensions in the making of a work, but not necessarily so. The artist is required to integrate the breadth of applicable components within the fields of visual art, technology, and the humanities, while at the same time being answerable to ‘experts’ from all these areas.

In short, as an artist working with inter-disciplinary and critical technical practice methodologies I needed to consider and weigh a number of issues, including art practice concerns; experience concerns; the need to speak, think, and operate in many tongues; as well as the physical-material aspects of my work.

7. TECHNICAL DETAILS

A web cam file transfer protocols motion-sensor activated images to an updating web page using JavaScript and php, recording and archiving interactions in the space. A mobile phone is connected to a computer that runs a program to retrieve messages sent to the phone by participants. These messages are inserted into the MOO environment via a virtual client.

Mac OS X speech technology is modified to act on certain words or phrases, which are then pasted into a telnet client that operates as the audio input interface for the MOO environment. A Savitar MOO client displays the visual output and manages the audio output. Triggers are set to speak regular expressions. A variety of Cepstral voices pick up on different words and themes. The voices were selected for their ‘not-machine’/‘not-human’ quality—not the manufacturer’s intentions I am sure—but for the purposes of this work these voices fit the idea of machine-inhabitingcharacters with active personality ‘twitches’.

Apple scripted auto-starts for speech, MOO server, telnet and Savitar ensure the environment is self-managing on start-up. Speech technology requires the use of three systems (Mac minis): one runs a MOO server; all run MOO clients and speech technology and connect via a LAN. Three microphones, suspended in the space, run via a splitter through a Firewire Audio interface into the Mac minis. Audio-out runs via the splitter to suspended headphones. Remote desktop to the three computers from an alternately situated computer ensures ease of access without major disruption, in particular whilst the exhibition is running.

The whole work is housed within a 6 metre wide by 3 metre high portable dome. I obtained ethical clearance and interviewed participants in Talk2Me in February 2006 using standard HCI/ Interaction Design style user surveys, gaining valuable feedback.

There were difficulties with screen and interface projection in a light-permeable rounded walled environment. I had also underestimated the instructional design required to interact with the work. The biggest misconception was that the speech was some kind of karaoke-style interaction. Naively I had initially hoped to make a solely audio work. I found that just as I had needed visual feedback in the making and fine-tuning of the interactivity with the MOO and the bots, so too the participants needed the same degree of visual feedback. Earlier technical difficulties meant I ran two versions with different means of interaction over the timespan of the exhibition. Feedback suggests that people were happiest with the first version that they had learned to interact with, and what I perceived to be the second significantly enhanced version took away a part of the play and sense of agency that the earliest participants had experienced, diminishing their next experience. Their disappointment was palpable. I had underestimated how invested these participants had become with their ability to successfully engage (and demonstrate and then facilitate others into this engagement) with the participants of this minisociety to which they had also become a part.

8. FUTURE WORK

Talk2Me had a second iteration with a small showing for filming in order to demo at an international conference in October, 2006 at a local development site in Brisbane. The participants were selected from the first interviewed participant member group. I implemented many changes from the feedback from Talk2Me, Feb, 2006. I added seating, made further information on the work available to the audience, and providing insight into the automata that form part of the interactions. The technology infrastructure of the installation was visible to the participants, sitting on vibrant coloured piles of felt. The screen interface, previously provided inside the curved dome was projected onto a suspended white wall within the dome. I added touch sensors so participants could easily flush the speech buffer when the bots get into a rant. They simply stomp on a stomper, any waiting speech was cleared and ready for their speech input. Most importantly simple clear instructions on how to work the installation were beside each participants seat. These changes answered most of the requests that came from the first feedback sessions.

Future work planned for this series will move from text to a visual and to an audio predominance. I could easily see these works—or indeed further iterations of the speech work—happening outside of the dome environment. On exhibiting a work the project is captured in stasis for an instant—presenting as an inert moment in its’ continuum. To build an ever-expanding work, there needs to be resistance to—and an engagement with—the ‘stasis’ that the artefact holds, enabling the iterative continuum to proceed. Future other works planned for the series of works relating to the dome environment, will move from text to a visual predominance, with an Interactive Painting work implementing RFID or natural interaction technologies. In this case, the audience changes the visual display—projected as a painting-like image in the dome—by way of their physical movements.

Talk2Me is the first iteration of a planned series of works designed for a dome space. Feedback on the Talk2Me, Feb, 2006 showing suggests a number of improvements could be made, such as including seating space, making further information on the work available to the audience, and providing insight into the automata that form part of the interactions. Interest was shown particularly in the bots: the audience would often ask who they are modeled on, how they work, and what a MOO is. Permission can readily be upgraded for dedicated participants, so they can join the MOO and build their own bots, rooms, and activities at programmer level. The technology infrastructure of the installation will be visible to the participants. The screen interface, previously provided inside the curved dome, will be projected externally to create a wall of interaction framing the work—a dome inside walls.

ACKNOWLEDGMENTS

My thanks to Interaction Design Research Division, University of Queensland, and in particular to Anna Cicognani and Zina Kaye; and Lorna Macdonald, Andrew Dekker, Jonathon Barrett, the ‘yurt raisers’ and Lubi Thomas.

9. REFERENCES

[1] Wittgenstein, Ludwig. Philosophical Investigations. New
York: The Macmillan Company, 1965.
[2] Csikszentmihalyi, M. Flow: The Psychology of Optimal
Experience, New York: Harper and Row, 1990.
[3] Lyotard, J. The Postmodern Condition, Univ.of Minnesota
Press, 1999.
[4] Wittgenstein ibid, secs 65-84.
[5] Searle, J. Speech Acts. Cambridge Univ Press, 1964
[6] Foucault, Michel. Language, Counter-Memory, Practice. Ed.
Donald F. Bouchard. Ithaca, New York: Cornell U, 1977
[7] Agre, P E. Computation and Human Experience, Cambridge
University Press, 1997
[8] Shedroff, N. 2001, Experience Design 1. Indianapolis, Ind.,
New Riders
[9] Heidegger, M. “Poetry, Language, Thought” Harper & Row,
NewYork, 1971, p. 149
[10] Penny, S. On Interdisciplinarity. An Interview with Simon
Penny. [iDC] http://mailman.thing.net/pipermail/idc/2006-
March/000270.html
[11] Searle, J. Speech Acts. Cambridge Univ Press, 1964

Last modified 15 February 2008 at 4:22 pm by hodie