Folks,

I am happy to announce that I will be a Fulbright Scholar at the Salzburg University of Applied Sciences MultiMediaArt department this Spring.  While there I will be designing a new tangible system for archiving narratives of refugee migration through Central Europe.  This project is based on my previous work, If These Walls Could Speak.  Here’s an excerpt from my research proposal, written over a year ago:

I create intermedia installations and experiential systems that bridge the physical and digital worlds. These systems manifest as freestanding interactive sculptures, immersive installations, and choreographed performance pieces. Dynamic media and a consideration of how these forms exist in three-dimensional space tie the works together. These works engage viewers with audio and visual feedback, enticing them to become participants. With a background in furniture design I am interested in how the body adapts to physical spaces, and through embodied gesture can generate meaning for an audience. My work blends new media, physical computing, relational aesthetics, computer programming, and traditional sculpture processes. Sculpture and new media works are unique in their ability to tie together ideas on material, space, site, and digital culture. Ultimately, I choose my media and materials based on what will best support the concept of the piece and convey the message of the work. My goal is not to inflict opinion on others but to begin dialogue about important issues that are often swept aside. For example, I have recently completed a sculptural data visualization of gun violence in the United States. The installation is neither for nor against gun control, but inspires people to discuss it. In doing so my work brings people together and provides them with a heightened sense of self-discovery and reflection.

My research proposal centers on creating new systems for tangible non-linear audio storytelling. I plan to design, prototype, test, and build this system while in Salzburg, and have given it the working title Namthar. Namthar uses physical objects as inputs to trigger playback of audio stories. The dynamic media system uses RFID tags embedded in physical objects to create a tangible and audio interface. When tagged objects are placed on a custom built RFID tag reader, Namthar plays snippets of audio narrative associated with that object. Each story will be comprised of several audio snippets linked to a small collection of 3-5 objects. The narrative that the objects reveal will depend on the order and number of times the objects are placed on the reader.

For the content of the narratives I would like to work with local refugee populations to capture their stories of migration, as well as those from people who would prefer tighter borders. The goal of this project is to foster dialog, understanding, and respect between people from both communities. After initial story gathering I will begin designing the Namthar system through sketching, 3D computer modeling, and prototyping the electronics. With a working prototype I can test, evaluate, and revise the system. Finally, I will record more stories and make a final version of the system in final materials. Possible outcomes of this system include interactive museum exhibits as well as tools for increasing understanding and communication across cultures through storytelling.

The context for this research is the intersection of the academic and professional fields for tangible interface design and fine art. Current research initiatives in this area include systems for embedding text, image, audio, or video data into physical objects through RFID or computer vision processes [4, 6, 7, 11, 14, 16, 18]. A few allow for objects to hold multiple narrative snippets, allowing for non-linearity, but are limited in the kinds of objects the systems use [14, 15]. Only one has seen widespread implementation using QR codes and websites [11]. The system I plan to develop focuses on audio output, and will allow for any type of object input and an indefinite number of stories. Users can generate their own new narratives with the system as well by attaching RFID tags to their own objects and record their own audio stories.

My portfolio, available at http://matthewmosher.org, speaks to my professional experience and ability to complete physical computing art projects of this nature. This proposal builds on my previous work, If These Walls Could Speak, which allowed people to record audio memories of their travels into RFID tagged rocks from places they visited. If These Walls Could Speak will be presented at the upcoming Audio Mostly Conference in London this August. A recent residency at Sculpture Space gave me the confidence that I have the skill and focus to complete this research project in the given time frame.

This project is significant to Austria, the field of interaction design, and myself. It is designed to facilitate understanding between groups of people who have differing opinions by telling multi-perspective stories. In today’s global political climate, understanding and valuing the views of others is of the utmost importance. This project also builds on existing research in the field of tangible interaction while contributing a precise application and new implementation. As an artist, this opportunity will allow me to develop substantial new creative work.

Austria has accepted scores of refugees since the crisis in Syria, and has recently stopped accepting more. This gives me solid context for the project and makes them an excellent host country. I plan to record the initial stories for the system in Austria, and expect the local culture to have a direct impact on the form of these narratives. I am particularly interested in topics where there is a lack of understanding, where communication can be difficult, and where there are many competing perspectives. I would like to work with a local immigrant and/or refugee population so that they can tell their stories and help to build a dialogue with the citizens of their new nation. I do not expect all the people who I will need to talk to for this project to speak English, or even German, so I plan to work with immigration centers that can facilitate communications. At Salzburg, the MultiMedia Art department’s recent graduate student project ‘Personage’ investigates using social media as a platform for writing new fiction, critique, and multi-viewpoint narrative. This research has a direct conceptual link with my proposed non-linear narrative project, which contributes to my excitement about this opportunity at Salzburg University.

And here’s the bibliography for that:

  1. Scott Brave and Andrew Dahley. 1997. inTouch: a medium for haptic interpersonal communication. In CHI ’97 Extended Abstracts on Human Factors in Computing Systems(CHI EA ’97). ACM, New York, NY, USA, 363-364. DOI: https://doi-org.ezproxy.net.ucf.edu/10.1145/1120212.1120435
  2. Angela Chang, Sile O’Modhrain, Rob Jacob, Eric Gunther, and Hiroshi Ishii. 2002. ComTouch: design of a vibrotactile communication device. In Proceedings of the 4th conference on Designing interactive systems: processes, practices, methods, and techniques(DIS ’02). ACM, New York, NY, USA, 312-320. DOI=http://dx.doi.org.ezproxy.net.ucf.edu/10.1145/778712.778755
  3. Angela Chang, Ben Resner, Brad Koerner, Xing Chen Wang, and Hiroshi Ishii. 2001. LumiTouch: an emotional communication device. In CHI ’01 Extended Abstracts on Human Factors in Computing Systems(CHI EA ’01). ACM, New York, NY, USA, 313-314. DOI=http://dx.doi.org.ezproxy.net.ucf.edu/10.1145/634067.634252
  4. Jean Ho Chu. 2015. Designing Tangible Interfaces to Support Expression and Sensemaking in Interactive Narratives. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction(TEI ’15). ACM, New York, NY, USA, 457-460. DOI: https://doi-org.ezproxy.net.ucf.edu/10.1145/2677199.2693161
  5. Tom Djajadiningrat, Ben Matthews, and Marcelle Stienstra. 2007. Easy doesn’t do it: skill and expression in tangible aesthetics. Personal Ubiquitous Comput.11, 8 (December 2007), 657-676. DOI=http://dx.doi.org.ezproxy.net.ucf.edu/10.1007/s00779-006-0137-9
  6. Augusto Esteves and Ian Oakley. 2010. Mementos: a tangible interface supporting travel. In Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries(NordiCHI ’10). ACM, New York, NY, USA, 643-646. DOI=http://dx.doi.org.ezproxy.net.ucf.edu/10.1145/1868914.1868994
  7. David Frohlich and Rachel Murphy. 2000. The Memory Box. Personal Ubiquitous Comput.4, 4 (January 2000), 238-240. DOI=http://dx.doi.org.ezproxy.net.ucf.edu/10.1007/PL00000011
  8. Elizabeth Goodman. 2003. The sensing beds. In Proceedings of the ACM conference on Ubiquitous Computing (UbiComp ‘3). ACM, New York, NY, USA.
  9. Caroline Hummels, Kees C. Overbeeke, and Sietske Klooster. 2007. Move to get moved: a search for methods, tools and knowledge to design for expressive and rich movement-based interaction. Personal Ubiquitous Comput.11, 8 (December 2007), 677-690. DOI=http://dx.doi.org.ezproxy.net.ucf.edu/10.1007/s00779-006-0135-y
  10. Hiroshi Ishii and Brygg Ullmer. 1997. Tangible bits: towards seamless interfaces between people, bits and atoms. In Proceedings of the ACM SIGCHI Conference on Human factors in computing systems(CHI ’97). ACM, New York, NY, USA, 234-241. DOI=http://dx.doi.org.ezproxy.net.ucf.edu/10.1145/258549.258715
  11. Martin L. de Jode, Ralph Barthel, and Andrew Hudson-Smith. 2011. Tales of things: the story so far. In Proceedings of the 2011 international workshop on Networking and object memories for the internet of things(NoME-IoT ’11). ACM, New York, NY, USA, 19-20. DOI=http://dx.doi.org.ezproxy.net.ucf.edu/10.1145/2029932.2029940
  12. Hanchuan Li, Eric Brockmeyer, Elizabeth J. Carter, Josh Fromm, Scott E. Hudson, Shwetak N. Patel, and Alanson Sample. 2016. PaperID: A Technique for Drawing Functional Battery-Free Wireless Interfaces on Paper. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems(CHI ’16). ACM, New York, NY, USA, 5885-5896. DOI: https://doi-org.ezproxy.net.ucf.edu/10.1145/2858036.2858249
  13. Rong-Hao Liang, Han-Chih Kuo, and Bing-Yu Chen. 2016. GaussRFID: Reinventing Physical Toys Using Magnetic RFID Development Kits. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems(CHI ’16). ACM, New York, NY, USA, 4233-4237. DOI: https://doi-org.ezproxy.net.ucf.edu/10.1145/2858036.2858527
  14. Ali Mazalek and Glorianna Davenport. 2002. Tangible viewpoints: a physical interface for exploring character-driven narratives. In ACM SIGGRAPH 2002 conference abstracts and applications(SIGGRAPH ’02). ACM, New York, NY, USA, 256-256. DOI=http://dx.doi.org.ezproxy.net.ucf.edu/10.1145/1242073.1242269
  15. Daniela Petrelli, Nicolas Villar, Vaiva Kalnikaite, Lina Dib, and Steve Whittaker. 2010. FM radio: family interplay with sonic mementos. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI ’10). ACM, New York, NY, USA, 2371-2380. DOI=http://dx.doi.org.ezproxy.net.ucf.edu/10.1145/1753326.1753683
  16. Molly Stevens, Florian Vollmer, and Gregory D. Abowd. 2002. The living memory box: function, form and user centered design. In CHI ’02 Extended Abstracts on Human Factors in Computing Systems(CHI EA ’02). ACM, New York, NY, USA, 668-669. DOI=http://dx.doi.org.ezproxy.net.ucf.edu/10.1145/506443.506537
  17. Rob Strong and Bill Gaver. 1996. Feather, scent, and shaker: supporting simple intimacy. In Proceedings of the ACM conference on Computer Supported Cooperative Work (CSCW ’96). ACM, New York, NY, USA.
  18. Alex S. Taylor et all. 2006. Building bowls for miscellaneous media. In position papers of Physicallity. Lancaster University, UK, 7-10.
  19. Brygg Ullmer, Hiroshi Ishii, and Robert J. K. Jacob. 2005. Token+constraint systems for tangible interaction with digital information. ACM Trans. Comput.-Hum. Interact.12, 1 (March 2005), 81-118. DOI=http://dx.doi.org.ezproxy.net.ucf.edu/10.1145/1057237.1057242

Please note that the views expressed here are my own.  I do not officially represent the U.S. Government or Fulbright Program.