Challenges in the Intermodal Translation of Art

David Rokeby, Very Nervous Systems Ltd.

This project is an inquiry into the challenges and possibilities of translating cultural works to other sensory modalities in order to make them accessible to those who are unable to experience them in their native mode. The first section is a general examination of the problem of intermodal translation (indeed for any sort of translation) for the full range of artistic expression. The second part focusses on issues specific to the translation of music into some sort of visual representation.


My first major work, Very Nervous System (1983-2004) translated visual information in the form of live video input into sound. Though the work was experienced by the public as an interactive sound installation with no visual component, a large part of the system was a vision system. The translation from moving image to sound or music was highly subjective, but it made me think a lot about the relationships between image and sound, and between seeing and hearing. More recently, I have been producing works that are predominantly or entirely visual in their output, but which often reflect an aesthetic as grounded in sound and hearing as in image and seeing. This may be because of my background in sound and music, or it may be a result of the fact I am very interested in time, the experience of time, the experience of changes across time, and the temporal behaviour of systems. Until the advent of the cinema, visual aesthetics seems to have largely ignored time, whereas musical aesthetics are strongly tied to temporal qualities like pitch, timbre, melody, rhythm, tempo, and development. I have also been very interested in phenomenology, in particular, the ways that the characteristics of our sense and perception systems shape our understanding of the world and the ways in which we find meaning in our experience.

Most of my works have been translation systems in some way or other. Translation seems to me to be a process of refraction. Just as light undergoes a transformation as it leaves one physical medium and enters another, anything undergoing translation (whether from language to language, from presentation medium to presentation medium, or from sensory mode to sensory mode) is changed in a way that reflects the differences between the two substrates (or contexts?). In my work, this refractory transformation implicit in translation has been my medium of expression. By creating translation systems, I create a kind of stereoscopy, in which the source and the translated result are put into dialog. Just as the eyes and ears use differences between two perspectives on a single scene to establish depth, the source and its translation together potentially enhance the understanding of the source.

In the context of this project as part of Culturall 2.0, the focus shifts from the generative character of this translation-generated stereoscopy to the more acute problem of the translation of cultural experience for people who are unable to fully experience the work in its native sensory mode. For people with limited but not insignificant capacity in the native mode, the stereoscopic effect is still present and can most likely significantly enhance the experience of the original work, but for those with profound restrictions in the original mode, the translation becomes the one and only source. This raises a variety of difficult issues regarding the limits of translation between modes and the interpretative role of the translator. It also draws us into deeper questions about the nature of creation, the artist's intent and the various dimensions of the public experience of a work of art and its cultural reception.