Talking with Neurons
Institute of Industrial Science, The University of Tokyo in collaboration with National University of Singapore
Exhibited at:
SIGGRAPH 2023 Art Gallery
21_21 Design Sight 2024
Prasanth Kumaar
Yuri Klebanov
Siu Yu Angela Chow
Tomoya Duenki
Atsuhiro Nabeta
Yoshiho Ikeuchi
Miles Pennington
Hyunjung Kim
Yu Uchikura
Naoto Takayama
Ryota Murai
Interaction Design
Motion Design
Art Direction
Graphic Design
Communication
Brief
How can we demystify and publicise Ikeuchi Lab’s research on in vitro neural tissues?
The Ikeuchi Lab is developing methods to build neural tissues outside of our body. Our brain and the nervous system is complex and many different parts coordinate to execute higher-order functions. The researchers aim to comprehend how the network of sections of the brain form and operate, and to contribute to find cure for diseases of the brain. To achieve these, they utilize microfabricated culture vessels and multielectrode arrays. This approach aids them in studying in greater detail how the many parts of the brain form and organize themselves. Read more about the Ikeuchi Lab here.
Video script, concept, graphics & animation, voiceover, and editing by Alexandra Arguelles.
Filmed with the help of Prasanth Kumaar.
Special thanks to Angela Siu Yu Chow, Ryota Murai, and Huaruo Hu.
Process
Creating a speculative, yet current, design artefact
Speculative design is an effective way of elucidating cutting edge-research which may seem too detached and unfamiliar to lay people. Through a design artefact, speculative design contextualises seemingly abstract research, bridging the gap between scientists and the general population and allowing for greater discussions to be made within a larger community.
The Ikeuchi Lab has previously partnered with the DLX Lab to create speculative design artefacts—creating fictional consumer products which prompt discussion on the future of biocomputing (See Bio-Intelligent Machine — Aura). While provocative, it is still fictional.
In this project, we set out to create a design artefact that would close the gap between the research and the general population—by letting audiences directly interact with the in-vitro neural tissue.
Alignment workshop using sacrificial concepts
An alignment workshop to gain insight on Ikeuchi Lab’s interests, worries, and available technologies, which would then help us iterate on selected concepts. In particular, we asked the lab researchers to position our ideas on axes according to what they liked/disliked, and what technology was available now and available far in the future.
Prototyping and concept development
Creating proof-of-concept prototypes (experiential and visual prototypes) to gain insight on feasibility and desirability of the concept.
Concept explorations included creating games from neural activity, turning neural activity in sensory experiences, and ‘decoding’ neural activity into messages.
First iteration @ UTokyo
We settled on a direction of decoding neural activity into an interactive audiovisual experience, in which audiences would be able to stimulate neurons with sound and receive an audiovisual response.
The first iteration was a proof-of-concept, experimenting and validating the possibility of remote real-time interaction with neural tissue. At this stage, we received data from only one data channel, and visuals were randomised.
Final iteration @ SIGGRAPH 2023
With the intention of exhibiting at SIGGRAPH 2023, Talking with Neurons was refined to be of higher fidelity, with more intentional sounds and visuals which would better communicate the real-time neural activity.
In pursuit of achieving an impactful experience, the Ikeuchi Lab was able to develop a method to read neural activity from 32 data channels, with the ability to distinguish bursts and spikes. The interactive installation was redesigned for a better user experience and to more deliberately and compellingly convey the neural activity.
For a more in-depth look into our explorations, visit Talking with Neurons — Inspire Talks #23 where we share our design process and demo our first iterative prototype of the Talking with Neurons interactive exhibit at the University of Tokyo.
Outcome
Interacting with in-vitro neuronal tissue through audio-visual representation of neuronal activity
"Talking with Neurons" is an interactive installation that enables people to remotely connect with live neural tissues that mimic our nervous system. Neurons are the essential units of our nervous system, including the brain, responsible for information processing and transmission. "Talking with Neurons" invites audiences to conduct a reciprocal conversation through sending voice inputs and receiving audiovisual representations of neuronal activity by in vitro neurons that respond to their vocal cues. By exploring a new method of human-neuron interaction, this installation enables us to imagine a future where we integrate living neurons into the technology we use every day.
Live interaction with neuronal tissue
The configuration of "Talking with Neurons" consists of an installation virtually connected between an on-site exhibition and a remote lab in Tokyo. The on-site location is equipped with a projector, screen, and headset to allow the audience to digitally interact with the neurons. In the virtual setup, in vitro neural circuits are connected to a microelectrode array (MEA) that stimulates the organoids based on the remote input – the audience's voice – and records neuronal activity patterns in response to the stimulation. The captured signals are then transmitted to the on-site computer in real-time and instantly converted into auditory and visual data.
Audio-visual representation of neuronal activity
The aural interaction between the user and in vitro neurons is based on intonation, as changes in pitch and rhythm allow oneself to express meaning beyond words. Key activity markers of the neuronal patterns, such as spikes (neuronal action potentials) and bursts (a group of action potentials generated in rapid succession), are encoded into simple sounds and graphics. Thus, users can experience the inception of basic communication with lab-grown organoids.
Neuronal spikes
Neuronal action potentials
Visually encoded as pulsing stars, and aurally encoded as muted, yet, staccato-like chimes.
Burst events
A group of action potentials generated in rapid succession
Visually encoded as bright flares which grow larger and brighter as the signals grow stronger, and aurally encoded as melodic hums.
Audiences are given the opportunity to witness a stunning visual rendition of neuronal harmony, demonstrating the spatial and temporal intricacies of the neuronal activities exhibited by in vitro neural circuits. The attached neural organoids, depicted in an abstract manner as two large circles bridged with a thin line surrounded by a dark background, contrast with bright flares that light up during burst events and glimmers that pulsate in time with neuronal spikes. The abstracted visuals of the connected organoid's activity are juxtaposed with live footage of the experiment conducted at the remote lab. The installation re-imagines the technical, data-based experiments as an expressive and engaging experience by relaying complex knowledge to the audience.