Talking with Neurons (2023)
Demystifying research on in-vitro neural tissues through an interactive audio-visual representation of neuronal activity
I led the concept and experience design of an installation bridging a live neuroscience lab with a public audience.
Role
Experience Designer
Design Research
Concept Direction
Prototyping and Iteration
Project Communication (Storyboarding, Writing, Animation, Filming)
Team
1 Experience Designer, with the help of Ikeuchi Lab
Duration
Jun–Aug 2022, Nov 2022–Aug 2023 (Subcontractor)
Exhibited at
SIGGRAPH 2023 Art Gallery
Design Sight 21_21 (2024)
Context
How can we demystify and publicise Ikeuchi Lab’s research on in vitro neural tissues?
The Ikeuchi Lab is developing methods to build neural tissues outside of our body. Our brain and the nervous system is complex and many different parts coordinate to execute higher-order functions. The researchers aim to comprehend how the network of sections of the brain form and operate, and to contribute to find cure for diseases of the brain. To achieve these, they utilize microfabricated culture vessels and multielectrode arrays. This approach aids them in studying in greater detail how the many parts of the brain form and organize themselves. Read more about the Ikeuchi Lab here.
Research & Direction
Creating a speculative, yet current, design artefact
Testing the waters with Ikeuchi Lab
We conducted an alignment workshop to gain insight on Ikeuchi Lab’s interests, worries, and available technologies, which would then help us iterate on selected concepts. In particular, we asked the lab researchers to position our ideas on axes according to what they liked/disliked, and what technology was available now and available far in the future.
Exploring and testing different outcomes
We created proof-of-concept prototypes (experiential and visual prototypes) to gain insight on feasibility and desirability of the concept. Concept explorations included creating games from neural activity, turning neural activity in sensory experiences, and ‘decoding’ neural activity into messages.
Outcome
Interacting with in-vitro neuronal tissue through audio-visual representation of neuronal activity
"Talking with Neurons" is an interactive installation that enables people to remotely connect with live neural tissues that mimic our nervous system. Neurons are the essential units of our nervous system, including the brain, responsible for information processing and transmission. "Talking with Neurons" invites audiences to conduct a reciprocal conversation through sending voice inputs and receiving audiovisual representations of neuronal activity by in vitro neurons that respond to their vocal cues. By exploring a new method of human-neuron interaction, this installation enables us to imagine a future where we integrate living neurons into the technology we use every day.
Live Interaction with Neuronal Tissue
The configuration of "Talking with Neurons" consists of an installation virtually connected between an on-site exhibition and a remote lab in Tokyo. The on-site location is equipped with a projector, screen, and headset to allow the audience to digitally interact with the neurons. In the virtual setup, in vitro neural circuits are connected to a microelectrode array (MEA) that stimulates the organoids based on the remote input – the audience's voice – and records neuronal activity patterns in response to the stimulation. The captured signals are then transmitted to the on-site computer in real-time and instantly converted into auditory and visual data.
Audiovisual Representation of Neuronal Activity
The aural interaction between the user and in vitro neurons is based on intonation, as changes in pitch and rhythm allow oneself to express meaning beyond words. Key activity markers of the neuronal patterns, such as spikes (neuronal action potentials) and bursts (a group of action potentials generated in rapid succession), are encoded into simple sounds and graphics. Thus, users can experience the inception of basic communication with lab-grown organoids.
Neuronal Spikes
NEURONAL ACTION POTENTIALS
Visually encoded as pulsing stars, and aurally encoded as muted, yet, staccato-like chimes.
Wellness Reminders
A GROUP OF ACTION POTENTIALS GENERATED IN RAPID SUCCESSION
Visually encoded as bright flares which grow larger and brighter as the signals grow stronger, and aurally encoded as melodic hums.
Reflections 🤔
Portability is an outreach strategy
The impact of this project was greatest when experienced first-hand, so the installation had to be travel-ready. By relying only on walls, a computer, a microphone, and projectors, we kept the setup light enough to go anywhere.
The real and the messy enhances the magic
Contrary to our instinct to only show a polished abstraction, exposing the raw lab setup—the wires, the organoids, the incomprehensible grid of green graphs—created a contrast that made the audiovisual translation feel even more compelling.
Our vision pushed the lab forward
The first iteration was a proof-of-concept with a single data channel and randomised visuals. In pursuit of a more impactful experience for SIGGRAPH 2023, the Ikeuchi Lab developed the ability to read from 32 data channels and distinguish bursts from spikes—advancing their own real-time analysis capabilities in the process.