Generative Data Sonification

---

The Concept

I believe in an optimistic future. A place where humans find a way to bring themselves in union with nature and technology, and all three can provide for the other.

When asked to be a part of the collective showing for data sonification pieces with Sound Obsessed, a choice was made to take a different approach from using a static dataset about one of the UN’s development goals. Instead, an approach with a focus on a vision for the future bringing sounds back to the present through its data was decided, and realized with help of musician and sound designer Tom Guida.

While telling the stories of various cultures in the near future and their discoveries with the connection humans, technology, and nature, we let the images create the audio in a generative way, through their data.

The Process

The process involved using ComfyUI to create a large series of explorations into solarpunk innovations, artifacts or rituals which exemplify the union of nature, technology, and humanity. Once these were rendered and upscaled, these were brought into TouchDesigner, a node-based programming environment.

In TouchDesigner, a 16 track sequencer cycles horizontally across the videos as they are played, sampling the pixels as they move through from the left edge to the right, looping back around.

In the pixel sampling an RGB average is taken, and converted into a MIDI signal sent via ethernet to a computer running Ableton.

In Ableton, the MIDI signals play back the instruments constructed by Tom, and compose the music generatively as the pixels are sampled in their different timings.

The result is a hopeful, yet speculative soundscape to accompany the visual journey, as poetry is incorporated into the video throughout telling the story.