Synaesthesia was my Final Major Project at Hertfordshire University.

The concept for the project was to create an augmented reality application which reads colour values picked up by a phone’s camera and translates them into sound in most cases.

The bulk of the app is built in Unity with the help of Rhys St Romaine whilst the audio was programmed using Wwise.

Synaesthesia features five libraries which show the applications capabilities:

A library titled The Basic which changes reverb wetness level depending on the environment’s colour.
A library which utilises five different points on the screen to detect colours. Each one of the five points is dedicated to a sound and a loop corresponding to a colour allowing thousands of combinations.
A storybook (with physical book) which narrates an original story written by myself and Kathryn Kitchener, who also narrates. The colours branch the story in different directions.
The 808 library is similar to the second library. There are three colour detectors dedicated to an instrument and two more dedicated to effects. The colours change the patterns of the drum tracks.
The final library (not shown in the video) is a collaboration with Maxime Gilbert. There are three colour detectors dedicated to an instrument which loop. The colour picked up changes the pattern of the instrument.
The video will bring more understanding to the app.

Synaesthesia was showcased at the Hertfordshire University End of Year Arts Show.
Paul-Jo Nielsen
Co-Founder of Bay6 Games and Nüwa Sounds - Other