Rainbows in my ears - Synesthetic color perception with
partial-reduced and morphed musical instrument timbres
Christoph Reuter1, Jörg Jewanski1, Sarah Ambros1, Isabella Czedik-Eysenberg1, Saleh Siddiq1, Solange Glasser2, Jamie Ward3

1 SInES (Space for Interdisciplinary Experiments on Sound), University of Vienna, Vienna, Austria
2 Music Psychology Department, Melbourne Conservatorium of Music, Melbourne, Australia
3 School of Psychology, University of Sussex, Brighton, United Kingdom

Stimuli


Mean ΔE Differences between synesthetes and non-synesthetes

 

Instrument-dependent color perception with partial-reduced sounds:

Flute Flute with reduced partials Flute with reduced partials
French Horn French Horn with reduced partials French Horn with reduced partials
Oboe Oboe with reduced partials Oboe with reduced partials
Piano Piano with reduced partials Paino with reduced partials
Violin Violin with reduced partials Violin with reduced partials

 

Perceived color values with partial-reduced sounds:

Perceived color saturation in partial-reduced sounds
Saturation
Perceived color lightness in partial-reduced sounds
Lightness
Perceived rgb values in partial-reduced sounds
Red - Green - Blue values

 

 

Partial-reduced sounds and their audio features


Sound examples and their audio features

 

Partial-reduced sounds and timbre-induced color lightness perception

Perceived color saturation in partial-reduced sounds
Spectral Centroid and
Timbre-induced Lightness Perception
(Synesthetes)

Perceived color lightness and harmonicity strength (Synesthetes)
Harmonicity Strength and
Timbre-induced Lightness Perception
(Synesthetes)
Perceived color lightness and percussive loudness (Synesthetes)
Percussive Loudness and
Timbre-induced Lightness Perception
(Synesthetes)

Perceived color lightness in partial-reduced sounds
Spectral Centroid and
Timbre-induced Lightness Perception
(non-Synesthetes)

Perceived color lightness and harmonicity strength (non-Synesthetes)
Harmonicity Strength and
Timbre-induced Lightness Perception
(non-Synesthetes)
Perceived color lightness and percussive loudness (non-Synesthetes)
Percussive Loudness and
Timbre-induced Lightness Perception
(non-Synesthetes)

 

Partial-reduced sounds and timbre-induced color blue value perception

Perceived color blue value in partial-reduced sounds
Spectral Centroid and
Timbre-induced Blue Value Perception
(Synesthetes)

Perceived color lightness and harmonicity strength (non-Synesthetes)
Harmonicity Strength and
Timbre-induced Blue Value Perception
(Synesthetes)
Perceived color blue value and percussive loudness (Synesthetes)
Percussive Loudness and
Timbre-induced Blue Value Perception
(Synesthetes)

Perceived color blue value in partial-reduced sounds
Spectral Centroid and
Timbre-induced Blue Value Perception
(non-Synesthetes)

Perceived color blue value and harmonicity strength (non-Synesthetes)
Harmonicity Strength and
Timbre-induced Blue Value Perception
(non-Synesthetes)
Perceived color blue value and percussive loudness (non-Synesthetes)
Percussive Loudness and
Timbre-induced Blue Value Perception
(non-Synesthetes)

Tables 3 and 4 with Bonferroni-corrected p values

 

 

Instrument-dependent color perception with morphed sounds:

Violin French Horn Piano Oboe
Flute Flute morphed with violin in different ratios Flute morphed with French horn in different ratios Flute morphed with piano in different ratios Flute morphed with oboe in different ratios
Violin Violin morphed with French horn in different ratios Violin morphed with piano in different ratios Violin morphed with oboe in different ratios
French Horn French horn morphed with piano in different ratios French horn morphed with oboe in different ratios
Piano Piano morphed with oboe in different ratios

 

Perceived ΔE color differences with morphed sounds:

Violin French Horn Piano Oboe
Flute Perceived color differences: Flute morphed with violin in different ratios Perceived color differences: Flute morphed with French horn in different ratios Perceived color differences: Flute morphed with piano in different ratios Perceived color differences: Flute morphed with oboe in different ratios
Violin Perceived color differences: Violin morphed with French horn in different ratios Perceived color differences: Violin morphed with piano in different ratios Perceived color differences: Violin morphed with oboe in different ratios
French Horn Perceived color differences: French horn morphed with piano in different ratios Perceived color differences: French horn morphed with oboe in different ratios
Piano Perceived color differences: Piano morphed with oboe in different ratios

All ΔE colour difference plots side by side for comparison

Morphed sounds and their audio features


Sound examples and their audio features

 

Morphed sounds and timbre-induced color lightness and blue value perception

Perceived color lightness and blue values depending on spectral centroid (Synesthetes)
Spectral Centroid and
Timbre-induced Lightness
and Blue Value Perception
(Synesthetes)

Perceived color lightness and blue values depending on harmonicity strength (Synesthetes)
Harmonicity Strength and
Timbre-induced Lightness
and Blue Value Perception
(Synesthetes)
Perceived color lightness and blue values depending on percussive loudness (Synesthetes)
Percussive Loudness and
Timbre-induced Lightness
and Blue Value Perception
(Synesthetes)

Perceived color lightness and blue values depending on spectral centroid (non-Synesthetes)
Spectral Centroid and
Timbre-induced Lightness
and Blue Value Perception
(non-Synesthetes)
Perceived color lightness and blue values depending on harmonicity strength (non-Synesthetes)
Harmonicity Strength and
Timbre-induced Lightness
and Blue Value Perception
(non-Synesthetes)
Perceived color lightness and blue values depending on percussive loudness (non-Synesthetes)
Percussive Loudness and
Timbre-induced Lightness
and Blue Value Perception
(non-Synesthetes)

 

Timbre-Color Synesthesia Space
The perceived colors of all participants for all conditions in a RGB and HSL space


RGB and HSL Timbre-Color Synesthesia Space