‘Hear’ this Kandinsky painting with Google’s psychedelic AI tool

Google has launched a new AI tool that lets you listen to the sounds that Vassily Kandinsky may have heard as he painted.

The experiment explores the artist’s synesthesia, a neurological condition that causes one sense to trigger another.

In Kandinsky’s case, he’d see colors and shapes when he heard music, and hear music when he painted.

The abstract art pioneer tried to evoke the sensation through his artworks and writings:

The sound of colors is so definite that it would be hard to find anyone who would express bright yellow with bass notes or dark lake with treble.

Google’s collaboration with Paris’ Centre Pompidou imagines the sounds he heard in 1925 as he painted “Yellow, Red, Blue,” a striking display of geometric art.

[Read: How Polestar is using blockchain to increase transparency]

The Google Arts & Culture team describes the lithograph as a small instrumental work:

An extraordinarily gifted artist, he arranged ‘colored sounds’ into ‘color symphonies’… The character of the colors reflected for him the harmonies and dissonances of a melody.

The researchers first trained Google’s Transformer neural network on music from Kandinsky’s time. They then used the system to generate new scores inspired by the sounds.

Sound artist Antoine Bertine, who worked on the project, said the initial idea was to use machine learning as an archeological tool:

By inviting the machine to express itself, the music kind of emerged by itself, and it felt that as artists we were more the intermediary from Kandinsky’s time to today, rather than the interpreters of the theory ourselves.

The experiment resulted in Play a Kandinsky, an interactive tool that lets you explore Kandinsky’s synesthesia and generate your own musical compositions based on his work.

You can hear the sounds he associated with different colors and shapes, or listen to the entire masterpiece as the artist may have heard it.

But the best part lets you become the composer of the painting, by clicking on colored shapes to produce the music that you see.

In my case, the string ensemble of the red triangle caused an acute sense of restlessness, but I attribute much of that to this interminable lockdown. The feeling subsided slowly when I tapped on the peaceful violins behind the multi-colored plane.

It doesn’t replicate the connections between sights and sounds that a synaesthetic feels, but it’s an intriguing exploration of the connections between music and painting.