One Sound Two Frames is a playful interactive game to inspire discovery of artworks in the Google Arts & Culture collection - through music. To pass a level, you have to guess which artwork inspired the music you are hearing. This experiment uses an image-to-music model which analyzes an artwork, creates a description of it, translates it into a music description text using Gemini, and passes it on to Lyria for music generation. As you progress through levels, the game becomes more challenging as the artworks become more similar. 

This experiment was a collaboration between Emmanuel Durgoni, artist in residence at the Google Arts & Culture Lab, and Timo Denk, Research Engineer with Google DeepMind. Emmanuel was interested in exploring new ways of connecting and discovering artworks that go beyond traditional search. He was also curious about what sound could accompany a painting. In collaboration with Timo, he worked with the image-to-music model to create a playful interface that challenges the viewer to look at artworks in greater detail to make sure they match it to the music they are hearing. Emmanuel's highest score was 27, can you beat that?