The Klimt Color Enigma

Gustav Klimt’s three masterpieces: Medicine, Jurisprudence, and Philosophy were destroyed during the Second World War. Only black and white photos and articles describing the paintings remain. Google Arts & Culture in partnership with Belvedere have collaborated to restore Klimt’s Faculty paintings.

In 1894, Gustav Klimt was commissioned by the art committee at the Ministry of Education to decorate the ceiling of the University of Vienna. The mission was to capture the essence of three faculties: medicine, law, and philosophy.

The paintings created a clash. The ministry of education expected the paintings to reflect renaissance thinking, but Klimt changed his original sketches and made them in a secessionist style.

Klimt’s interpretation critiqued both the state and Austrian narrow-mindedness. They were supposed to be realistic paintings capturing the truth, instead, Klimt used bold analogies to convey how people experienced the faculty subjects.

Many thought the paintings were pornographic, superstitious, and ugly. The professors at the university revolted against the paintings and Klimt ended up buying back the artworks. He committed to never work for the state again.

To save Klimt’s paintings during the Second World War, many of Klimt’s artworks were stored in the Immendorf Castle, Lower Austria. The day before the war ended, SS officers burnt the castle. The Nazis refused to have their art confiscated by the Russians.

As the Faculty Paintings vanished, they left an enigma. Through several decades, the paintings left a series of scattered clues. They left a trail of photos and descriptions of what they once looked like, but their original color remains a mystery.

“Three avenging goddesses of terrifyingly beautiful form, with golden snakes in their hair, surround him menacingly. " is how Ludwig Hevesi described Jurisprudence at Klimt’s exhibition in 1903. Six years later, Hevesi describes the Jurisprudence again: “... a luxury hell, where golden instruments of torture are encrusted with diamonds and martyrs bleed rubies.”

These descriptions can be found in news articles, exhibition catalogs, and in letters. Dr. Franz Smola, a curator at Belvedere, has sourced all the comments mentioning the Faculty Paintings. Dr. Smola then went on to match the scenes and color comments with Klimt’s remaining paintings.

For example, the golden snakes that appear in the three women at the forefront in Jurisprudence also appear in Beethoven Frieze. Since it’s the same motif and they were made during the same time, it’s likely that Klimt used a similar nuance of gold.

The main color information Dr. Franz Smola sourced, as well as the closest matching motif in Klimt’s remaining work, can be seen in the following graphics.

Once the color information was sourced, Emil Wallner, a resident at the Google Arts & Culture Lab, developed an algorithm to use Dr. Smola’s research to restore the Faculty Paintings. Instead of manually coloring the paintings, Wallner’s algorithm does a statistical analysis of Klimt’s existing artworks and learns how to mimic Klimt’s colorization style. Based on the data analysis together with Dr. Franz Smola’s research, the algorithm reconstructs the Faculty Paintings.

Machine learning is a subset of artificial intelligence. It’s a scientific study of statistical models that perform specific tasks without using explicit instructions. They rely on patterns which are created through a process of trial and error.

To understand this process, imagine an engineer in the early 19-century. He’s building a machine to print textiles. The machine receives a white textile and is configured to return a printed textile. Inside of the machine are several layers of toothed metal wheels, cogwheels, that work together to print the textile.

At first, the machine is not configured and the printed textile looks terrible. To improve it, the engineer works backward from the final step in the machine, the printed textile. The poorly printed textile is the result of all the accumulated errors in the machine.

He adjusts the errors at the final printing step, looks at the previous layer of metal wheels, and keeps adjusting the combined errors until he reaches the point where the blank textile is inserted. The engineer tries the machine with a new textile, looks at the results, and repeats the same process until the machine returns a good result.

Machine learning models are trained in a similar way. Instead of receiving physical materials and using an engineer to correct the accumulated errors, the algorithm deals with digits and the error-correcting is done with a series of mathematical operations. The algorithm receives training examples and it’s automatically configured to match the input and output pairs it receives.

Whether it being self-driving cars or Google’s voice assistant, all these technologies use the automated trial and error process mentioned earlier. Most of the underlying mathematics was invented in the 1970s, but it took until 2012 to have powerful enough computers to leverage the technology.

Klimt’s paintings can be represented in numbers. A digital camera slices a colored painting into tiny squares, pixels, that we can barely see with the naked eye. Each tiny square is assigned three digits: one value for red, green, and blue. A combination of these three numbers represents all the colors we can see.

The area inside of the red square is enlarged and each pixel is assigned a color value. Machine learning algorithms learn from digits. Whether we work with images or sound, we have to use sensors to translate each medium into numbers.

The textile machine we mentioned before, received blank textiles as input and returned printed textiles. Our coloring algorithm receives a digitized black and white painting and returns the corresponding colored version. While the textile machine was configured manually with a few hundred trials and errors, the colorizing algorithm learns from several million examples. And instead of manual tuning, it’s done automatically using a series of mathematical operations.

We only have black and white photos of the Faculty Paintings, but once the colorizing algorithm is configured, we can insert the monochrome paintings and the algorithm will predict their colored versions. A monochrome photo has one-third of the information compared to its colored counterpart, thus the algorithm uses the black and white photo to predict the missing two-thirds of the information.

At the Google Arts & Culture Lab, we had 80 images of Klimt’s colored artworks to teach the algorithm how to colorize the Faculty Paintings. When we only used these images to teach the Pix2pix algorithm, an algorithm developed by Isola et al. (2016), it learned about Klimt’s color palette, however, it did not have enough understanding of the scenes in the paintings to make a coherent colorization. See the below image to the left.

As a rough indicator, an algorithm needs 5000 images to learn one object, and 80 images are not enough for the algorithm to model Klimt’s coloring style. In the below image in the middle, we tried the DeOldify algorithm, proposed by Jason Antic (2018). The model is trained on one million pictures of things in the real world, including people, animals, and buildings. As you see, the colorization is more coherent and it mimics the real-world, giving people vibrant skin tones, a range of hair hues, and a blue-grey sky. However, this model has no understanding of art nor Klimt’s colorizing style.

The image to the right is made by guiding the colorization with human-made color annotations, a model made by Zang et al. (2017). A user adds a handful of color dots to the black and white painting which informs the algorithm how to colorize it. The machine learning model detects textures and objects and propagates the color hints to similar regions.

We developed a novel model by combining these approaches. The model has a similar structure to DeOldify, a U-net with a pre-trained ResNet-34 with self-attention, spectral normalization, and a 3-channel RGB input with color hints. It’s progressively trained with a custom feature loss from a pre-trained GAN critic.

The algorithm is trained on 91749 artworks from Google Arts & Culture. This allows the machine learning model to learn object boundaries, textures, and frequent compositions in artworks. It makes the colorization coherent and it learns how to adapt to colorization styles from several thousand artists.

As a final step, we trained it on Klimt’s colored paintings. This creates a colorization bias towards color themes from Klimt’s artworks. Although it does not model Klimt’s artistic style in full, it has a prejudice for moods, colors, and recurring motifs in Klimt’s paintings.

To make the paintings historically accurate, we guide the algorithm with Dr. Franz Smola’s research. If we know that a certain object has a specific color, we add that color directly to the black and white photos.

When we train the machine learning model, the algorithm receives the black and white photos with random patches that are already colored. Since the colored patches are always correct, it informs the algorithm that the surrounding areas often have a similar color, as well as regions with a similar shape and structure.

To capture the main impression of the Faculty Paintings and to add the correct color details, Romain Cazier at the Google Arts & Culture Lab, developed an interface to interact with the machine learning algorithm.

Below is the interface we used to colorize the Faculty Paintings. With the mouse pointer to the right we selected reference colors according to Dr. Smola’s research, and then we applied it to the black and white artworks to the left. It takes a few seconds for the algorithm to take the monochrome paintings with the color references to restore the artworks.

We made several iterations: it had to match the ten pages worth of historical commentary about the paintings, in addition to Dr. Smola’s expert judgment as a result of studying Klimt’s artworks over several decades.

It’s been an exciting journey to unravel the enigma left by the Faculty Paintings. In 1905, when Klimt showed the paintings together for the first time, many visitors were in shock. We too were surprised several times as we applied the clues we found. As our journey comes to an end, we hope it creates a new beginning for three of the world’s most controversial and important paintings during the Vienna Secession.