Museum Close-Looking with AI
Exploring how GenAI could help facilitate, personalize and deepen visitors’ interactions with artworks.
"I want to learn more"
Our visitors constantly tell us they want more ways to learn about artworks.
We also know that when visitors spend more time with an artwork, they form a deeper connection to it, and as a result, have a more rewarding museum experience.
The Met Museum ShopThe Metropolitan Museum of Art
The Met is certainly not short on educational resources: we have expert knowledge on almost every artwork.
The Challenge
The challenge often lies in format: Many of our resources are not meant for in-gallery use. Others are limited to English, or written in an academic tone that isn’t always kid-friendly.
Our Global Audience
Our audience is global, with diverse languages and learning needs. Without a way to tailor our content to individual visitors, we limit who can actually enjoy our resources.
Can AI Help?
AI is good at managing content at a big scale, and personalizing information. This made us curious:
Could GenAI help tailor our learning resources to meet visitors exactly where they are?
In collaboration with Google Arts & Culture, The Met is exploring how generative AI might reshape the museum experience. One series of experiments focused on how AI could help visitors deepen their engagement with individual artworks.
AI is a broad tool with broad applications. To find out what visitors actually want (and don’t) we relied on rapid prototyping, teamwork across departments, and user testing with a wide range of visitor profiles.
Prototype Overview
In this article we will look at the following three prototypes:
1. The Wall Label Prototype
2. The Met Resources Prototype
3. The Circle Details Prototype
The Wall Label Prototype
We started by exploring wall labels, the most common way visitors learn about artworks.
We used Gemini to translate wall labels into different languages and tones.
As the video below shows, visitors simply take a picture of an artwork. AI automatically identifies the artwork using a new API developed by our Digital Product team. Then visitors can choose how they want read the wall label: With added definitions, as bullet points, or in a different language for example. Each variant is generated by Gemini in real-time, based on the original wall label.
We invited multilingual groups to test the experience. Their feedback was unanimously positive, showing that AI can make high-quality translations at scale.
In collaboration with the Asian Art department, we set up a QR code on a busy day.
The prototype had almost 300 scans in a day, showing that visitors are genuinely looking for this functionality.
Although excited, curators raised valid concerns around tone accuracy: AI sometimes adds decorative phrasing that can change the tone of the wall label. We still need to explore if fine-tuning the prompt can help improve accuracy to avoid the need for manual editing.
The Met Resources Prototype
Moving beyond the wall label, our second prototype connects artworks directly to in-depth resources The Met already provides: programming, online content, books, and more.
Similar to the wall label prototype, visitors simply take a picture of an artwork to identify it with our new API. AI then pulls five related resources from our archives and website. Each resource links directly to The Met's website.
This prototype was particularly exciting for knowledgeable visitors that want to dive deeper.
It also shows that AI can be used as a tool for discovery rather than just generation of content.
Yet, some of The Met's resources like essays and publications, can be too dense to read in-gallery.
The Circling Details Prototype
We wanted to take the previous prototype one step further by giving visitors an even easier way to engage with The Met's resources.
The Circle Detail PrototypeThe Metropolitan Museum of Art
We tested a tool that allows visitors to circle a specific detail of an artwork to get digestible insights using a process called Retrieval Augmented Generation (RAG), which helps the AI pull content directly from specific sources.
Especially first-time visitors were excited that this prototype provided them with interesting jumping-off points, while still sourcing all insights directly from The Met's content.
Comparing all areas visitors circled gives us a clear picture of what our guests are most curious about.
This data could help us improve in-gallery content like wall labels. For example we could target its content more to specific visitor interests.
Main Learnings
AI can't replace or imitate expert voices in museums. Our prototypes showed that AI's strength lies in customizing existing Met resources to meet visitors exactly where they are: their interests, preferred languages, and even their level of familiarity with art history.
Bridging the gap between our world-class scholarship and our visitors is an exciting opportunity for AI.
You are all set!
Your first Culture Weekly will arrive this week.