Jake Barton on how to create meaningful interactions through technology

In his presentation at Design Indaba Conference 2014, Jake Barton starts off by questioning the design profession’s relationship with technology as a medium and the ways in which tools influence what is made. He reflects on the reciprocal relationship between maker and medium when it comes to more typical forms of art such as painting, and how there is a level of intuition that comes out of the creative process itself.
He presents four projects completed by the media design firm he founded, Local Projects. The firm creates installations for museums, brands and public spaces using a broad array of technological platforms.
Barton comes from a theatre background, which gives him insight into performance and the different ways in which people create through intuition. He says he envies the relationship that designers who work with physically concrete materials have with the materiality of what they are creating, as he believes that it stimulates their expressiveness. “It made me understand that maybe there is something missing about working with technology and how we are able to create with technology,” he says.
For examples of how creative expression works, he looks to masters of improvisation such as comedians and jazz musicians. 
Improvisation and seeing 
Barton shares the work Local Projects did for the Cleveland Museum of Art as an example of how improvisation can enable different ways of seeing a collection of art. 
When the Cleveland Museum of Art, one of the top five art museums in U.S., added a new wing, a visionary board member noted that since it is not located in a tourist hub such as Paris or Tokyo, the museum would have to expand its audience in new ways. The institution looked to technology to both build visitorship and reinvent some of its gallery displays. 
Barton’s approach was to pull screens away from the artwork to allow a direct relationship between viewer and artwork and to use technology to amplify this experience rather than take the visitor into another world or space.
“We needed to amplify people’s curiosity about the objects in front of them,” he explains.
Once such way they accomplished this was to locate artworks and artefacts in their original contexts through visualisations – or, as someone described it, “filling in the white space around the artefact in the museum”. 
“This is not about replacing the role of the curator who is still interpreting the works, but rather providing a new way for people to understand these artworks,” says Barton.
Local Projects also developed a facial detection algorithm that matches visitors’ facial gestures to bring forth artworks that contain the exact same expression. Instead of being a passive viewer, the visitor becomes a participant or performer now engaged in an emotional relationship with the artworks.
“The way in which people are making this emotional relationship with these artworks came out of early conversations with curators about the nature of empathy,” explains Barton. “That is literally what portraiture trades on: our human capacity to look into the eyes of someone else and to make that connection, one to the next. Taking ideas like this and making them into interfaces is a lot of what we are doing.”  
Another installation challenges gallery visitors to mimic the pose of sculpted figures. “Curators lamented the fact that when people look at figurative sculpture they often only look at the facial expression when in fact the pose itself is key to figurative expression.”
He explains that they prototype first instead of using the typical Waterfall process of digital design, where one phase cascades into the next in a sequential manner and the designer plans and refines an idea before implementing it (“and crossing your fingers that it works”). The process of prototyping thus becomes a creative tool instead of a testing tool.
“Until you see something working – not just a picture of it, not just an idea, but how this media moves and reacts – you don’t get that moment of ‘wow’. That is what we want and what we aim for over and over again. And so much of our ideas end up on the floor until we get that.”
Barton also believes that there is nothing that will age faster than cutting-edge technology and that most new technology is awash in its own novelty. 
“The most important thing is that you actually try to make a point with it, that you are making some meaning out of it. It is very easy to be the first of anything. What is really difficult is to make meaning out of it. That is what will age,” he says. “If you tell an amazing story or if you use what we call experiential learning – a way to engage and teach people something through experience – that is what will age well and what people will return for year after year.”
Improvisation and learning 
In another project in collaboration with the Gates Foundation and the U.S. Federal Department of Education, Barton wanted to employ technology to aid children’s learning by helping them see and experience the world in a different way.
The project was based on Confucius’ wisdom: “I hear and I forget, I see and I remember; I do and I understand,” which he says speaks to our very DNA and how we learn.
The challenge was to teach middle-school learners physics because the traditional methods of drilling them with equations and algorithms on a blackboard was not working. “What they weren’t realising is that they are actually enacting those very forces and dynamics on their own bodies out on the playground,” he says.  
So through a system of interfaces the learning process was turned on its head. Instead of starting off with the theory and then trying to apply it to reality, they used reality ­– what the children already knew ­– as the starting point, and worked their why back to the equations and formulas.
Improvisation and audience
Probably one of Local Projects’ biggest commissions to date is the National September 11 Memorial & Museum. The group was involved in the original masterplan and design of the media installations and went on to produce all of the media content for the exhibitions.   
According to Barton the biggest challenge was to cater to the diverse audiences that would visit the museum. 
“The challenge can be boiled down to two opposite types of visitors: on the one hand, you will have the 15-year-olds, young adults, who literally know nothing about the event, who were two or three when it happened and don't have any first-hand experience of it,” he explains. “And on the other hand, you have survivors. Almost 10 000 people who ran out of the buildings who literally made history that day and who arguably know more than any curator could ever know about the event.  And, so, how do you bring together this incredibly diverse audience?”
They came up with the idea of the museum as a platform where people could tell their own stories about 9/11. They put out a global call for stories told in videos, photos and written format, with people sharing what happened to them that day being overlaid between past and present.
They also installed a story booth that people enter, locate themselves on a map and record their experience of 9/11. So when visitors go to the museum, the first voice they will hear is not that of an historian or curator but another visitor telling you their 9/11 story. 
“As you can imagine this was a highly politically charged commission. This was not an easy institution to put forward radical ideas for; it is a very tough place to make work that is unconventional,” he says. 
While the client embraced the idea, Local Projects used prototyping to test the project with a large number of stakeholders, thereby also getting them to buy into it. 
“Throughout the exhibition we employed this same technique of overlapping oral histories, these memories,” he says, so that “you don't have an historian giving you the top-down view but you get a sense from these bits and pieces of recall.”
Several other ideas for exhibitions went through this prototyping or improvisation process as a means of negotiating ideas.
“When you are making failures, that is how you eventually get to something that is effective; that is an actual solution that you are going to like,” he explains. “Instead of even trying to talk to our clients often, particularly for 9/11, we would just make things and if they didn’t like it we would just make something else.” 
He goes to explain that is it a highly “unfun” way to work because most of your work ends up on the cutting-room floor but in the end you have something that both you and the client like and the audience responds to.
Improvisation and love 
Barton ends off with a project done in collaboration with architect Bjarke Ingles in New York’s Times Square. They installed an interactive sculpture of a heart that beats in sync with users’ own heartbeat when they touch the interface. As you chain together more people it would beat faster and faster.
“One thing that we learned from the 9/11 memorial is the ability to make a platform that engages so many different people,” he says. “It is actually exceptionally hard to make something for lots and lots of people that has some value to it. As you get bigger and bigger and try to satisfy more and more people, you end up with no value at all.”
He concludes: “In the end it is the love that flows through what you are doing into your audience; that is what is resonating and creating the level of impact.”