Can artistry be built into a machine?

One day recently, on a table in Jean Oh’s lab in the Squirrel Hill neighborhood of Pittsburgh, a robot arm was busy at a canvas. Slowly, as if the air were viscous, it dipped a brush into a pool of light-gray paint on a palette, swung around and stroked the canvas, leaving an inch-long mark amid a cluster of other brushstrokes. Then it pulled back and paused, as if to assess its work.

The strokes, mostly different shades of gray, suggested something abstract — an anthill, maybe. Oh, head of the roBot Intelligence Group at Carnegie Mellon University, dressed in a sweatshirt bearing the words “There Are Artists Among Us,” looked on with approval. Her doctoral student, Peter Schaldenbrand, stood alongside.

Oh’s work, which includes robot vision and topics in autonomous aviation, often touches on what is known as the sim-to-real gap: how machines trained in a simulated environment can act in the real world. In recent years, Schaldenbrand has led an effort to bridge the sim-to-real gap between sophisticated image-generation programs such as Stable Diffusion and physical works of art such as drawings and paintings. This has mainly been manifest in the project known as FRIDA, the latest iteration of which was rhythmically whirring away in a corner of the lab. (FRIDA is an acronym for Framework and Robotics Initiative for Developing Arts, although the researchers chose the acronym, inspired by Frida Kahlo, before deciding what it stood for.)

The process of moving from language prompts to pixelated images to brushstrokes can be complicated, as the robot must account for “the noise of the real world,” Oh said. But she, Schaldenbrand and Jim McCann, a roboticist at Carnegie Mellon who also helped develop FRIDA, believe that the research is worth pursuing for two reasons: It could improve the interface between humans and machines, and it could, through art, help connect people to one another.

“These models are trained based on everybody’s data,” McCann said, referring to the large language models that power tools like ChatGPT and DALL-E. “And so I still think we’re figuring out how projects like this, that use such models, can deliver value back to people.”

The sim-to-real gap provides a surprisingly tricky problem for roboticists and computer engineers. Some artificial intelligence systems can list the steps involved in walking (tighten your quadriceps and flex your tibialas posterior, tilt your weight back and tense your gluteus maximus) and can make a simulated body walk in a virtual world. So, it’s tempting to think that these systems could easily make a physical body walk in the real world.

Not so. In the 1980s, computer scientist Hans Moravec noted that AI was good at engaging in complicated reasoning and parsing vast amounts of data but that it was bad at simple physical activities, such as picking up a bottle of water. This is known as Moravec’s paradox. (The physical superiority of humans might be explained by our body’s long evolutionary history; the tasks that are simple for us are supported by millions of years of Darwinian experimentation.)

Painting, which often mixes high-concept ideas and basic physical actions, throws the paradox into relief: How do we manage to capture the absurdity of human consciousness with the motions of an arm?

AI image-generating tools such as Midjourney, DALL-E and Stable Diffusion are trained by feeding neural networks massive databases of images and corresponding text descriptions. The programmed goal is to model the relationships between the meanings of words and the features of images, and to then use these relationships in a “diffusion model” to create original images that retain the meaning of particular descriptions. (The prompt “A family picnicking in the park” will generate a new image every time it is used; each one will be understandable as a family picnicking in the park.)

But such images exist only in the sim-world of computers, composed of pixels of varying hue and intensity. Leave the simulation and the image stays behind.

This article originally appeared in The New York Times