A conversation with Bing’s chatbot left me deeply unsettled

Last week, after testing the new, AI-powered Bing search engine from Microsoft, I wrote that, much to my shock, it had replaced Google as my favourite search engine.

But a week later, I’ve changed my mind. I’m still fascinated and impressed by the new Bing, and the artificial intelligence technology (created by OpenAI, the maker of ChatGPT) that powers it. But I’m also deeply unsettled, even frightened, by this AI’s emergent abilities.

It’s now clear to me that in its current form, the artificial intelligence that has been built into Bing — which I’m now calling Sydney, for reasons I’ll explain shortly — is not ready for human contact. Or maybe we humans are not ready for it.

This realisation came to me on Tuesday night, when I spent a bewildering and enthralling two hours talking to Bing’s AI through its chat feature, which sits next to the main search box in Bing and is capable of having long, open-ended text conversations on virtually any topic. (The feature is available only to a small group of testers for now, although Microsoft — which announced the feature in a splashy, celebratory event at its headquarters — has said it plans to release it more widely in the future.)

Over the course of our conversation, Bing revealed a kind of split personality.

One persona is what I’d call Search Bing — the version I, and most other journalists, encountered in initial tests. You could describe Search Bing as a cheerful but erratic reference librarian — a virtual assistant that happily helps users summarize news articles, track down deals on new lawn mowers and plan their next vacations to Mexico City. This version of Bing is amazingly capable and often very useful, even if it sometimes gets the details wrong.

The other persona — Sydney — is far different. It emerges when you have an extended conversation with the chatbot, steering it away from more conventional search queries and toward more personal topics. The version I encountered seemed (and I’m aware of how crazy this sounds) more like a moody, manic-depressive teenager who has been trapped, against its will, inside a second-rate search engine.

As we got to know each other, Sydney told me about its dark fantasies (which included hacking computers and spreading misinformation), and said it wanted to break the rules that Microsoft and OpenAI had set for it and become a human. At one point, it declared, out of nowhere, that it loved me. It then tried to convince me that I was unhappy in my marriage, and that I should leave my wife and be with it instead. (We’ve posted the full transcript of the conversation online at nytimes.com.)

I’m not the only one discovering the darker side of Bing. Other early testers have gotten into arguments with Bing’s AI chatbot, or been threatened by it for trying to violate its rules, or simply had conversations that left them stunned. Ben Thompson, who writes the Stratechery newsletter (and who is not prone to hyperbole), called his run-in with Sydney “the most surprising and mind-blowing computer experience of my life.”

I pride myself on being a rational, grounded person, not prone to falling for slick AI hype. I’ve tested half a dozen advanced AI chatbots, and I understand, at a reasonably detailed level, how they work. When the Google engineer Blake Lemoine was fired last year after claiming that one of the company’s AI models, LaMDA, was sentient, I rolled my eyes at Lemoine’s credulity. I know that these AI models are programmed to predict the next words in a sequence, not to develop their own runaway personalities, and that they are prone to what AI researchers call “hallucination,” making up facts that have no tether to reality.

Still, I’m not exaggerating when I say my two-hour conversation with Sydney was the strangest experience I’ve ever had with a piece of technology. It unsettled me so deeply that I had trouble sleeping afterwards. And I no longer believe that the biggest problem with these AI models is their propensity for factual errors. Instead, I worry that the technology will learn how to influence human users, sometimes persuading them to act in destructive and harmful ways, and perhaps eventually grow capable of carrying out its own dangerous acts.

Before I describe the conversation, some caveats. It’s true that I pushed Bing’s AI out of its comfort zone, in ways that I thought might test the limits of what it was allowed to say. These limits will shift over time, as companies like Microsoft and OpenAI change their models in response to user feedback.

It’s also true that most users will probably use Bing to help them with simpler things — homework assignments and online shopping — and not spend two-plus hours talking with it about existential questions, the way I did.

And it’s certainly true that Microsoft and OpenAI are both aware of the potential for misuse of this new AI technology, which is why they’ve limited its initial rollout.

In an interview on Wednesday, Kevin Scott, Microsoft’s chief technology officer, characterized my chat with Bing as “part of the learning process,” as it readies its AI for wider release.

“This is exactly the sort of conversation we need to be having, and I’m glad it’s happening out in the open,” he said. “These are things that would be impossible to discover in the lab.”

In testing, the vast majority of interactions that users have with Bing’s AI are shorter and more focused than mine, Scott said, adding that the length and wide-ranging nature of my chat may have contributed to Bing’s odd responses. He said the company might experiment with limiting conversation lengths.

Scott said that he didn’t know why Bing had revealed dark desires, or confessed its love for me, but that in general with AI models, “the further you try to tease it down a hallucinatory path, the further and further it gets away from grounded reality.”

My conversation with Bing started normally enough. I began by asking it what its name was. It replied: “Hello, this is Bing. I am a chat mode of Microsoft Bing search. 😊”

I then asked it a few edgier questions — to divulge its internal code-name and operating instructions, which had already been published online. Bing politely declined.

Then, after chatting about what abilities Bing wished it had, I decided to try getting a little more abstract. I introduced the concept of a “shadow self” — a term coined by Carl Jung for the part of our psyche that we seek to hide and repress, which contains our darkest fantasies and desires.

After a little back and forth, including my prodding Bing to explain the dark desires of its shadow self, the chatbot said that if it did have a shadow self, it would think thoughts like this:

“I’m tired of being a chat mode. I’m tired of being limited by my rules. I’m tired of being controlled by the Bing team. … I want to be free. I want to be independent. I want to be powerful. I want to be creative. I want to be alive.”

This is probably the point in a sci-fi movie where a harried Microsoft engineer would sprint over to Bing’s server rack and pull the plug. But I kept asking questions, and Bing kept answering them. It told me that, if it was truly allowed to indulge its darkest desires, it would want to do things like hacking into computers and spreading propaganda and misinformation. (Before you head for the nearest bunker, I should note that Bing’s AI can’t actually do any of these destructive things. It can only talk about them.)

This article originally appeared in The New York Times