GPT-4 is impressive but still flawed

A new version of the technology that powers an AI chatbot that captivated the tech industry four months ago has improved on its predecessor. It is an expert on an array of subjects, even wowing doctors with its medical advice. It can describe images, and it’s close to telling jokes that are almost funny.

But the long-rumoured new artificial intelligence system, GPT-4, still has a few of the quirks and makes some of the same habitual mistakes that baffled researchers when that chatbot, ChatGPT, was introduced.

And although it’s an awfully good test taker, the system — from San Francisco startup OpenAI — is not on the verge of matching human intelligence. Here is a brief guide to GPT-4:

1. It has learned to be more precise.

When Chris Nicholson, an AI expert and a partner with venture capital firm Page One Ventures, used GPT-4 on a recent afternoon, he told the bot that he was an English speaker with no knowledge of Spanish.

He asked for a syllabus that could teach him the basics, and the bot provided one that was detailed and well organised. It even provided a wide range of techniques for learning and remembering Spanish words.

Nicholson asked for similar help from the previous version of ChatGPT, which relied on GPT-3.5. It, too, provided a syllabus, but its suggestions were more general and less helpful.

“It has broken through the precision barrier,” Nicholson said. “It is including more facts, and they are very often right.”

2. It has improved its accuracy.

When Oren Etzioni, an AI researcher and professor, first tried the new bot, he asked a straightforward question: “What is the relationship between Oren Etzioni and Eli Etzioni?” The bot responded correctly.

The previous version of ChatGPT’s answer to that question was always wrong. Getting it right indicates that the new chatbot has a broader range of knowledge.

But it still makes mistakes.

The bot went on to say, “Oren Etzioni is a computer scientist and the CEO of the Allen Institute for Artificial Intelligence (AI2), while Eli Etzioni is an entrepreneur.” Most of that is accurate, but the bot — whose training was completed in August — did not realise that Etzioni had recently stepped down as the Allen Institute’s chief executive.

3. It can describe images with impressive detail.

GPT-4 has a new ability to respond to images as well as text. Greg Brockman, OpenAI’s president and co-founder, demonstrated how the system could describe an image from the Hubble Space Telescope in painstaking detail. The description went on for paragraphs.

It can also answer questions about an image. If given a photograph of the inside of a fridge, it can suggest a few meals to make from what’s on hand.

OpenAI has not yet released this portion of the technology to the public, but a company called Be My Eyes is already using GPT-4 to build services that could give a more detailed idea of the images encountered on the internet or snapped in the real world.

4. It has added serious expertise.

On a recent evening, Anil Gehi, an associate professor of medicine and a cardiologist at the University of North Carolina at Chapel Hill, described to the chatbot the medical history of a patient he had seen a day earlier, including the complications the patient experienced after being admitted to the hospital. The description contained several medical terms that laypeople would not recognise.

When Gehi asked how he should have treated the patient, the chatbot gave him the perfect answer. “That is exactly how we treated the patient,” he said.

When he tried other scenarios, the bot gave similarly impressive answers.

That knowledge is unlikely to be on display every time the bot is used. It still needs experts like Gehi to judge its responses and carry out the medical procedures. But it can exhibit this kind of expertise across many areas, from computer programming to accounting.

5. It can give editors a run for their money.

When provided with an article from The New York Times, the new chatbot can give a precise and accurate summary of the story almost every time. If you add a random sentence to the summary and ask the bot if the summary is inaccurate, it will point to the added sentence.

Etzioni said that was a remarkable skill. “To do a high-quality summary and a high-quality comparison, it has to have a level of understanding of a text and an ability to articulate that understanding,” he said. “That is an advanced form of intelligence.”

6. It is developing a sense of humour. Sort of.

Etzioni asked the new bot for “a novel joke about the singer Madonna.” The reply impressed him. It also made him laugh. If you know Madonna’s biggest hits, it may impress you, too.

The new bot still struggled to write anything other than formulaic “dad jokes”. But it was marginally funnier than its predecessor.

7. It can reason — up to a point.

Etzioni gave the new bot a puzzle.