Why people plus AI can do worse than either alone

It seems obvious that a person working with AI will beat either one alone. A large review of the evidence says that is often untrue, and explains when it is.

The short version

  • A 2024 review pooled 106 experiments and 370 results on people working with AI.
  • On average the combination did better than people alone, and worse than the better of the person or the AI working alone.
  • Results improved when the person was the stronger party and when the task involved creating something.

What the researchers did

A team at the MIT Center for Collective Intelligence gathered experiments published between 2020 and 2023 in which people, AI systems, and the two together each did the same task. They compared the combination with the best of the other two.

What they found

Pairing a person with AI usually beat the person alone. It often fell short of whichever was better alone. When the AI was more accurate than the person, adding the person tended to make results worse. When the person was more accurate, adding AI helped. Tasks that involved creating content showed gains, while tasks that involved making a decision showed losses.

What it means for a manager

Putting a person in the loop does not make the work better by itself. If the person cannot tell when the AI is right and when it is wrong, they will overrule good answers and accept bad ones.

The answer is to design the pairing: decide who does which part, give the person what they need to judge the AI's work, and check the result against how each would have done alone. That is ordinary job design, applied to a new kind of colleague.

What to do

  1. For one task where people now work with AI, find out how each performs alone. Without that you cannot know whether the pairing helps.
  2. Give each part of the task to whoever is better at it. Avoid having both do everything.
  3. Make sure the person reviewing AI output has the time, information and skill to disagree with it.
  4. Measure the combined result after a month and compare it with your starting point.

Words explained

Human in the loop
A person who reviews or approves what an AI system does.
Meta-analysis
A study that combines the results of many separate studies to see the overall pattern.
Synergy
When two working together do better than the best of either alone.

Source: Vaccaro, Almaatouq and Malone, Nature Human Behaviour, 2024.

All read-outs and the plain-language glossary