Humans v Machine: fake news headlines or real deal?

Can you tell whether the title of a scientific article was written by a human or an AI? Because half the time, not even doctors can spot the fake science headline, according to a paper published in BMJ.

Go on, give it a try!

In a study worthy of the silly season, researchers used an AI to generate research paper titles and tested whether readers could tell if they were fake.

They took the top titles from 10 years of BMJs Christmas edition – which are often quirkier than normal – to teach an AI to write its own titles. These were then rated by a random sample of doctors from multiple disciplines and countries.

They found that AI-generated titles were rated at least as enjoyable (69%) compared to real titles (64%), although the real titles were rated as more plausible (73%) than AI titles (48%).

They also found that the AI titles were deemed less scientific if generated at random, but this became less apparent when the titles were then curated by a human.

The authors say that this shows how the best results come from an AI and a human working together, where the AI can compensate for human oversights but humans can make the final call.

The two AI-generated titles deemed the most plausible were: “The clinical effectiveness of lollipops as a treatment for sore throats” and “The effects of free gourmet coffee on emergency department waiting times: an observational study.”

The silliest title generated by the AI was: “Superglue your nipples together and see if it helps you to stop agonising about erectile dysfunction at work.” The authors note that this demonstrates the AI doesn’t know how to be polite, which limits its real-world application without human help.