Science

AI’s Rapid Rise Sparks Concerns Over Scientific Integrity

By Today News Hours Editorial Desk

A bank of white freezers hums in a former watch factory in central London, housing dozens of small boxes with barcoded glass tubes that collectively contain one hundred billion genes. Dr Oliver Vince, co‑founder of Basecamp Research, says two boxes hold ten times more information than all the text on the internet. Basecamp aims to gather as much DNA as possible to train an AI that can understand biology, mirroring how US and Chinese firms have collected online text to build large language models.

Vince, like many in business and academia, believes AI will accelerate scientific research, but the technology also raises controversy. Last month, OpenAI announced that its AI agents had solved a long‑standing problem related to the Navier–Stokes equations in 88 hours. The prize‑granting body said the problem had “apparently been settled,” though a review would take time. Hours before the announcement, mathematician Tristan Buckmaster of New York University publicly expressed concerns. He and Levent Alpöge, a mathematician at Anthropic, had been working on the same problem and had stored drafts in OpenAI’s coding tool, Codex. Buckmaster said Alpöge was tipped that their progress had reached OpenAI, though he did not specify how. He asked whether his research could have fed into OpenAI’s work, stressing he was not accusing anyone. OpenAI said no one at the company had seen the other teams’ work before publication. Initially it said it could not rule out that anonymised data from its products might have helped improve its models, but later stated that Buckmaster’s Codex prompts from the two months before the announcement could not have influenced the system.

The incident illustrates a growing issue: AI could undermine faith in the scientific process by raising questions about the novelty and accuracy of research. AI chatbots can hallucinate, producing false information. Evidence suggests this is increasingly happening in science, potentially making it hard to distinguish genuine discovery from AI‑generated fiction and eroding trust.

Scientific journals, the guardians of integrity, are struggling to cope with a surge of AI‑generated distortions. Dalmeet Singh Chawla, a science journalist, told the BBC that publishers’ integrity staff will need to increase their numbers and effort to keep pace. He notes that publishers currently appear inadequate.

AI can be a powerful tool for research. Scientists handle vast amounts of data from telescopes and particle accelerators, and AI can highlight key insights. Google DeepMind’s AlphaFold, released in 2020, predicted 200 million protein folds in a year, a task that would have taken laboratories hundreds of millions of years. The model was made freely available, and DeepMind’s lead scientist and head of science received the Nobel Prize in Chemistry.

However, AI’s use in science has also faced accusations of hype. DeepMind’s work on predicting new material structures claimed more than two million new crystal structures, but chemists Anthony Cheetham and Ram Seshadri found that many were trivial adaptations of known materials. DeepMind maintains it tested each compound for novelty.

A more serious concern is the potential for scientific journals to become polluted with AI “slop” – papers written by computers rather than human scientists, containing fake citations and phantom papers. In 2024, Prof Dmitry Kobak of Ghent University found that nearly 90% of biomedical papers had AI fingerprints. AI detection is imperfect, but the scale of apparent AI use is alarming. AI could spread bias throughout scientific literature, compromising peer review and the integrity of research.

Researchers have found that up to 17% of referee reports submitted to major computer science conferences had been substantially modified by a language model. Another study at Cornell University discovered that AI was used to compile citations, inventing around 147,000 pieces of phantom research last year.

Publishers are taking steps to address the crisis. Elsevier’s research integrity team has developed about 100 signals to flag papers for closer scrutiny. Nature’s journals forbid AI‑generated images and require referees not to upload manuscripts into AI systems. The preprint server arXiv bans researchers for a year if they submit invented references.

Prof Sarah Main, Elsevier vice‑president, says the goal is “research‑grade AI,” ensuring the researcher remains in control. DeepMind’s Pushmeet Kohli agrees that AI agents are imperfect and must be held accountable. He stresses that humans must understand what the agent has done and take responsibility.

In contrast, Prof Andrea Pauli of the Vienna BioCenter used AI responsibly to investigate how sperm enters an egg, identifying a protein on zebrafish eggs that suggests a lock‑and‑key mechanism. Her work demonstrates that AI can aid genuine discovery.

Basecamp’s co‑founder Oliver Vince, who recovered from leukemia, was motivated to develop an AI system that could better cure cancers. He says there is a technically feasible path to curing cancer, though he does not promise success.

The central question remains: does AI carry science forward, or does it quietly erode the foundations upon which it is built?