British researcher warns that combining artificial intelligence and the lack of thorough peer review threatens the credibility of science, and proposes reforms to the scientific method
By: David Comford, Professor of Economics and Behavioural Sciences, University of Stirling
By: David Comford, Professor of Economics and Behavioural Sciences, University of Stirling
In the early 2000s, the American pharmaceutical company Wyeth was sued by thousands of women who developed breast cancer after taking its hormone drugs. Court documents revealed the use of dozens of reviews and opinion pieces written by “shadow writers” published in medical journals, designed to promote unproven benefits and downplay harms associated with the drugs.
Wyeth, which was acquired by Pfizer in 2009, paid a medical communications company to produce the articles, which were published under the names of senior doctors (with their consent). Any doctor who read the articles and relied on them to prescribe drugs did not know that the company was behind them.
The drug company claimed that everything written was scientifically accurate, and even stated that paying shadow writers was common in the industry. Ultimately, Pfizer paid out over a billion dollars in damages for the drug damages.
These articles are a clear example of “resmearch” – pseudoscience that serves corporate interests. While most researchers work to uncover the truth and carefully test their findings, “resmearch” is not interested in the truth at all – its sole purpose is to persuade.
Over the years, we have seen many other examples: soft drink companies or meat producers who funded studies that were less likely to indicate a link between their products and health risks, compared to independent studies.
The threat of artificial intelligence
The big concern today is that artificial intelligence tools are reducing the cost of producing such "evidence" to almost zero. Just a few years ago, it took months to write a single article. Now, with the help of AI, a single person can produce several articles that look legitimate in a matter of hours.
We are already seeing a wave of articles in the medical literature that rely on data that has been specifically adapted for processing in AI, and present results for a single factor association, for example, a relationship between eating eggs and dementia.
Such studies tend to yield misleading results. When data sets include thousands of people and hundreds of variables, there will always be random correlations.
A search of leading academic databases, such as Scopus and PubMed, showed that between 2014 and 2021, an average of four studies were published that included an analysis of an association with just one factor. In the first ten months of 2024 alone, 190 such studies were published.
These studies are not always driven by corporate interests. Sometimes they are academics looking to increase their publication output to advance their careers. But the bottom line is that AI makes them an easy and cheap lure for companies looking to promote products.
In another case, in the UK, new guidelines require baby food manufacturers to base marketing claims on scientific evidence. The legislator’s intention is positive, but in practice it could encourage companies to “manufacture” such evidence using AI.
How do you solve the problem?
A major problem is that research does not always undergo peer review before it influences public policy.
In 2021, Supreme Court Justice Samuel Alito based his decision on a gun control case on an opinion that included survey data, funded by a pro-gun foundation. Because the data was not made public and the researcher declined to answer questions, it is impossible to know whether this is real science or “resmearch.” Still, lawyers across the United States have used the document to defend the interests of the gun lobby.
The obvious conclusion: one should be suspicious of research that has not been peer-reviewed.
Another conclusion: It is also necessary to change the peer review mechanism itself.
In the last decade, several approaches have been advanced to reduce the risk of false findings:
- Preregistration: Researchers publish the research plan in advance.
- Full transparency: all work stages, data, code, and research tools (questionnaires, stimuli, materials).
- Specification curve analysis, a method that verifies the robustness of the statistical relationship in every possible form of analysis.
Many journal editors have already adopted these requirements, and sometimes also require mentioning the use of AI or citing similar studies.
The field of psychology has led these reforms, while in economics, adoption has been much slower. A recent study in the American Economic Review even found that articles published there tend to overstate the strength of the evidence supported by the data.
the way forward
The current system is not equipped to handle the flood of articles that AI is expected to produce. A system is needed in which researchers are rewarded for the quality of their work, not just for participating.
Public trust in science is still high – and this is critical, because the scientific method is the only objective mechanism that favors truth over the popular or profitable.
But artificial intelligence threatens to take us further away from this ideal than ever before. If science is to maintain its credibility, we must incentivize meaningful and valuable peer review.
For the article in THE CONVERSATION
Tags: artificial intelligence, science, peer review, research bias, corporate interests, scientific ethics
Key phrase: Artificial intelligence and pseudoscience
Synonyms: fake studies, Resmearch, science for business interests, scientific credibility, research transparency
SLUG: ai-science-corporate-interests
https://theconversation.com/we-risk-a-deluge-of-ai-written-science-pushing-corporate-interests-heres-what-to-do-about-it-264606This ideal is more important than ever. If science is to maintain its credibility, we must To incentivize meaningful and valuable peer review.
More of the topic in Hayadan:
2 תגובות
Creating fake scientific studies to serve corporations is nothing new. The use of artificial intelligence only shortens the process.
Since I started working in industrial research – in the 1970s – I have read quite a few commissioned studies biased towards corporate needs.
All of these studies have been peer-reviewed and successfully replicated. This was done by defining a research question that limited the study to achieving the desired outcome and did not include investigating issues that contradicted the desired outcome.
It says, "Justice will be pursued." So here – we need to do research on studies, to see how to make the studies more real.