AI companies have been struggling to identify whether a piece of writing has been generated using a large language model. Researchers have developed a new method to estimate LLM usage in scientific writing by analyzing the frequency of “excess words” that appeared more frequently during the LLM era, particularly in 2023 and 2024. The study, conducted by researchers from Germany’s University of Tübingen and Northwestern University, aimed to reveal the impact of LLMs on scientific abstracts published on PubMed between 2010 and 2024.
The researchers found that at least 10 percent of abstracts in 2024 showed signs of LLM usage based on the sudden surge of certain style words. Words like “delves,” “showcasing,” and “underscores” saw a significant increase in usage after the introduction of LLMs. Additionally, common words like “potential,” “findings,” and “crucial” also experienced a notable increase in frequency in post-LLM abstracts. These vocabulary changes were unprecedented in both quality and quantity, indicating a shift in language use within scientific writing.
The researchers compared the frequency of words in abstracts from 2023 and 2024 to the expected frequency based on pre-2023 trends. They observed a sharp rise in the usage of certain words that were previously uncommon in scientific abstracts. While language evolution can explain some changes in vocabulary, the researchers found that the sudden and massive increases in word frequency were directly linked to the introduction of LLMs. This contrasted with previous trends where major health events like the Ebola outbreak or the COVID-19 pandemic influenced word usage in scientific writing.
The study identified hundreds of “marker words” that became significantly more common in post-LLM abstracts, indicating the use of LLMs in writing. These marker words were primarily verbs, adjectives, and adverbs, differentiating them from the noun-heavy excess words observed during the COVID-19 pandemic. By analyzing the prevalence of these marker words across individual papers, the researchers estimated that at least 10 percent of post-2022 papers on PubMed were written with LLM assistance. However, this number could be higher due to the potential exclusion of LLM-assisted abstracts that do not contain the identified marker words.
The findings of this research shed light on the profound impact of large language models on scientific writing. The identification of marker words and the analysis of vocabulary changes provide valuable insights into the prevalence of LLM usage in academic literature. As the use of LLMs continues to grow, it is crucial for researchers and publishers to be aware of the implications of automated writing tools on language evolution in scientific communication. This study serves as a critical analysis of the evolving landscape of scientific writing in the era of artificial intelligence.
Leave a Reply