New paper: The emergence of Large Language Models (LLM) as a tool in literature

[maykcaldas]

Paper: The emergence of Large Language Models (LLM) as a tool in literature

Authors: Dmitry Scherbakov, Nina Hubig, Vinita Jansari, Alexander Bakumenko,

Abstract: Objective: This study aims to summarize the usage of Large Language Models(LLMs) in the process of creating a scientific review. We look at the range ofstages in a review that can be automated and assess the currentstate-of-the-art research projects in the field. Materials and Methods: Thesearch was conducted in June 2024 in PubMed, Scopus, Dimensions, and GoogleScholar databases by human reviewers. Screening and extraction process tookplace in Covidence with the help of LLM add-on which uses OpenAI gpt-4o model.ChatGPT was used to clean extracted data and generate code for figures in thismanuscript, ChatGPT and Scite.ai were used in drafting all components of themanuscript, except the methods and discussion sections. Results: 3,788 articleswere retrieved, and 172 studies were deemed eligible for the final review.ChatGPT and GPT-based LLM emerged as the most dominant architecture for reviewautomation (n=126, 73.2%). A significant number of review automation projectswere found, but only a limited number of papers (n=26, 15.1%) were actualreviews that used LLM during their creation. Most citations focused onautomation of a particular stage of review, such as Searching for publications(n=60, 34.9%), and Data extraction (n=54, 31.4%). When comparing pooledperformance of GPT-based and BERT-based models, the former were better in dataextraction with mean precision 83.0% (SD=10.4), and recall 86.0% (SD=9.8),while being slightly less accurate in title and abstract screening stage(Maccuracy=77.3%, SD=13.0). Discussion/Conclusion: Our LLM-assisted systematicreview revealed a significant number of research projects related to reviewautomation using LLMs. The results looked promising, and we anticipate thatLLMs will change in the near future the way the scientific reviews areconducted.

Link: https://arxiv.org/abs/2409.04600

Reasoning: produce the answer. We start by examining the title, which mentions "Large Language Models (LLM)" and their use in literature. This strongly suggests that the paper is about language models. Next, we look at the abstract, which discusses the usage of Large Language Models (LLMs) in automating various stages of creating a scientific review. It mentions specific models like OpenAI's GPT-4 and their applications in data extraction, drafting, and other tasks. The abstract also compares the performance of GPT-based and BERT-based models, which are types of language models.