Can Language Models Recognize Convincing Arguments?
The capabilities of large language models (LLMs) have raised concerns about their potential to create and propagate convincing narratives.Here, we study their performance in detecting convincing arguments to gain insights into LLMs' persuasive capabilities without directly engaging in experimentation with humans.We extend a dataset by Durmus and Cardie (2018) with debates, votes, and user traits and propose tasks measuring LLMs' ability to (1) distinguish between strong and weak arguments, (2) predict stances based on beliefs and demographic characteristics, and (3) determine the appeal of an argument to an individual based on their traits.We show that LLMs perform on par with humans in these tasks and that combining predictions from different LLMs yields significant performance gains, surpassing human performance.The data and code released with this paper contribute to the crucial effort of continuously evaluating and monitoring LLMs' capabilities and potential impact.(https://go.epfl.ch/persuasion-llm).
2024.findings-emnlp.515.pdf
Main Document
http://purl.org/coar/version/c_970fb48d4fbd8a85
openaccess
CC BY
582.09 KB
Adobe PDF
6536c558264015af158ed461f04fcd99