Fine-Tuned Large Language Models Can Replicate Expert Coding Better Than Trained Coders: A Study on Informative Signals Sent by Interest Groups

Abstract

Understanding how political information is transmitted requires tools that can reliably and scalably capture complex signals in text. While existing studies point to interest groups as strategic information providers, studying this aspect empirically has been challenging due to the need for expert-level annotation in measurement. Using a case study of policy documents released by interest groups on U.S. trade policy, this study shows that fine-tuned LLMs outperform lightly-trained workers, crowdworkers, and zero-shot LLMs in identifying two difficult-to-separate categories of signals: 1) informative signals that help agents improve political decisions, and 2) associative signals that influence preference formation but lack direct relevance to the substantive topic of interest. We demonstrate the utility of this approach using two applications where our classifier generalizes out of distribution. While the empirical focus is domain-specific, we offer a scalable approach to expert-driven text coding that can be adapted to other domains of political inquiry.