Publications

Abstract

We introduce EgoPolice, a carefully curated dataset of real, egocentric police-civilian interactions, sourced from publicly available body-worn camera videos. We select police-civilian action labels that are critical for police behavioral research and annotate them at a second-by-second granularity. The videos feature rapid and irregular camera motion, dense human interactions, and rare high-stakes events, making the dataset a challenging benchmark for motion-robust and context-aware egocentric perception. We provide two different tasks, classification and multiple-choice question-answering, and benchmark both open-source and closed-source models. We find that even the best video models like Gemini 2.5 Pro still struggle to accurately predict high-risk actions such as "Weapon Out". Beyond serving as a benchmark, EgoPolice provides a foundation for developing models capable of identifying events of interest in large-scale body-worn camera video repositories, enabling more efficient downstream human review.

@inproceedings{gonzalezsaezdiez2026egopolice,
  title = {EgoPolice: A Benchmark for Egocentric Video Understanding in High-Stakes Police Body-Worn Camera Footage},
  author = {Gonzalez Saez-Diez, Max and Chung, Jihoon and Wolsky, Adam D. and Lanzalotto, Gregory and Knox, Dean and Mummolo, Jonathan and Stewart, Brandon M. and Russakovsky, Olga},
  booktitle = {European Conference on Computer Vision},
  year = {2026},
  url = {https://arxiv.org/abs/2607.06468}
}

Abstract

Survey researchers try to align respondent question interpretations with their own by applying theoretical rules, such as writing questions simply, concretely, and without biasing words or double-barreled inquiries, or via post hoc plausibility checks. Few, however, conduct ex ante empirical evaluations of their application of these rules. The widely recognized best empirical approach is "cognitive debriefing," that follows a survey question with a detailed conversation with the respondent. Unfortunately, this time-consuming procedure is rarely used, especially for novel surveys on online platforms. This paradoxically leaves the veracity of the discipline's most used empirical methodology depending largely on theoretical assumptions. For cognitive debriefing to live up to its intended potential, we first formalize the essential interpretation-related assumptions underlying all survey questions. We then automate cognitive debriefing to satisfy these assumptions using a specially-tuned chatbot, apply it to larger numbers than heretofore possible, analyze the collection of transcripts, and summarize the interpretations it reveals. By iteratively rewriting questions and rerunning this procedure, researchers can help ensure respondents understand questions as they do. We also apply our procedure to some of the most common survey questions and demonstrate large disconnects between researcher intentions and respondent understandings. We make available easy-to-use open source software that implements all our suggestions.

@unpublished{hinck2026debrief,
  title = {Where's the Evidence that Respondents Understand Your Survey Questions?},
  author = {Hinck, Musashi and King, Gary and Nakamura, Kentaro and Stewart, Brandon M.},
  year = {2026}
}

Abstract

Millions of people around the world query large language models for information. While several studies have compellingly documented the persuasive potential of these models, there is limited evidence of who or what influences the models themselves, leading to a flurry of concerns about which companies and governments build and regulate the models. We show through six studies that government control of the media across the world already influences the output of large language models (LLMs) via their training data. We use a cross-national audit to show that LLMs exhibit a stronger pro-government valence in the languages of countries with lower media freedom than those with higher media freedom. This result is correlational so to triangulate the specific mechanism of how state media control can influence LLMs, we develop a multi-part case study on China's media. We demonstrate that media scripted and curated by the Chinese state appears in large language model training datasets. To evaluate the plausible effect of this inclusion, we use an open-weight model to show that additional pretraining on Chinese state-coordinated media generates more positive answers to prompts about Chinese political institutions and leaders. We link this phenomenon to commercial models through two audit studies demonstrating that prompting models in Chinese generates more positive responses about China's institutions and leaders than do the same queries in English. The combination of influence and persuasive potential across languages suggests the troubling conclusion that states and powerful institutions have increased strategic incentives to leverage media control in the hopes of shaping large language model output.

@article{Waight_2026, title={State media control influences large language models}, ISSN={1476-4687}, url={https://doi.org/10.1038/s41586-026-10506-7}, DOI={10.1038/s41586-026-10506-7}, journal={Nature}, publisher={Springer Science and Business Media LLC}, author={Waight, Hannah and Yang, Eddie and Yuan, Yin and Messing, Solomon and Roberts, Margaret E. and Stewart, Brandon M. and Tucker, Joshua A.}, year={2026}, month=May }

Abstract

Understanding how political information is transmitted requires tools that can reliably and scalably capture complex signals in text. While existing studies point to interest groups as strategic information providers, studying this aspect empirically has been challenging due to the need for expert-level annotation in measurement. Using a case study of policy documents released by interest groups on U.S. trade policy, this study shows that fine-tuned LLMs outperform lightly-trained workers, crowdworkers, and zero-shot LLMs in identifying two difficult-to-separate categories of signals: 1) informative signals that help agents improve political decisions, and 2) associative signals that influence preference formation but lack direct relevance to the substantive topic of interest. We demonstrate the utility of this approach using two applications where our classifier generalizes out of distribution. While the empirical focus is domain-specific, we offer a scalable approach to expert-driven text coding that can be adapted to other domains of political inquiry.

@article{choi2026finetuned,
  title = {Fine-Tuned Large Language Models Can Replicate Expert Coding Better Than Trained Coders: A Study on Informative Signals Sent by Interest Groups},
  author = {Choi, Dahyun and Peskoff, Denis and Stewart, Brandon M.},
  journal = {Political Science Research and Methods},
  pages = {1--19},
  year = {2026},
  doi = {10.1017/psrm.2025.10086}
}

Abstract

Instrumental variables (IV) analysis is a powerful, but fragile, tool for drawing causal inferences from observational data. Sociologists have increasingly turned to this strategy in settings where unmeasured confounding between the treatment and outcome is likely. This paper provides an introduction to the assumptions required for IV and consequences of their violations for applications in sociology. We review three methodological problems IV faces: identification bias (asymptotic bias from assumption violations), estimation bias (finite-sample bias that persists even when assumptions hold), and type-M error (exaggeration of effects given statistical significance). In each case, we emphasize how weak instruments exacerbate these problems and make results sensitive to minor violations of assumptions. Our discussion is informed by a new survey of IV papers published in top sociology journals showing that assumptions often go unstated and robust uncertainty measures are rarely used. We provide a practical checklist to show how IV, despite its fragility, can still be useful when handled with care.

@article{Felton_2024, title={Handle with Care: A Sociologist’s Guide to Causal Inference with Instrumental Variables}, volume={55}, ISSN={1552-8294}, url={https://doi.org/10.1177/00491241241235900}, DOI={10.1177/00491241241235900}, number={1}, journal={Sociological Methods \& Research}, publisher={SAGE Publications}, author={Felton, Chris and Stewart, Brandon M.}, year={2024}, month=Aug, pages={3–50} }

Abstract

Inference-Time-Compute (ITC) methods like Best-of-N and Tree-of-Thoughts are meant to produce output candidates that are both high-quality and diverse, but their use of high-temperature sampling often fails to achieve meaningful output diversity. Moreover, existing ITC methods offer limited control over how to perform reasoning, which limits both their performance and explainability. We present STATe-of-Thoughts (STATe), an explainable ITC method that searches over high-level reasoning patterns. STATe replaces stochastic sampling with discrete and interpretable textual interventions: a controller selects actions encoding high-level reasoning choices, a generator produces reasoning steps conditioned on those choices, and an evaluator scores candidates to guide search. This structured approach yields four main advantages. First, action-guided textual interventions produce greater response diversity than temperature-based sampling. Second, when paired with reliable evaluation, STATe's diverse yet high-quality outputs surpass existing ITC methods on instruction following. Third, in a case study on argument generation, STATe's explicit action sequences capture interpretable features that are highly predictive of output quality. Fourth, estimating the association between performance and action choices allows us to identify promising yet unexplored regions of the action space and steer generation directly toward them. Together, these results establish STATe as a practical framework for generating high-quality, diverse, and interpretable text. Our framework is available at https://github.com/zbambergerNLP/dspy-reasoning.

@inproceedings{bamberger2026stateofthoughts,
      title={STATe-of-Thoughts: Structured Action Templates for Tree-of-Thoughts},
      author={Zachary Bamberger and Till R. Saenger and Gilad Morad and Ofra Amir and Brandon M. Stewart and Amir Feder},
      booktitle={Conference on Language Modeling (COLM)},
      year={2026},
      note={Accepted; preprint available at arXiv:2602.14265},
      url={https://arxiv.org/abs/2602.14265}
}

Abstract

Social scientists use automated annotation methods, such as supervised machine learning and, more recently, large language models (LLMs), that can predict labels and generate text-based variables. While such predicted text-based variables are often analyzed as if they were observed without errors, we show that ignoring prediction errors in the automated annotation step leads to substantial bias and invalid confidence intervals in downstream analyses, even if the accuracy of the automated annotations is high, e.g., above 90%. We propose a framework of design-based supervised learning (DSL) that can provide valid statistical estimates, even when predicted variables contain non-random prediction errors. DSL employs a doubly robust procedure to combine predicted labels and a smaller number of expert annotations. DSL allows scholars to apply advances in LLMs to social science research while maintaining statistical validity. We illustrate its general applicability using two applications where the outcome and independent variables are text-based.

@unpublished{egami2026llmannotations,
  title = {Using Large Language Model Annotations for the Social Sciences: A General Framework of Using Predicted Variables in Downstream Analyses},
  author = {Egami, Naoki and Hinck, Musashi and Stewart, Brandon M. and Wei, Hanying},
  note = {Conditionally accepted at American Journal of Political Science},
  year = {2026}
}

Abstract

We argue that almost all empirical social science research employs a mode of argumentation called "Inference to the Best Explanation" (IBE). While IBE is widely used, it is seldom explicated, leading to confusion about the role that evidence plays in social science claims. We describe what IBE is and its implications for the evaluation of studies of various types from quantitative description, to traditional regression studies, to work with modern causal identification. We contend that debates over the merits of these approaches should be understood as debates over the relative weight scholars place on quality of explanations, the quality of evidence and the bridge between the two. Seeing the process this way helps highlight the merits of different research traditions and explains common pathologies of the same. Informed by IBE, we clarify the motivation for certain advice in regression studies and offer guidance on best practice.

@article{Spirling_2025, title={What Good Is a Regression? Inference to the Best Explanation and the Practice of Political Science Research}, volume={87}, ISSN={1468-2508}, url={https://doi.org/10.1086/734280}, DOI={10.1086/734280}, number={4}, journal={The Journal of Politics}, publisher={University of Chicago Press}, author={Spirling, Arthur and Stewart, Brandon M.}, year={2025}, month=Oct, pages={1587–1599} }

Abstract

Generative artificial intelligence (AI) has shown incredible leaps in performance across data of a variety of modalities including texts, images, audio, and videos. This affords social scientists the ability to annotate variables of interest from unstructured media. While rapidly improving, these methods are far from perfect and, as we show, even ignoring the small amounts of error in high accuracy systems can lead to substantial bias and invalid confidence intervals in downstream analysis. We review how using design-based supervised learning (DSL) guarantees asymptotic unbiasedness and proper confidence interval coverage by making use of a small number of expert annotations. While originally developed for use with large language models in text, we present a series of applications in the context of image analysis, including an investigation of visual predictors of the perceived level of violence in protest images, an analysis of the images shared in the Black Lives Matter movement on Twitter, and a study of US outlets reporting of immigrant caravans. These applications are representative of the type of analysis performed in the visual social science landscape today, and our analyses will exemplify how DSL helps us attain statistical guarantees while using automated methods to reduce human labor.

@article{Rister_Portinari_Maranca_2025, title={Correcting the Measurement Errors of AI-Assisted Labeling in Image Analysis Using Design-Based Supervised Learning}, volume={54}, ISSN={1552-8294}, url={https://doi.org/10.1177/00491241251333372}, DOI={10.1177/00491241251333372}, number={3}, journal={Sociological Methods \& Research}, publisher={SAGE Publications}, author={Rister Portinari Maranca, Alessandra and Chung, Jihoon and Hinck, Musashi and Wolsky, Adam D. and Egami, Naoki and Stewart, Brandon M.}, year={2025}, month=Apr, pages={984–1016} }

Abstract

Word embeddings are now a vital resource for social science research. But it can be difficult to obtain high quality embeddings for non-English languages, and it may be computationally expensive to do so. In addition, social scientists typically want to make statistical comparisons and do hypothesis tests on embeddings, yet this is non-trivial with current approaches. We provide three new data resources designed to ameliorate the union of these issues: (1) a new version of fastText model embeddings; (2) a multi-language “a la carte” (ALC) embedding version of the fastText model; (3) a multi-language ALC embedding version of the well-known GloVe model. All three are fit to Wikipedia corpora. These materials are aimed at “low resource” settings where the analysts lack access to large corpora in their language of interest, or lack access to the computational resources required to produce high-quality vector representations. We make these resources available for 30 languages, along with a code pipeline for another 127 languages available from Wikipedia corpora. We provide extensive validation of the materials, via reconstruction tests and other proofs-of-concept. We also conduct human crowdworker tests, for our embeddings for Arabic, French, (traditional, Mandarin) Chinese, Japanese, Korean, Russian and Spanish. Finally, we offer some advice to practitioners using our resources.

@article{Wirsching_2024, title={Multilanguage Word Embeddings for Social Scientists: Estimation, Inference, and Validation Resources for 157 Languages}, volume={33}, ISSN={1476-4989}, url={https://doi.org/10.1017/pan.2024.17}, DOI={10.1017/pan.2024.17}, number={2}, journal={Political Analysis}, publisher={Cambridge University Press (CUP)}, author={Wirsching, Elisa M. and Rodriguez, Pedro L. and Spirling, Arthur and Stewart, Brandon M.}, year={2024}, month=Dec, pages={156–163} }

Abstract

Autocratic governments around the world use clandestine propaganda campaigns to influence the media. We document a decade-long trend in China towards the planting of government-authored articles in party and commercial newspapers. To examine this phenomenon, we develop a new approach to identifying scripted propaganda - the coerced reprinting of lightly-adapted government-authored articles in newspapers - that leverages the footprints left by the government when making media interventions. We show that in China, scripted propaganda is a daily phenomenon - on 90% of days from 2012-2022, the vast majority of party newspapers include at least some scripted propaganda at the direction of a central directive. On particular sensitive days, the amount of scripted propaganda can spike to 30% of the articles appearing in major newspapers. We show that scripted propaganda has strengthened under President Xi Jinping. In the last decade the front page of party newspapers has evolved from 5% scripted articles to approximately 20% scripted. This government-authored content throughout the paper is increasingly homogeneous - fewer and fewer adaptations are done by individual newspapers. In contrast to popular speculation, we show that scripted content is not only on ideological topics (although it is increasingly ideological) and is also very prevalent in commercial papers. Using a case study of domestic coverage of COVID-19, we demonstrate how the regime uses scripting to shape, constrain, and delay information during crises. Our findings reveal the wide-ranging influence of government-authored propaganda in the Chinese media ecosystem.

@article{Waight_2025, title={The decade-long growth of government-authored news media in China under Xi Jinping}, volume={122}, ISSN={1091-6490}, url={https://doi.org/10.1073/pnas.2408260122}, DOI={10.1073/pnas.2408260122}, number={11}, journal={Proceedings of the National Academy of Sciences}, publisher={National Academy of Sciences}, author={Waight, Hannah and Yuan, Yin and Roberts, Margaret E. and Stewart, Brandon M.}, year={2025}, month=Mar }

Abstract

An enormous literature argues that recommendation algorithms drive political polarization by creating "filter bubbles" and "rabbit holes." Using four experiments with nearly 9,000 participants, we show that manipulating algorithmic recommendations to create these conditions has limited effects on opinions. Our experiments employ a custom-built video platform with a naturalistic, YouTube-like interface presenting real YouTube videos and recommendations. We experimentally manipulate YouTube's actual recommendation algorithm to simulate "filter bubbles" and "rabbit holes" by presenting ideologically balanced and slanted choices. Our design allows us to intervene in a feedback loop that has confounded the study of algorithmic polarization - the complex interplay between supply of recommendations and user demand for content - to examine downstream effects on policy attitudes. We use over 130,000 experimentally manipulated recommendations and 31,000 platform interactions to estimate how recommendation algorithms alter users' media consumption decisions and, indirectly, their political attitudes. Our results cast doubt on widely circulating theories of algorithmic polarization by showing that even heavy-handed (although short-term) perturbations of real-world recommendations have limited causal effects on policy attitudes. Given our inability to detect consistent evidence for algorithmic effects, we argue the burden of proof for claims about algorithm-induced polarization has shifted. Our methodology, which captures and modifies the output of real-world recommendation algorithms, offers a path forward for future investigations of black-box artificial intelligence systems. Our findings reveal practical limits to effect sizes that are feasibly detectable in academic experiments.

@article{Liu_2025, title={Short-term exposure to filter-bubble recommendation systems has limited polarization effects: Naturalistic experiments on YouTube}, volume={122}, ISSN={1091-6490}, url={https://doi.org/10.1073/pnas.2318127122}, DOI={10.1073/pnas.2318127122}, number={8}, journal={Proceedings of the National Academy of Sciences}, publisher={National Academy of Sciences}, author={Liu, Naijia and Hu, Xinlan Emily and Savas, Yasemin and Baum, Matthew A. and Berinsky, Adam J. and Chaney, Allison J. B. and Lucas, Christopher and Mariman, Rei and de Benedictis-Kessner, Justin and Guess, Andrew M. and Knox, Dean and Stewart, Brandon M.}, year={2025}, month=Feb }

Abstract

Analysts often seek to compare representations in high-dimensional space, e.g., embedding vectors of the same word across groups. We show that the distance measures calculated in such cases can exhibit considerable statistical bias, that stems from uncertainty in the estimation of the elements of those vectors. This problem applies to Euclidean distance, cosine similarity, and other similar measures. After illustrating the severity of this problem for text-as-data applications, we provide and validate a bias correction for the squared Euclidean distance. This same correction also substantially reduces bias in ordinary Euclidean distance and cosine similarity estimates, but corrections for these measures are not quite unbiased and are (non-intuitively) bimodal when distances are close to zero. The estimators require obtaining the variance of the latent positions. We (will) implement the estimator in free software, and we offer recommendations for related work.

@article{Green_2025, title={Measuring Distances in High Dimensional Spaces: Why Average Group Vector Comparisons Exhibit Bias, And What to Do about it}, volume={33}, ISSN={1476-4989}, url={https://doi.org/10.1017/pan.2024.22}, DOI={10.1017/pan.2024.22}, number={3}, journal={Political Analysis}, publisher={Cambridge University Press (CUP)}, author={Green, Breanna and Hobbs, William and Avila, Sofia and Rodriguez, Pedro L. and Spirling, Arthur and Stewart, Brandon M.}, year={2025}, month=Jan, pages={266–273} }

Abstract

We introduce AutoPersuade, a three-part framework for constructing persuasive messages. First, we curate a large dataset of arguments with human evaluations. Next, we develop a novel topic model to identify argument features that influence persuasiveness. Finally, we use this model to predict the effectiveness of new arguments and assess the causal impact of different components to provide explanations. We validate AutoPersuade through an experimental study on arguments for veganism, demonstrating its effectiveness with human studies and out-of-sample predictions.

@inproceedings{Saenger_2024, title={AutoPersuade: A Framework for Evaluating and Explaining Persuasive Arguments}, url={https://doi.org/10.18653/v1/2024.emnlp-main.913}, DOI={10.18653/v1/2024.emnlp-main.913}, booktitle={Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing}, publisher={Association for Computational Linguistics}, author={Saenger, Till Raphael and Hinck, Musashi and Grimmer, Justin and Stewart, Brandon M.}, year={2024}, pages={16325–16342} }

Abstract

The boardgame Diplomacy is a challenging setting for communicative and cooperative artificial intelligence. The most prominent communicative Diplomacy AI, Cicero, has excellent strategic abilities, exceeding human players. However, the best Diplomacy players master communication, not just tactics, which is why the game has received attention as an AI challenge. This work seeks to understand the degree to which Cicero succeeds at communication. First, we annotate in-game communication with abstract meaning representation to separate in-game tactics from general language. Second, we run two dozen games with humans and Cicero, totaling over 200 human-player hours of competition. While AI can consistently outplay human players, AI-Human communication is still limited because of AI's difficulty with deception and persuasion. This shows that Cicero relies on strategy and has not yet reached the full promise of communicative and cooperative AI.

@inproceedings{Wongkamjan_2024, title={More Victories, Less Cooperation: Assessing Cicero’s Diplomacy Play}, url={https://doi.org/10.18653/v1/2024.acl-long.672}, DOI={10.18653/v1/2024.acl-long.672}, booktitle={Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, publisher={Association for Computational Linguistics}, author={Wongkamjan, Wichayaporn and Gu, Feng and Wang, Yanze and Hermjakob, Ulf and May, Jonathan and Stewart, Brandon and Kummerfeld, Jonathan and Peskoff, Denis and Boyd-Graber, Jordan}, year={2024}, pages={12423–12441} }

Abstract

Machine learning (ML) methods are proliferating in scientific research. However, the adoption of these methods has been accompanied by failures of validity, reproducibility, and generalizability. These failures can hinder scientific progress, lead to false consensus around invalid claims, and undermine the credibility of ML-based science. ML methods are often applied and fail in similar ways across disciplines. Motivated by this observation, our goal is to provide clear recommendations for conducting and reporting ML-based science. Drawing from an extensive review of past literature, we present the REFORMS checklist (recommendations for machine-learning-based science). It consists of 32 questions and a paired set of guidelines. REFORMS was developed on the basis of a consensus of 19 researchers across computer science, data science, mathematics, social sciences, and biomedical sciences. REFORMS can serve as a resource for researchers when designing and implementing a study, for referees when reviewing papers, and for journals when enforcing standards for transparency and reproducibility.

@article{Kapoor_2024, title={REFORMS: Consensus-based Recommendations for Machine-learning-based Science}, volume={10}, ISSN={2375-2548}, url={https://doi.org/10.1126/sciadv.adk3452}, DOI={10.1126/sciadv.adk3452}, number={18}, journal={Science Advances}, publisher={American Association for the Advancement of Science (AAAS)}, author={Kapoor, Sayash and Cantrell, Emily M. and Peng, Kenny and Pham, Thanh Hien and Bail, Christopher A. and Gundersen, Odd Erik and Hofman, Jake M. and Hullman, Jessica and Lones, Michael A. and Malik, Momin M. and Nanayakkara, Priyanka and Poldrack, Russell A. and Raji, Inioluwa Deborah and Roberts, Michael and Salganik, Matthew J. and Serra-Garcia, Marta and Stewart, Brandon M. and Vandewiele, Gilles and Narayanan, Arvind}, year={2024}, month=May }

Abstract

Markets and policymakers around the world hang on the consequential monetary policy decisions made by the Federal Open Market Committee (FOMC). Publicly available textual documentation of their meetings provides insight into members’ attitudes about the economy. We use GPT-4 to quantify dissent among members on the topic of inflation. We find that transcripts and minutes reflect the diversity of member views about the macroeconomic outlook in a way that is lost or omitted from the public statements. In fact, diverging opinions that shed light upon the committee’s “true” attitudes are almost entirely omitted from the final statements. Hence, we argue that forecasting FOMC sentiment based solely on statements will not sufficiently reflect dissent among the hawks and doves.

@inproceedings{Peskoff_2023_fedspeak, title={GPT Deciphering Fedspeak: Quantifying Dissent Among Hawks and Doves}, url={https://doi.org/10.18653/v1/2023.findings-emnlp.434}, DOI={10.18653/v1/2023.findings-emnlp.434}, booktitle={Findings of the Association for Computational Linguistics: EMNLP 2023}, publisher={Association for Computational Linguistics}, author={Peskoff, Denis and Visokay, Adam and Schulhoff, Sander and Wachspress, Benjamin and Blinder, Alan and Stewart, Brandon}, year={2023}, pages={6529–6539} }

Abstract

In computational social science (CSS), researchers analyze documents to explain social and political phenomena. In most scenarios, CSS researchers first obtain labels for documents and then explain labels using interpretable regression analyses in the second step. One increasingly common way to annotate documents cheaply at scale is through large language models (LLMs). However, like other scalable ways of producing annotations, such surrogate labels are often imperfect and biased. We present a new algorithm for using imperfect annotation surrogates for downstream statistical analyses while guaranteeing statistical properties -- like asymptotic unbiasedness and proper uncertainty quantification -- which are fundamental to CSS research. We show that direct use of surrogate labels in downstream statistical analyses leads to substantial bias and invalid confidence intervals, even with high surrogate accuracy of 80--90\%. To address this, we build on debiased machine learning to propose the design-based supervised learning (DSL) estimator. DSL employs a doubly-robust procedure to combine surrogate labels with a smaller number of high-quality, gold-standard labels. Our approach guarantees valid inference for downstream statistical analyses, even when surrogates are arbitrarily biased and without requiring stringent assumptions, by controlling the probability of sampling documents for gold-standard labeling. Both our theoretical analysis and experimental results show that DSL provides valid statistical inference while achieving root mean squared errors comparable to existing alternatives that focus only on prediction without inferential guarantees.

@inproceedings{Egami_2023, series={NeurIPS 2023}, title={Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models}, url={https://doi.org/10.52202/075280-3000}, DOI={10.52202/075280-3000}, booktitle={Advances in Neural Information Processing Systems 36}, publisher={Neural Information Processing Systems Foundation, Inc. (NeurIPS)}, author={Egami, Naoki and Hinck, Musashi and Stewart, Brandon and Wei, Hanying}, year={2023}, pages={68589–68601}, collection={NeurIPS 2023} }

Abstract

Language models have recently broken into the public consciousness with the release of the wildly popular ChatGPT. Commentators have argued that language models could replace search engines, make college essays obsolete, or even write academic research papers. All of these tasks rely on accuracy of specialized information which can be difficult to assess for non-experts. Using 10 domain experts across science and culture, we provide an initial assessment of the coherence, conciseness, accuracy, and sourcing of two language models across 100 expert-written questions. While we find the results are consistently cohesive and concise, we find that they are mixed in their accuracy. These results raise questions of the role language models should play in general-purpose and expert knowledge seeking.

@inproceedings{Peskoff_2023_credible, title={Credible without Credit: Domain Experts Assess Generative Language Models}, url={https://doi.org/10.18653/v1/2023.acl-short.37}, DOI={10.18653/v1/2023.acl-short.37}, booktitle={Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)}, publisher={Association for Computational Linguistics}, author={Peskoff, Denis and Stewart, Brandon}, year={2023}, pages={427–438} }

Abstract

Social scientists commonly seek to make statements about how word use varies over circumstances—including time, partisan identity, or some other document-level covariate. For example, researchers might wish to know how Republicans and Democrats diverge in their understanding of the term “immigration.” Building on the success of pretrained language models, we introduce the à la carte on text (conText) embedding regression model for this purpose. This fast and simple method produces valid vector representations of how words are used—and thus what words “mean”—in different contexts. We show that it outperforms slower, more complicated alternatives and works well even with very few documents. The model also allows for hypothesis testing and statements about statistical significance. We demonstrate that it can be used for a broad range of important tasks, including understanding US polarization, historical legislative development, and sentiment detection. We provide open-source software for fitting the model.

@article{RODRIGUEZ_2023, title={Embedding Regression: Models for Context-Specific Description and Inference}, volume={117}, ISSN={1537-5943}, url={https://doi.org/10.1017/s0003055422001228}, DOI={10.1017/s0003055422001228}, number={4}, journal={American Political Science Review}, publisher={Cambridge University Press (CUP)}, author={RODRIGUEZ, PEDRO L. and SPIRLING, ARTHUR and STEWART, BRANDON M.}, year={2023}, month=Jan, pages={1255–1274} }

Abstract

We collect and analyze a corpus of more than 300,000 political emails sent during the 2020 US election cycle. These emails were sent by over 3000 political campaigns and organizations including federal and state level candidates as well as Political Action Committees. We find that in this corpus, manipulative tactics—techniques using some level of deception or clickbait—are the norm, not the exception. We measure six specific tactics senders use to nudge recipients to open emails. Three of these tactics—“dark patterns”—actively deceive recipients through the email user interface, for example, by formatting “from:” fields so that they create the false impression the message is a continuation of an ongoing conversation. The median active sender uses such tactics 5% of the time. The other three tactics, like sensationalistic clickbait—used by the median active sender 37% of the time—are not directly deceptive, but instead, exploit recipients’ curiosity gap and impose pressure to open emails. This can further expose recipients to deception in the email body, such as misleading claims of matching donations. Furthermore, by collecting emails from different locations in the US, we show that senders refine these tactics through A/B testing. Finally, we document disclosures of email addresses between senders in violation of privacy policies and recipients’ expectations. Cumulatively, these tactics undermine voters’ autonomy and welfare, exacting a particularly acute cost for those with low digital literacy. We offer the complete corpus of emails at https://electionemails2020.org for journalists and academics, which we hope will support future work.

@article{Mathur_2023, title={Manipulative tactics are the norm in political emails: Evidence from 300K emails from the 2020 US election cycle}, volume={10}, ISSN={2053-9517}, url={https://doi.org/10.1177/20539517221145371}, DOI={10.1177/20539517221145371}, number={1}, journal={Big Data \& Society}, publisher={SAGE Publications}, author={Mathur, Arunesh and Wang, Angelina and Schwemmer, Carsten and Hamin, Maia and Stewart, Brandon M and Narayanan, Arvind}, year={2023}, month=Jan }

Abstract

The empirical practice of using factor models to adjust for shared, unobserved confounders, Z, in observational settings with multiple treatments, A, is widespread in fields including genetics, networks, medicine, and politics. Wang and Blei (2019, WB) formalizes these procedures and develops the "deconfounder," a causal inference method using factor models of A to estimate "substitute confounders," Ẑ , then estimating treatment effects by regressing the outcome, Y, on part of A while adjusting for Ẑ . WB claim the deconfounder is unbiased when there are no single-cause confounders and Ẑ is "pinpointed." We clarify pinpointing requires each confounder to affect infinitely many treatments. We prove under these assumptions, a naïve semiparametric regression of Y on A is asymptotically unbiased. Deconfounder variants nesting this regression are therefore also asymptotically unbiased, but variants using Ẑ and subsets of causes require further untestable assumptions. We replicate every deconfounder analysis with available data and find it fails to consistently outperform naïve regression. In practice, the deconfounder produces implausible estimates in WB's case study to movie earnings: estimates suggest comic author Stan Lee's cameo appearances causally contributed $15.5 billion, most of Marvel movie revenue. We conclude neither approach is a viable substitute for careful research design in real-world applications.

@article{grimmer2023naive,
  title = {Naive Regression Requires Weaker Assumptions than Factor Models to Adjust for Multiple Cause Confounding},
  author = {Grimmer, Justin and Knox, Dean and Stewart, Brandon M.},
  journal = {Journal of Machine Learning Research},
  volume = {24},
  number = {182},
  pages = {1--70},
  year = {2023}
}

Abstract

Text as data techniques offer a great promise: the ability to inductively discover measures that are useful for testing social science theories with large collections of text. Nearly all text-based causal inferences depend on a latent representation of the text, but we show that estimating this latent representation from the data creates underacknowledged risks: we may introduce an identification problem or overfit. To address these risks, we introduce a split-sample workflow for making rigorous causal inferences with discovered measures as treatments or outcomes. We then apply it to estimate causal effects from an experiment on immigration attitudes and a study on bureaucratic responsiveness.

@article{Egami_2022, title={How to make causal inferences using texts}, volume={8}, ISSN={2375-2548}, url={https://doi.org/10.1126/sciadv.abg2652}, DOI={10.1126/sciadv.abg2652}, number={42}, journal={Science Advances}, publisher={American Association for the Advancement of Science (AAAS)}, author={Egami, Naoki and Fong, Christian J. and Grimmer, Justin and Roberts, Margaret E. and Stewart, Brandon M.}, year={2022}, month=Oct }

Abstract

A fundamental goal of scientific research is to learn about causal relationships. However, despite its critical role in the life and social sciences, causality has not had the same importance in Natural Language Processing (NLP), which has traditionally placed more emphasis on predictive tasks. This distinction is beginning to fade, with an emerging area of interdisciplinary research at the convergence of causal inference and language processing. Still, research on causality in NLP remains scattered across domains without unified definitions, benchmark datasets and clear articulations of the challenges and opportunities in the application of causal inference to the textual domain, with its unique properties. In this survey, we consolidate research across academic areas and situate it in the broader NLP landscape. We introduce the statistical challenge of estimating causal effects with text, encompassing settings where text is used as an outcome, treatment, or to address confounding. In addition, we explore potential uses of causal inference to improve the robustness, fairness, and interpretability of NLP models. We thus provide a unified overview of causal inference for the NLP community.

@article{Feder_2022, title={Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond}, volume={10}, ISSN={2307-387X}, url={https://doi.org/10.1162/tacl_a_00511}, DOI={10.1162/tacl_a_00511}, journal={Transactions of the Association for Computational Linguistics}, publisher={MIT Press}, author={Feder, Amir and Keith, Katherine A. and Manzoor, Emaad and Pryzant, Reid and Sridhar, Dhanya and Wood-Doughty, Zach and Eisenstein, Jacob and Grimmer, Justin and Reichart, Roi and Roberts, Margaret E. and Stewart, Brandon M. and Veitch, Victor and Yang, Diyi}, year={2022}, pages={1138–1158} }

Abstract

From social media posts and text messages to digital government documents and archives, researchers are bombarded with a deluge of text reflecting the social world. This textual data gives unprecedented insights into fundamental questions in the social sciences, humanities, and industry. Meanwhile new machine learning tools are rapidly transforming the way science and business are conducted. Text as Data shows how to combine new sources of data, machine learning tools, and social science research design to develop and evaluate new insights. Text as Data is organized around the core tasks in research projects using text—representation, discovery, measurement, prediction, and causal inference. The authors offer a sequential, iterative, and inductive approach to research design. Each research task is presented complete with real-world applications, example methods, and a distinct style of task-focused research. Bridging many divides—computer science and social science, the qualitative and the quantitative, and industry and academia— Text as Data is an ideal resource for anyone wanting to analyze large collections of text in an era when data is abundant and computation is cheap, but the enduring challenges of social science remain.

@book{grimmer2022text,
  title = {Text as Data: A New Framework for Machine Learning and the Social Sciences},
  author = {Grimmer, Justin and Roberts, Margaret E. and Stewart, Brandon M.},
  publisher = {Princeton University Press},
  year = {2022}
}

Abstract

Topic models, as developed in computer science, are effective tools for exploring and summarizing large document collections. When applied in social science research, however, they are commonly used for measurement, a task that requires careful validation to ensure that the model outputs actually capture the desired concept of interest. In this paper, we review current practices for topic validation in the field and show that extensive model validation is increasingly rare, or at least not systematically reported in papers and appendices. To supplement current practices, we refine an existing crowd-sourcing method by Chang and coauthors for validating topic quality and go on to create new procedures for validating conceptual labels provided by the researcher. We illustrate our method with an analysis of Facebook posts by U.S. Senators and provide software and guidance for researchers wishing to validate their own topic models. While tailored, case-specific validation exercises will always be best, we aim to improve standard practices by providing a general-purpose tool to validate topics as measures.

@article{Ying_2021, title={Topics, Concepts, and Measurement: A Crowdsourced Procedure for Validating Topics as Measures}, volume={30}, ISSN={1476-4989}, url={https://doi.org/10.1017/pan.2021.33}, DOI={10.1017/pan.2021.33}, number={4}, journal={Political Analysis}, publisher={Cambridge University Press (CUP)}, author={Ying, Luwei and Montgomery, Jacob M. and Stewart, Brandon M.}, year={2021}, month=sep, pages={570–589} }

Abstract

We make only one point in this article. Every quantitative study must be able to answer the question: what is your estimand? The estimand is the target quantity—the purpose of the statistical analysis. Much attention is already placed on how to do estimation; a similar degree of care should be given to defining the thing we are estimating. We advocate that authors state the central quantity of each analysis—the theoretical estimand—in precise terms that exist outside of any statistical model. In our framework, researchers do three things: (1) set a theoretical estimand, clearly connecting this quantity to theory; (2) link to an empirical estimand, which is informative about the theoretical estimand under some identification assumptions; and (3) learn from data. Adding precise estimands to research practice expands the space of theoretical questions, clarifies how evidence can speak to those questions, and unlocks new tools for estimation. By grounding all three steps in a precise statement of the target quantity, our framework connects statistical evidence to theory.

@article{Lundberg_2021, title={What Is Your Estimand? Defining the Target Quantity Connects Statistical Evidence to Theory}, volume={86}, ISSN={1939-8271}, url={https://doi.org/10.1177/00031224211004187}, DOI={10.1177/00031224211004187}, number={3}, journal={American Sociological Review}, publisher={SAGE Publications}, author={Lundberg, Ian and Johnson, Rebecca and Stewart, Brandon M.}, year={2021}, month=June, pages={532–565} }

Abstract

Social scientists are now in an era of data abundance, and machine learning tools are increasingly used to extract meaning from data sets both massive and small. We explain how the inclusion of machine learning in the social sciences requires us to rethink not only applications of machine learning methods but also best practices in the social sciences. In contrast to the traditional tasks for machine learning in computer science and statistics, when machine learning is applied to social scientific data, it is used to discover new concepts, measure the prevalence of those concepts, assess causal effects, and make predictions. The abundance of data and resources facilitates the move away from a deductive social science to a more sequential, interactive, and ultimately inductive approach to inference. We explain how an agnostic approach to machine learning methods focused on the social science tasks facilitates progress across a wide range of questions.

@article{Grimmer_2021, title={Machine Learning for Social Science: An Agnostic Approach}, volume={24}, ISSN={1545-1577}, url={https://doi.org/10.1146/annurev-polisci-053119-015921}, DOI={10.1146/annurev-polisci-053119-015921}, number={1}, journal={Annual Review of Political Science}, publisher={Annual Reviews}, author={Grimmer, Justin and Roberts, Margaret E. and Stewart, Brandon M.}, year={2021}, month=May, pages={395–419} }

Abstract

How predictable are life trajectories? We investigated this question with a scientific mass collaboration using the common task method; 160 teams built predictive models for six life outcomes using data from the Fragile Families and Child Wellbeing Study, a high-quality birth cohort study. Despite using a rich dataset and applying machine-learning methods optimized for prediction, the best predictions were not very accurate and were only slightly better than those from a simple benchmark model. Within each outcome, prediction error was strongly associated with the family being predicted and weakly associated with the technique used to generate the prediction. Overall, these results suggest practical limits to the predictability of life outcomes in some settings and illustrate the value of mass collaborations in the social sciences.

@article{salganik2020predictability,
  title = {Measuring the Predictability of Life Outcomes with a Scientific Mass Collaboration},
  author = {Salganik, Matthew J. and others},
  journal = {Proceedings of the National Academy of Sciences},
  volume = {117},
  number = {15},
  pages = {8398--8403},
  year = {2020},
  doi = {10.1073/pnas.1915006117}
}

Abstract

Abstract: We identify situations in which conditioning on text can address confounding in observational studies. We argue that a matching approach is particularly well-suited to this task, but existing matching methods are ill-equipped to handle high-dimensional text data. Our proposed solution is to estimate a low-dimensional summary of the text and condition on this summary via matching. We propose a method of text matching, topical inverse regression matching, that allows the analyst to match both on the topical content of confounding documents and the probability that each of these documents is treated. We validate our approach and illustrate the importance of conditioning on text to address confounding with two applications: the effect of perceptions of author gender on citation counts in the international relations literature and the effects of censorship on Chinese social media users.

@article{Roberts_2020, title={Adjusting for Confounding with Text Matching}, volume={64}, ISSN={1540-5907}, url={https://doi.org/10.1111/ajps.12526}, DOI={10.1111/ajps.12526}, number={4}, journal={American Journal of Political Science}, publisher={Wiley}, author={Roberts, Margaret E. and Stewart, Brandon M. and Nielsen, Richard A.}, year={2020}, month=July, pages={887–903} }

Abstract

Studies of economic mobility summarize the distribution of offspring incomes for each level of parent income. Mitnik and Grusky (2020) highlight that the conventional intergenerational elasticity (IGE) targets the geometric mean and propose a parametric strategy for estimating the arithmetic mean. We decompose the IGE and their proposal into two choices: (1) the summary statistic for the conditional distribution and (2) the functional form. These choices lead us to a different strategy—visualizing several quantiles of the offspring income distribution as smooth functions of parent income. Our proposal solves the problems Mitnik and Grusky highlight with geometric means, avoids the sensitivity of arithmetic means to top incomes, and provides more information than is possible with any single number. Our proposal has broader implications: the default summary (the mean) used in many regressions is sensitive to the tail of the distribution in ways that may be substantively undesirable.

@article{Lundberg_2020, title={Comment: Summarizing Income Mobility with Multiple Smooth Quantiles Instead of Parameterized Means}, volume={50}, ISSN={1467-9531}, url={https://doi.org/10.1177/0081175020931126}, DOI={10.1177/0081175020931126}, number={1}, journal={Sociological Methodology}, publisher={SAGE Publications}, author={Lundberg, Ian and Stewart, Brandon M.}, year={2020}, month=July, pages={96–111} }

Abstract

Social scientists are now in an era of data abundance, and machine learning tools are increasingly used to extract meaning from data sets both massive and small. We explain how the inclusion of machine learning in the social sciences requires us to rethink not only applications of machine learning methods but also best practices in the social sciences. In contrast to the traditional tasks for machine learning in computer science and statistics, when machine learning is applied to social scientific data, it is used to discover new concepts, measure the prevalence of those concepts, assess causal effects, and make predictions. The abundance of data and resources facilitates the move away from a deductive social science to a more sequential, interactive, and ultimately inductive approach to inference. We explain how an agnostic approach to machine learning methods focused on the social science tasks facilitates progress across a wide range of questions.

@inbook{de_Marchi_2020, title={Wrestling with Complexity in Computational Social Science: Theory, Estimation and Representation}, ISBN={9781526486387}, url={https://doi.org/10.4135/9781526486387.n20}, DOI={10.4135/9781526486387.n20}, booktitle={The SAGE Handbook of Research Methods in Political Science and International Relations}, publisher={SAGE Publications Ltd}, author={de Marchi, Scott and Stewart, Brandon M.}, year={2020}, pages={289–310} }

Abstract

This paper demonstrates how to use the R package stm for structural topic modeling. The structural topic model allows researchers to flexibly estimate a topic model that includes document-level metadata. Estimation is accomplished through a fast variational approximation. The stm package provides many useful features, including rich ways to explore topics, estimate uncertainty, and visualize quantities of interest.

@article{Roberts_2019, title={<b>stm</b>: An <i>R</i> Package for Structural Topic Models}, volume={91}, ISSN={1548-7660}, url={https://doi.org/10.18637/jss.v091.i02}, DOI={10.18637/jss.v091.i02}, number={2}, journal={Journal of Statistical Software}, publisher={Foundation for Open Access Statistic}, author={Roberts, Margaret E. and Stewart, Brandon M. and Tingley, Dustin}, year={2019} }

Abstract

When do groups—be they countries, administrations, or other organizations—more or less accurately understand the world around them and assess political choices? Some argue that group decision-making processes often fail due to biases induced by groupthink. Others argue that groups, by aggregating knowledge, are better at analyzing the foreign policy world. To advance knowledge about the intersection of politics and group decision making, this paper draws on evidence from a multiyear geopolitical forecasting tournament with thousands of participants sponsored by the US government. We find that teams outperformed individuals in making accurate geopolitical predictions, with regression discontinuity analysis demonstrating specific teamwork effects. Moreover, structural topic models show that more cooperative teams outperformed less cooperative teams. These results demonstrate that information sharing through groups, cultivating reasoning to hedge against cognitive biases, and ensuring all perspectives are heard can lead to greater success for groups at forecasting and understanding politics.

@article{Horowitz_2019, title={What Makes Foreign Policy Teams Tick: Explaining Variation in Group Performance at Geopolitical Forecasting}, volume={81}, ISSN={1468-2508}, url={https://doi.org/10.1086/704437}, DOI={10.1086/704437}, number={4}, journal={The Journal of Politics}, publisher={University of Chicago Press}, author={Horowitz, Michael and Stewart, Brandon M. and Tingley, Dustin and Bishop, Michael and Resnick Samotin, Laura and Roberts, Margaret and Chang, Welton and Mellers, Barbara and Tetlock, Philip}, year={2019}, month=Oct, pages={1388–1404} }

Abstract

Massive open online courses (MOOCs) attract diverse student bodies, and course forums could potentially be an opportunity for students with different political beliefs to engage with one another. We test whether this engagement actually takes place in two politically-themed MOOCs, on education policy and American government. We collect measures of students’ political ideology, and then observe student behavior in the course discussion boards. Contrary to the common expectation that online spaces often become echo chambers or ideological silos, we find that students in these two political courses hold diverse political beliefs, participate equitably in forum discussions, directly engage (through replies and upvotes) with students holding opposing beliefs, and converge on a shared language rather than talking past one another. Research that focuses on the civic mission of MOOCs helps ensure that open online learning engages the same breadth of purposes that higher education aspires to serve.

@article{Yeomans_2018, title={The Civic Mission of MOOCs: Engagement across Political Differences in Online Forums}, volume={28}, ISSN={1560-4292}, url={https://doi.org/10.1007/s40593-017-0161-0}, DOI={10.1007/s40593-017-0161-0}, number={4}, journal={International Journal of Artificial Intelligence in Education}, publisher={Elsevier BV}, author={Yeomans, Michael and Stewart, Brandon M. and Mavon, Kimia and Kindel, Alex and Tingley, Dustin and Reich, Justin}, year={2018}, month=Dec, pages={553–589} }

Abstract

Recommendation systems are ubiquitous and impact many domains; they have the potential to influence product consumption, individuals' perceptions of the world, and life-altering decisions. These systems are often evaluated or trained with data from users already exposed to algorithmic recommendations; this creates a pernicious feedback loop. Using simulations, we demonstrate how using data confounded in this way homogenizes user behavior without increasing utility.

@inproceedings{Chaney_2018, series={RecSys ’18}, title={How algorithmic confounding in recommendation systems increases homogeneity and decreases utility}, url={https://doi.org/10.1145/3240323.3240370}, DOI={10.1145/3240323.3240370}, booktitle={Proceedings of the 12th ACM Conference on Recommender Systems}, publisher={ACM}, author={Chaney, Allison J. B. and Stewart, Brandon M. and Engelhardt, Barbara E.}, year={2018}, month=sep, pages={224–232}, collection={RecSys ’18} }

Abstract

Motivations like domain adaptation, transfer learning, and feature learning have fueled interest in inducing embeddings for rare or unseen words, n-grams, synsets, and other textual features. This paper introduces a la carte embedding, a simple and general alternative to the usual word2vec-based approaches for building such representations that is based upon recent theoretical results for GloVe-like embeddings. Our method relies mainly on a linear transformation that is efficiently learnable using pretrained word vectors and linear regression. This transform is applicable on the fly in the future when a new text feature or rare word is encountered, even if only a single usage example is available. We introduce a new dataset showing how the a la carte method requires fewer examples of words in context to learn high-quality embeddings and we obtain state-of-the-art results on a nonce task and some unsupervised document classification tasks.

@inproceedings{Khodak_2018, title={A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors}, url={https://doi.org/10.18653/v1/p18-1002}, DOI={10.18653/v1/p18-1002}, booktitle={Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, publisher={Association for Computational Linguistics}, author={Khodak, Mikhail and Saunshi, Nikunj and Liang, Yingyu and Ma, Tengyu and Stewart, Brandon and Arora, Sanjeev}, year={2018}, pages={12–22} }

Abstract

In the past few decades new laws criminalizing certain transnational activities have proliferated: from money laundering, corruption, and insider trading to trafficking in weapons and drugs. Human trafficking is one example. We argue that criminalization of trafficking in persons has diffused in large part because of the way the issue has been framed: primarily as a problem of organized crime rather than predominantly an egregious human rights abuse. Framing human trafficking as an organized crime practice empowers states to confront cross-border human movements viewed as potentially threatening. We show that the diffusion of criminalization is explained by road networks that reflect potential vulnerabilities to the diversion of transnational crime. We interpret our results as evidence of the importance of context and issue framing, which in turn affects perceptions of vulnerability to neighbors' policy choices. In doing so, we unify diffusion studies of liberalization with the spread of prohibition regimes to explain the globalization of aspects of criminal law.

@article{Simmons_2018, title={The Global Diffusion of Law: Transnational Crime and the Case of Human Trafficking}, volume={72}, ISSN={1531-5088}, url={https://doi.org/10.1017/s0020818318000036}, DOI={10.1017/s0020818318000036}, number={2}, journal={International Organization}, publisher={Cambridge University Press (CUP)}, author={Simmons, Beth A. and Lloyd, Paulette and Stewart, Brandon M.}, year={2018}, pages={249–281} }

Abstract

We present Discourse, a tool for coding and annotating MOOC discussion forum data. Despite the centrality of discussion forums to learning in online courses, few tools are available for analyzing these discussions in a context-aware way. Discourse scaffolds the process of coding forum data by enabling multiple coders to work with large amounts of forum data. Our demonstration will enable attendees to experience, explore, and critique key features of the app.

@inproceedings{Kindel_2017, series={L@S 2017}, title={Discourse: MOOC Discussion Forum Analysis at Scale}, url={https://doi.org/10.1145/3051457.3053967}, DOI={10.1145/3051457.3053967}, booktitle={Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale}, publisher={ACM}, author={Kindel, Alexander and Yeomans, Michael and Reich, Justin and Stewart, Brandon and Tingley, Dustin}, year={2017}, month=Apr, pages={141–142}, collection={L@S 2017} }

Abstract

Statistical models of text have become increasingly popular in statistics and computer science as a method of exploring large document collections. Social scientists often want to move beyond exploration, to measurement and experimentation, and make inference about social and political processes that drive discourse and content. In this paper, we develop a model of text data that supports this type of substantive research. Our approach is to posit a hierarchical mixed membership model for analyzing topical content of documents, in which mixing weights are parameterized by observed covariates. In this model, topical prevalence and topical content are specified as a simple generalized linear model on an arbitrary number of document-level covariates, such as news source and time of release, enabling researchers to introduce elements of the experimental design that informed document collection into the model, within a generally applicable framework. We demonstrate the proposed methodology by analyzing a collection of news reports about China, where we allow the prevalence of topics to evolve over time and vary across newswire services. Our methods quantify the effect of news wire source on both the frequency and nature of topic coverage.

@article{Roberts_2016_experimentation, title={A Model of Text for Experimentation in the Social Sciences}, volume={111}, ISSN={1537-274X}, url={https://doi.org/10.1080/01621459.2016.1141684}, DOI={10.1080/01621459.2016.1141684}, number={515}, journal={Journal of the American Statistical Association}, publisher={Informa UK Limited}, author={Roberts, Margaret E. and Stewart, Brandon M. and Airoldi, Edoardo M.}, year={2016}, month=July, pages={988–1003} }

Abstract

In this study, we develop methods for computationally measuring the degree to which students engage in MOOC forums with other students holding different political beliefs. We examine a case study of a single MOOC about education policy, Saving Schools, where we obtain measures of student education policy preferences that correlate with political ideology. Contrary to assertions that online spaces often become echo chambers or ideological silos, we find that students in this case hold diverse political beliefs, participate equitably in forum discussions, directly engage (through replies and upvotes) with students holding opposing beliefs, and converge on a shared language rather than talking past one another. Research that focuses on the civic mission of MOOCs helps ensure that open online learning engages the same breadth of purposes that higher education aspires to serve.

@inproceedings{Reich_2016, series={L@S 2016}, title={The Civic Mission of MOOCs: Measuring Engagement across Political Differences in Forums}, url={https://doi.org/10.1145/2876034.2876045}, DOI={10.1145/2876034.2876045}, booktitle={Proceedings of the Third (2016) ACM Conference on Learning @ Scale}, publisher={ACM}, author={Reich, Justin and Stewart, Brandon and Mavon, Kimia and Tingley, Dustin}, year={2016}, month=Apr, pages={1–10}, collection={L@S 2016} }

Abstract

This chapter examines multimodality in topic models and related latent variable models for large-scale text analysis. Using structural topic models and a corpus of political blog posts from the 2008 U.S. presidential election, it explains why nonconvex optimization can make topic-model results sensitive to starting values. The chapter shows how these local modes arise, how they can affect substantive interpretation, and how researchers can diagnose and manage sensitivity when using topic models to analyze large, unstructured text collections.

@inbook{Roberts_2016_local_modes, title={Navigating the Local Modes of Big Data: The Case of Topic Models}, ISBN={9781316257340}, url={https://doi.org/10.1017/cbo9781316257340.004}, DOI={10.1017/cbo9781316257340.004}, booktitle={Computational Social Science}, publisher={Cambridge University Press}, author={Roberts, Margaret E. and Stewart, Brandon M. and Tingley, Dustin}, year={2016}, month=Jan, pages={51–97} }

Abstract

Dealing with the vast quantities of text that students generate in a Massive Open Online Course (MOOC) is a daunting challenge. Computational tools are needed to help instructional teams uncover themes and patterns as MOOC students write in forums, assignments, and surveys. This paper introduces to the learning analytics community the Structural Topic Model, an approach to language processing that can (1) find syntactic patterns with semantic meaning in unstructured text, (2) identify variation in those patterns across covariates, and (3) uncover archetypal texts that exemplify the documents within a topical pattern. We show examples of computationally- aided discovery and reading in three MOOC settings: mapping students’ self-reported motivations, identifying themes in discussion forums, and uncovering patterns of feedback in course evaluations.

@article{Reich_2014, title={Computer-Assisted Reading and Discovery for Student Generated Text in Massive Open Online Courses}, volume={2}, ISSN={1929-7750}, url={https://doi.org/10.18608/jla.2015.21.8}, DOI={10.18608/jla.2015.21.8}, number={1}, journal={Journal of Learning Analytics}, publisher={Society for Learning Analytics Research}, author={Reich, Justin and Tingley, Dustin and Leder-Luis, Jetson and Roberts, Margaret E. and Stewart, Brandon}, year={2014}, month=Nov }

Abstract

Recent advances in research tools for the systematic analysis of textual data are enabling exciting new research throughout the social sciences. For comparative politics scholars who are often interested in non-English and possibly multilingual textual datasets, these advances may be difficult to access. This paper discusses practical issues that arise in the processing, management, translation and analysis of textual data with a particular focus on how procedures differ across languages. These procedures are combined in two applied examples of automated text analysis using the recently introduced Structural Topic Model. We also show how the model can be used to analyze data that has been translated into a single language via machine translation tools. All the methods we describe here are implemented in open-source software packages available from the authors.

@article{Lucas_2015, title={Computer-Assisted Text Analysis for Comparative Politics}, volume={23}, ISSN={1476-4989}, url={https://doi.org/10.1093/pan/mpu019}, DOI={10.1093/pan/mpu019}, number={2}, journal={Political Analysis}, publisher={Cambridge University Press (CUP)}, author={Lucas, Christopher and Nielsen, Richard A. and Roberts, Margaret E. and Stewart, Brandon M. and Storer, Alex and Tingley, Dustin}, year={2015}, pages={254–277} }

Abstract

In political science, research using computer-assisted text analysis techniques has exploded in the last fifteen years. This scholarship spans work studying political ideology,1 congressional speech,2 representational style,3 American foreign policy,4 climate change attitudes,5 media,6 Islamic clerics,7 and treaty making,8 to name but a few. As these examples illustrate, computer-assisted text analysis—a prime example of mixed-methods research—allows gaining new insights from long-familiar political texts, like parliamentary debates, and altogether enables the analysis to new forms of political communication, such as those happening on social media. While the new methods greatly facilitate the analysis of many aspects of texts and hence allow for content analysis on an unprecedented scale, they also challenge traditional approaches to research transparency and replication.9 Specific challenges range from new forms of data pre-processing and cleaning, to terms of service for websites, which may explicitly prohibit the redistribution of their content. The Statement on Data Access and Research Transparency10 provides only very general guidance regarding the kind of transparency positivist empirical researchers should provide. In this paper, we consider the application of these general guidelines to the specific context of computer-assisted text analysis to suggest what transparency demands of scholars using such methods. We explore the implications of computer-assisted text analysis for data transparency by tracking the three main stages of a research project involving text as data: (1) acquisition, where the researcher decides what her corpus of texts will consist of; (2) analysis, to obtain inferences about the research question of interest using the texts; and (3) ex post access, where the researcher provides the data and/or other information to allow the verification of her results. To be transparent, we must document and account for decisions made at each stage in the research project. Transparency not only plays an essential role in replication11 but it also helps to communicate the essential procedures of new methods to the broader research community. Thus transparency also plays a didactic role and makes results more interpretable. Many transparency issues are not unique to text analysis. There are aspects of acquisition (e.g., random selection), analysis (e.g., outlining model assumptions), and access (e.g., providing replication code) that are important regardless of what is being studied and the method used to study it. These general issues, as well as a discussion of issues specific to traditional qualitative textual analysis, are outside of our purview. Instead, we focus here on those issues that are uniquely important for transparency in the context of computer-assisted text analysis.

@article{romney2015plain,
  title = {Plain Text: Transparency in the Acquisition, Analysis, and Access Stages of the Computer-Assisted Analysis of Texts},
  author = {Romney, David and Stewart, Brandon M. and Tingley, Dustin},
  journal = {Qualitative and Multi-Method Research},
  volume = {13},
  number = {1},
  pages = {32--37},
  year = {2015}
}

Abstract

Content analysis, a widely-applied social science research method, is increasingly being supplemented by topic modeling. However, while the discourse on content analysis centers heavily on reproducibility, computer scientists often focus more on scalability and less on coding reliability, leading to growing skepticism on the usefulness of topic models for automated content analysis. In response, we introduce TopicCheck, an interactive tool for assessing topic model stability. Our contributions are threefold. First, from established guidelines on reproducible content analysis, we distill a set of design requirements on how to computationally assess the stability of an automated coding process. Second, we devise an interactive alignment algorithm for matching latent topics from multiple models, and enable sensitivity evaluation across a large number of models. Finally, we demonstrate that our tool enables social scientists to gain novel insights into three active research questions.

@inproceedings{Chuang_2015, title={TopicCheck: Interactive Alignment for Assessing Topic Model Stability}, url={https://doi.org/10.3115/v1/n15-1018}, DOI={10.3115/v1/n15-1018}, booktitle={Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies}, publisher={Association for Computational Linguistics}, author={Chuang, Jason and Roberts, Margaret E. and Stewart, Brandon M. and Weiss, Rebecca and Tingley, Dustin and Grimmer, Justin and Heer, Jeffrey}, year={2015}, pages={175–184} }

Abstract

Content analysis, a labor-intensive but widely-applied research method, is increasingly being supplemented by computational techniques such as statistical topic modeling. However, while the discourse on content analysis centers heavily on reproducibility, computer scientists often focus more on increasing the scale of analysis and less on establishing the reliability of analysis results. The gap between user needs and available tools leads to justified skepticism, and limits the adoption and effective use of computational approaches. We argue that enabling human-in-the-loop machine learning requires establishing users’ trust in computer-assisted analysis. To this aim, we introduce our ongoing work on analysis tools for interac- tively exploring the space of available topic models. To aid tool development, we propose two studies to examine how a computer-aided workflow affects the uncovered codes, and how machine-generated codes impact analysis outcome. We present our prototypes and findings currently under submission.

@inproceedings{chuang2014computer,
  title = {Computer-Assisted Content Analysis: Topic Models for Exploring Multiple Subjective Interpretations},
  author = {Chuang, Jason and Wilkerson, John D. and Weiss, Rebecca and Tingley, Dustin and Stewart, Brandon M. and Roberts, Margaret E. and Poursabzi-Sangdeh, Forough and Grimmer, Justin and Findlater, Leah and Boyd-Graber, Jordan and Heer, Jeffrey},
  booktitle = {Advances in Neural Information Processing Systems Workshop on Human-Propelled Machine Learning},
  year = {2014}
}

Abstract

In this paper I present a general framework for regression in the presence of complex dependence structures between units such as in time-series cross-sectional data, relational/network data, and spatial data. These types of data are challenging for standard multilevel models because they involve multiple types of structure (e.g. temporal effects and cross-sectional effects) which are interactive. I show that interactive latent factor models provide a powerful modeling alternative that can address a wide range of data types. Although related models have previously been proposed in several different fields, inference is typically cumbersome and slow. I introduce a class of fast variational inference algorithms that allow for models to be fit quickly and accurately.

@misc{stewart2014latent,
  title = {Latent Factor Regressions for the Social Sciences},
  author = {Stewart, Brandon M.},
  year = {2014}
}

Abstract

This vignette introduces the lbfgs package for R, which consists of a wrapper built around the libLBFGS optimization library written by Naoaki Okazaki. The lbfgs package implements both the Limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) and the Orthant-Wise Limited-memory Quasi-Newton (OWL-QN) optimization algorithms. The L-BFGS algorithm solves the problem of minimizing an objective, given its gradient, by iteratively computing approximations of the inverse Hessian matrix. The OWL-QN algorithm finds the optimum of an objective plus the L1 norm of the problem’s parameters. The package offers a fast and memory-efficient implementation of these optimization routines, which is particularly suited for high-dimensional problems. The lbfgs package compares favorably with other optimization packages for R in microbenchmark tests.

@manual{coppola2014lbfgs,
  title = {lbfgs: Efficient L-BFGS and OWL-QN Optimization in R},
  author = {Coppola, Antonio and Stewart, Brandon M.},
  year = {2014},
  note = {R package vignette}
}

Abstract

Collection and especially analysis of open-ended survey responses are relatively rare in the discipline and when conducted are almost exclusively done through human coding. We present an alternative, semi-automated approach, the structural topic model (STM) (Roberts, Stewart, and Airoldi 2013; Roberts, Stewart, Tingley, Lucas, Leder-Luis, Gadarian, Albertson, and Rand 2013), that draws on recent developments in machine learning based analysis of textual data. A crucial contribution of the method is that it incorporates information about the document, such as the author’s gender, political affiliation, and treatment assignment (if an experimental study). This article focuses on how the STM is helpful for survey researchers and experimentalists. The STM makes analyzing open-ended responses easier, more revealing, and capable of being used to estimate treatment effects. We illustrate these innovations with analysis of text from surveys and experiments.

@article{Roberts_2014, title={Structural Topic Models for Open‐Ended Survey Responses}, volume={58}, ISSN={1540-5907}, url={https://doi.org/10.1111/ajps.12103}, DOI={10.1111/ajps.12103}, number={4}, journal={American Journal of Political Science}, publisher={Wiley}, author={Roberts, Margaret E. and Stewart, Brandon M. and Tingley, Dustin and Lucas, Christopher and Leder‐Luis, Jetson and Gadarian, Shana Kushner and Albertson, Bethany and Rand, David G.}, year={2014}, month=Mar, pages={1064–1082} }

Abstract

We describe a new probabilistic model for extracting events between major political actors from news corpora. Our unsupervised model brings together familiar components in natural language processing (like parsers and topic models) with contextual political information— temporal and dyad dependence—to infer latent event classes. We quantitatively evaluate the model’s performance on political science benchmarks: recovering expert-assigned event class valences, and detecting real-world conflict. We also conduct a small case study based on our model’s inferences. A supplementary appendix, and replication software/data are available online, at: http://brenocon.com/irevents

@inproceedings{oconnor-etal-2013-learning,
    title = "Learning to Extract International Relations from Political Context",
    author = "O{'}Connor, Brendan  and
      Stewart, Brandon M.  and
      Smith, Noah A.",
    editor = "Schuetze, Hinrich  and
      Fung, Pascale  and
      Poesio, Massimo",
    booktitle = "Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = aug,
    year = "2013",
    address = "Sofia, Bulgaria",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/P13-1108/",
    pages = "1094--1104"
}

Abstract

In examining the diffusion of social and political phenomena like regime transition, conflict, and policy change, scholars routinely make choices about how proximity is defined and which neighbors should be considered more important than others. Since each specification offers an alternative view of the networks through which diffusion can take place, one’s decision can exert a significant influence on the magnitude and scope of estimated diffusion effects. This problem is widely recognized, but is rarely the subject of direct analysis. In international relations research, connectivity choices are usually ad hoc, driven more by data availability than by theoretically informed decision criteria. We take a closer look at the assumptions behind these choices, and propose a more systematic method to asses the structural similarity of two or more alternative networks, and select one that most plausibly relates theory to empirics. We apply this method to the spread of democratic regime change, and offer an illustrative example of how neighbor choices might impact predictions and inferences in the case of the 2011 Arab Spring.

@article{ZHUKOV_2012, title={Choosing Your Neighbors: Networks of Diffusion in International Relations<sup>1</sup>: <i>Choosing Your Neighbors</i>}, volume={57}, ISSN={0020-8833}, url={https://doi.org/10.1111/isqu.12008}, DOI={10.1111/isqu.12008}, number={2}, journal={International Studies Quarterly}, publisher={Oxford University Press (OUP)}, author={ZHUKOV, YURI M. and STEWART, BRANDON M.}, year={2012}, month=Nov, pages={271–287} }

Abstract

This supplemental appendix provides additional methodological details, additional results and visualizations of models in the main paper. Results are organized according to the dependent variable under study.

@article{Andersen_2013, title={Psychological and Physiological Responses following Repeated Peer Death}, volume={8}, ISSN={1932-6203}, url={https://doi.org/10.1371/journal.pone.0075881}, DOI={10.1371/journal.pone.0075881}, number={9}, journal={PLoS ONE}, publisher={Public Library of Science (PLoS)}, author={Andersen, Judith Pizarro and Silver, Roxane Cohen and Stewart, Brandon and Koperwas, Billie and Kirschbaum, Clemens}, editor={Laks, Jerson}, year={2013}, month=sep, pages={e75881} }

Abstract

Politics and political conflict often occur in the written and spoken word. Scholars have long recognized this, but the massive costs of analyzing even moderately sized collections of texts have prevented political scientists from using texts in their research. Here lies the promise of automated text analysis: it substantially reduces the costs of analyzing large collections of text. We provide a guide to this exciting new area of research and show how, in many instances, the methods have already obtained part of their promise. But there are pitfalls to using automated methods. Automated text methods are useful, but incorrect, models of language: they are no substitute for careful thought and close reading. Rather, automated text methods augment and amplify human reading abilities. Using the methods requires extensive validation in any one application. With these guiding principles to using automated methods, we clarify misconceptions and errors in the literature and identify open questions in the application of automated text analysis in political science. For scholars to avoid the pitfalls of automated methods, methodologists need to develop new methods specifically for how social scientists use quantitative text methods.

@article{Grimmer_2013, title={Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts}, volume={21}, ISSN={1476-4989}, url={https://doi.org/10.1093/pan/mps028}, DOI={10.1093/pan/mps028}, number={3}, journal={Political Analysis}, publisher={Cambridge University Press (CUP)}, author={Grimmer, Justin and Stewart, Brandon M.}, year={2013}, pages={267–297} }

Abstract

We develop the Structural Topic Model which provides a general way to incorporate corpus structure or document metadata into the standard topic model. Document-level covariates enter the model through a simple generalized linear model framework in the prior distributions controlling either topical prevalence or topical content. We demonstrate the model’s use in two applied problems: the analysis of open-ended responses in a survey experiment about immigration policy, and understanding differing media coverage of China’s rise.

@inproceedings{roberts2013structural,
  title = {The Structural Topic Model and Applied Social Science},
  author = {Roberts, Margaret E. and Stewart, Brandon M. and Tingley, Dustin and Airoldi, Edoardo M.},
  booktitle = {Advances in Neural Information Processing Systems Workshop on Topic Models: Computation, Application, and Evaluation},
  year = {2013}
}

Abstract

This chapter examines the globalization of legal responses to transnational crime during the broader rule-of-law wave of the late twentieth and early twenty-first centuries. It asks how market liberalization and political democratization created pressure for stronger rule-of-law institutions while also opening new opportunities for illicit cross-border activity. Focusing on learning and norm diffusion, the chapter analyzes why states adopt legal commitments against transnational crime and how international and transnational governance shape domestic rule-of-law reforms.

@inbook{Lloyd_2012, title={Combating Transnational Crime: The Role of Learning and Norm Diffusion in the Current Rule of Law Wave}, ISBN={9781107666153}, url={https://doi.org/10.1017/cbo9781139175937.011}, DOI={10.1017/cbo9781139175937.011}, booktitle={Rule of Law Dynamics}, publisher={Cambridge University Press}, author={Lloyd, Paulette and Simmons, Beth A. and Stewart, Brandon M.}, year={2012}, month=June, pages={153–180} }

Abstract

Russia’s intervention in the Georgian–South Ossetian conflict has highlighted the need to rigorously examine trends in the public debate over the use of force in Russia. Approaching this debate through the prism of civil–military relations, we take advantage of recent methodological advances in automated content analysis and generate a new dataset of 8000 public statements made by Russia’s political and military leaders during the Putin period. The data show little evidence that military elites exert a restraining influence on Russian foreign and defence policy. Although more hesitant than their political counterparts to embrace an interventionist foreign policy agenda, Russian military elites are considerably more activist in considering the use of force as an instrument of foreign policy.

@article{Stewart_2009, title={Use of force and civil–military relations in Russia: an automated content analysis}, volume={20}, ISSN={1743-9558}, url={https://doi.org/10.1080/09592310902975455}, DOI={10.1080/09592310902975455}, number={2}, journal={Small Wars \& Insurgencies}, publisher={Informa UK Limited}, author={Stewart, Brandon M. and Zhukov, Yuri M.}, year={2009}, month=June, pages={319–343} }

Abstract

This study addresses the factors that lead individuals to flee their homes in search of refuge. Many argue that individuals abandon their homes in favor of an uncertain life elsewhere because of economic hardship, while others argue that threats to their lives, physical person, and liberty cause them to flee. This study engages the debate by analyzing flight patterns over time from Haiti to the United States as a function of economic and security factors. Which factors have the largest influence on Haitian-U.S. migratory patterns? Our results show that both economics and security play a role. However, our analyses are able to distinguish between the effects of different individual economic and security indicators on Haitian-U.S. migration.

@article{Shellman_2007_haitian_exodus, title={Political Persecution or Economic Deprivation? A Time-Series Analysis of Haitian Exodus, 1990—2004}, volume={24}, ISSN={1549-9219}, url={https://doi.org/10.1080/07388940701257523}, DOI={10.1080/07388940701257523}, number={2}, journal={Conflict Management and Peace Science}, publisher={SAGE Publications}, author={Shellman, Stephen M. and Stewart, Brandon M.}, year={2007}, month=Apr, pages={121–137} }

Abstract

This study predicts forced migration events by predicting the civil violence, poor economic conditions, and foreign interventions known to cause individuals to flee their homes in search of refuge. If we can predict forced migration, policy-makers can better plan for humanitarian crises. While the study is limited to predicting Haitian flight to the United States, its strength is its ability to predict weekly flows as opposed to annual flows, providing a greater level of predictive detail than its ‘country-year’ counterparts. We focus on Haiti given that it exhibits most, if not all, of the independent variables included in theories and models of forced migration. Within our temporal domain (1994–2004), Haiti experienced economic instability, low intensity civil conflict, state repression, rebel dissent, and foreign intervention and influence. Given the model’s performance, the study calls for the collection of disaggregated data in additional countries to provide more precise and useful early-warning models of forced migrant events.

@article{Shellman_2007_forced_migration, title={Predicting Risk Factors Associated with Forced Migration: An Early Warning Model of Haitian Flight}, volume={9}, ISSN={1743-968X}, url={https://doi.org/10.1080/13698240701207344}, DOI={10.1080/13698240701207344}, number={2}, journal={Civil Wars}, publisher={Informa UK Limited}, author={Shellman, Stephen M. and Stewart, Brandon M.}, year={2007}, month=June, pages={174–199} }

Abstract

This paper examines the effects of source bias on statistical inferences drawn from event data analyses. Most event data projects use a single source to code events. For example most of the early Kansas Event Data System (KEDS) datasets code only Reuters and Agence France Presse (AFP) reports. One of the goals of Project Civil Strife (PCS) –a new internal conflict-cooperation event data project– is to code event data from several news sources to garner the most extensive coverage of events and control for bias often found in a single source. Herein, we examine the effects that source bias has on the inferences we draw from statistical time-series models. In this study, we examine domestic political conflict in Indonesia and Cambodia from 1980-2004 using automated content analyzed datasets collected from multiple sources (i.e. Associated Press, British Broadcasting Corporation, Japan Economic Newswire, United Press International, and Xinhua). The analyses show that we draw different inferences across sources, especially when we disaggregate domestic political groups. We then combine our sources together and eliminate duplicate events to create a multi-source dataset and compare the results to the single-source models. We conclude that there are important differences in the inferences drawn dependent upon source use. Therefore, researchers should (1) check their results across multiple sources and/or (2) analyze multi-source data to test hypotheses when possible.

@techreport{reeves2006fair,
  title = {Fair \& Balanced or Fit to Print? The Effects of Media Sources on Statistical Inferences},
  author = {Reeves, Andrew M. and Shellman, Stephen M. and Stewart, Brandon M.},
  institution = {University of Georgia Occasional Paper Series},
  year = {2006}
}