Is That Even Checkable? An Experimental Study in Identifying Checkable Statements in Political Discourse

Abstract

The first step in journalistic fact-checking of political discourse is identifying whether statements contain “checkable facts” (i.e., not opinions). This randomized controlled experiment investigated how different demographic factors (age, gender, education, profession, and political affiliation) are associated with the ability to discern if statements contained checkable or noncheckable facts, as well as what impact training in identifying checkable facts can have on overall outcomes. A total of 3,357 participants identified checkable and noncheckable statements from a fictional political speech extract containing eight statements. Overall, participants were able to correctly identify an average of 69% of statements. Specific demographic factors (being male, young, and university educated) were positively associated with increased performance as well as working in professions that commonly analyze data, such as research. Participating in a short training session significantly increased participants’ performance. Initial political affiliation slightly reduces the ability to assess whether statements made by named politicians contained checkable facts.