ABSTRACT:
Political activity on social media presents a data-rich window into political behavior, but the vast amount of data means that almost all content analyses of social media require a data labeling step. Automated labeling methods, however, ignore the multimodality of posted content, focusing either on text or images. State-of-the-art vision-and-language models are unusable for most political science research: they require all observations have both image and text and require computationally expensive pretraining. This paper proposes a novel vision-and-language framework calledmultimodal representations using modality translation (MARMOT). MARMOT presents three methodological contributions: it can construct representations for observations missing image or text, it replaces the computationally expensive pretraining with modality translation, and it leverages off-the-shelf transfer learning models from computer vision and natural language processing. MARMOT outperforms an ensemble text-only classifier in 16 of 20 categories in multilabel classifications of tweets reporting election incidents during the 2016 U.S. general election. Moreover, MARMOT sets a new state-of-the-art result on the Hateful Memes dataset, improving the best result in terms of area under the receiver operating characteristic curve (AUC) from 0.7141 to 0.7530 and accuracy from 0.6473 to 0.6760.

The Interdisciplinary Workshop on American Politics (IWAP) is a forum for the presentation of ongoing interdisciplinary research in American politics. Most of our presentations are given by graduate students. Each graduate student presenter is assigned a faculty and student discussant. IWAP circulates the work beforehand and the student presents it briefly at the start of the meeting. After discussant feedback, the bulk of the time is reserved for group discussion among all workshop participants. This format leads to informal yet highly interactive and productive conversations.

Email zcwalker@umich.edu for meeting link.