Psychology graduate student Soo Ryu and Weinberg Institute Director Rick Lewis will present on transformers as integrative psycholinguistic models.

ABSTRACT

Transformers as integrative psycholinguistic models that combine expectation-based and memory-based accounts

Transformers are neural network attention-based architectures that represent the state of the art in natural language processing—completely transforming the field in less than two years, and enabling qualitative advances across multiple NL tasks. In this talk, we will describe an unexpected dividend for psycholinguistics. We show that Transformers provide a theoretical integration of two prominent classes of theories in sentence processing: expectation-based (surprisal) and memory-based (specifically, cue-based retrieval) theories. We first explain how and why Transformers can serve this integrative theoretical role, providing visualizations and analyses that show the learned internal attention-patterns in Transformers correspond to memory retrieval patterns expected in cue-based interference models of parsing. We then show that Transformers provide accounts of several interesting sentence processing phenomena that have previously resisted theoretically coherent surprisal or memory-based accounts. We conclude with a novel finding of attention-derived entropy effects in a large scale eye-tracking corpus.

For Zoom access information, please email
cogsci-seminar-requests@umich.edu