Affective polarization between supporters of the two major U.S. parties has been well documented. At the same time, evidence of issue based, ideological polarization in the American electorate is, at best, contradictory. What explains growing antagonism between ordinary Democrats and Republicans? Mason argues that socio-political sorting on several dimensions including religion, class, ideology and race all combine to produce affective polarization. In essence that theory argues that the more consistent and overlapping identities, the greater the affective polarization between Democrats and Republicans. We suspect the mechanism may be more narrow: Affective polarization is driven primarily by people’s standing schemas about the racial make-up of the two parties, and their attitudes about these groups. We predict other identities, like religion, class and ideology, are either less important or are downstream consequences of this schematic overlap between race and party. To test this theory, we combine two sources of empirical evidence. First, we use time series data from the ANES to demonstrate that the effect of racially explicit attitudes---racial resentment and support for government aid to blacks---on partisan affect has grown significantly during the last few decades. Second, we develop an original measure of implicit cognitive linkages between social groups and parties based on the IAT. Using this measure in an online M-Turk survey, we find that white respondents with racialized images of the Democratic party scored significantly higher on affective polarization. Contrary to our initial expectations, however, linking religious fundamentalists to the Republican party is also a powerful independent driver of affective polarization. Our findings have important implications for the understanding the phenomenon of affective polarization and, more generally, for the study of human cognition in politics.