Impact of response styles on inclusive measurement
Wednesday, October 27, noon to 1:10pm ET via Zoom

Speakers:
Fernanda Alvarado-Leiton
(PhD Candidate, Program in Survey and Data Science, University of Michigan)

Sunghee Lee
(Research Associate Professor, Program in Survey and Data Science, University of Michigan)

Rachel Davis
(Associate Professor, Arnold School of Public Health, University of South Carolina)

Abstracts:

Negated and Polar Opposite Items for Balanced Scale construction: An Empirical Cross-Cultural Assessment

Fernanda Alvarado-Leiton

Acquiescent Response Style (ARS) is a culturally patterned measurement error in surveys that threatens comparisons across groups with different cultural backgrounds potentially undermining inclusivity estimating attitudes and beliefs in a population. Balanced scales blend items written in different directions and are hypothesized as a method for controlling ARS. This study examined the differences in measurement properties between two types of balanced scales. The first balanced scale type included negated items, which were item reversals formed by inserting a negation, such as, “no” and “not.” The second type included polar opposite items, which used antonyms or opposite terms to reverse the item direction (e.g., “unhappy” as the opposite of “satisfied”). Participants were recruited to a Web survey and randomly assigned to (1) unbalanced, (2) negated balanced or (3) polar opposite balanced scales. Participants came from three groups with different ARS tendencies to contrast the effects of scale wording in mitigating ARS across groups and improving measurement across cultural subgroups. These groups were: Non-Hispanic White respondents, Hispanic respondents in Mexico and Hispanic respondents in the US. Both types of balanced scales outperformed unbalanced scales in convergent validity, with higher correlations between scale scores and validation variables for balanced than unbalanced scales. No statistical differences were observed between negated and polar opposite scales in fit indices of factor models, reliability measures or convergent validity for any group. These findings suggest that negated and polar opposite balanced scales are equivalent for ARS control, and that they yield adequate measurement properties for all groups included in the study.

Response Style and Measurement of Satisfaction with Life

Sunghee Lee

Satisfaction with Life (SWL), a five-item scale, is designed to assess global judgment about one’s satisfaction with life as a whole rather than specific domains of life. Popularly used by many organizations, including the World Health Organization (WHO), the Central Intelligence Agency (CIA) and the United Nations Educational, Scientific and Cultural Organization (UNESCO), it has been translated into over 30 languages. However, with its standard version using a 7-point Likert response scale, it is subject to measurement error due to response style and measurement non-comparability across groups associated with systematically different response styles. More importantly, whether and how this is addressed in research may have implications for its inclusivity. This study examines the utility of balancing the SWL scale experimentally with multiple racial/ethnic/linguistic groups in the US: Latinx dominant in English, Latinx dominant in Spanish, non- Latinx Whites, non-Latinx Blacks, non-Latinx Koreans dominant in English and non-Latinx Koreans dominant in Korean. The results suggest the benefit of balancing measurement scales but not for groups that engage in middle response style.

Reducing Acquiescent Response Style with Conversational Interviewing

Rachel Davis

Acquiescent response style (ARS), the tendency for survey respondents to select positive answers such as “Strongly Agree,” is of particular concern for increasing measurement error in surveys with populations who are more likely to acquiesce, such as U.S. Latinx respondents. This study enrolled 891 Latinx telephone survey respondents in an experiment to address two questions: (1) Does administering a questionnaire using conversational interviewing (CI) yield less ARS than standardized interviewing (SI)? (2) Do item-specific (IS) response scales reduce ARS when compared to disagree/agree (DA) response formats? No difference was observed in ARS between the DA and IS response scales. However, CI yielded significantly lower ARS than SI, likely due to the CI interviewers' efforts to clarify questions and help with response mapping. Findings from this study suggest that using CI to administer survey questions may decrease use of ARS and improve data quality among survey respondents who are more likely to engage in ARS.