Keywords: Challenges of comparative research and International Survey Projects, cross-cultural concerns in data collection and measurement issues

Mode choice in survey design can influence who is able to participate in a survey (coverage), who chooses to participate (nonresponse), and how respondents answer questions (measurement), meaning that the structure of errors affecting an estimate varies as a function of how the data were collected. The use of a combination of modes can, therefore, hinder analysts’ interpretation of differences between subgroups, to the extent that these groups are differentially represented in the response samples of the different modes. In 3MC surveys where the use of multiple modes is envisaged, comparability across groups may additionally be affected by differences in how the questions are formulated (e.g. due to translation errors or deliberate adaptations), as well as other methodological differences in survey implementation. From a total survey error perspective, it makes sense to ask whether the impact of mode effects is greater or smaller than the impact of other factors that vary between groups to affect measurement equivalence. In this paper, we present the results of an analysis of measurement invariance across modes, languages, countries, and variant question formulations using data collected as part of a methodological experiment conducted in the European Social Survey in Germany, Switzerland and France. In tests of measurement invariance and latent mean comparisons, we find translation errors (differences between countries and languages) to be a bigger source of non-invariance than mode. The findings suggest that 3MC survey designers concerned about the consequences of mixing modes should consider the risks to data quality alongside existing sources of survey error, and factor these considerations into decisions about optimal resource allocation.