Keywords: Challenges of comparative research and International Survey Projects, cross-cultural concerns in data collection and measurement issues

Due to the growing significance of international studies, the need for tools to assess the equivalence of items in such studies is pressing. Web probing, that is, implementing verbal probing techniques traditionally used in cognitive interviewing in online surveys, is a method to substitute for or to complement quantitative techniques to establish functional equivalence of item batteries in cross-cultural research. It can also been applied to single questions or items.

The web probing approach is illustrated with one question from the 2014 module on “Citizenship” of the International Social Survey Program (ISSP). In this survey, a question dealt with people’s rights in a democracy. One item was “that government authorities respect and protect the rights of minorities.” It was measured on a scale from 1 (not at all important) through 7 (very important). Support for this right was particularly strong in Spain (6.7, the same value was reached in Mexico in 2004) but weaker in Britain (5.9) as well as in Germany and the United States (both 6.1). However, whether the difference between Spain and the other countries, or the similarity observed among the other countries, means anything is unclear as long as we have no information on how respondents understand the term “minorities”.

Therefore, we implemented web probing in non-probability online surveys in Germany, Great Britain, Mexico, Spain, and the U.S. in June 2014. The 2,689 respondents were selected by a quota for age, gender, and education. In this survey, respondents were asked a specific probe immediately after having answered the closed item: “What particular minority groups did you have in mind when you were answering the question?” The open-ended answers were coded into an elaborated category schema, which represents the main categories of minorities that people had in mind, e.g., groups defined by their ethnicity, religion, social class, and gender. The paper presents the distribution of these categories across countries and their relationship to the responses to the closed item. In this way, it tries to answer the question whether the differences between the countries are real or just a consequence of different minority groups respondents in the different countries have in mind when answering the question.