Keywords: Methodological challenges and improvements, including in the areas of sampling, measurement, survey design and survey response or non-response

The use of short message service (SMS) surveys has increased over the last decade as mobile technologies penetrate hard-to-reach populations world-wide. Although SMS surveys are generally accepted as a viable mode of survey research, it is often criticized for the technical restrictions imposed on design when compared to questionnaires of other survey modes. Both open-ended and closed-ended questions are utilized in SMS surveys for two separate reasons. One is to discover the responses that individuals give spontaneously; the other is to avoid the bias that may result from suggesting responses to individuals. However, open-ended questions also have disadvantages in comparison to close-ended, such as the need for protracted coding and larger item non-response as well as notable quality issues. While these issues have already been well researched for traditional survey questionnaires, there is insufficient research on how open-ended questions can be improved when collecting datas.

Since 2011, GeoPoll has utilized open-ended questions in SMS surveys for a variety of entities across Sub-Saharan Africa, the Middle East, Central and Southeast Asian. The inclusion of open-ended questions generally results in outliers of 10-15 percent for open-ended data across Sub-Saharan Africa, but can be much higher depending on the country where the survey is fielded, the implementation language and the complexity of the question. The longevity of these results led GeoPoll to develop two distinct but interrelated research questions. First, how can the data for open-ended questions be improved at the design phase? Second, to what extent can the quality of open-ended data provided by respondents be improved through changes in survey design?

These questions led GeoPoll to develop the following experimental hypothesis:

The inclusion of specific instructions to potential respondents before they respond to the open-ended question will amount outlier data, improving the overall quality of the survey data yielded. To test this hypothesis, GeoPoll will establish three test groups and a control group, with each group receiving its own SMS survey. The control group will receive a series of open-ended and closed-ended questions in the standard questionnaire design that GeoPoll implements. The three test groups will receive the same open-ended and closed-ended questions as the control group, but will also have: one broad instructional message before the series of open-ended questions, a specific instructional message before each open-ended question and instructions imbedded within each of the open-ended questions posed to respondents. GeoPoll plans to implement this experimental study in English during the spring of 2018 in Kenya, Nigeria and South Africa. The selection of these countries is based on English being one of the official languages, the geographical coverage of Sub-Saharan Africa and the high-level of mobile phone penetration that will allow for the collection of a nationally representative sample. The results will provide insight on whether or not additional instructions to respondents will increase data quality and potentially reduce open-ended data outliers.