The Story of Subject Naught: A Cautionary but Optimistic Tale of Internet Survey Research
Konstan, J.A., Rosser, B.R.S., Ross, M.W., Stanton, J., & Edwards, W.M. (2005). The story of subject naught: A cautionary but optimistic tale of Internet survey research. Journal of Computer-Mediated Communication, 10(2), article 11. http://jcmc.indiana.edu/vol10/issue2/konstan.html
This article documents the verification process of The Men’s Internet Study (MINTS). Subjects were recruited through a banner ad on gay.com and offered $20 to complete an online survey. Manual validation was completed for residency, IP addresses, payment addresses, ZIP codes, age, birth dates, and email addresses. Duplicate surveys were weeded out by checking for duplicate IP addresses, email addresses, names, payment info, and e-payment receipts. Finally, surveys were checked for suspiciously quick completion times. These checks led to identification of 119 repeat surveys, 65 of which were completed by “Subject Naught”.
Four lessons were learned about validity threats in web-based survey research:
- Validity checking is absolutely essential when conducting Internet studies.
- Although automated testing can flag suspicious survey completion patterns, manual review is essential and the final decision to exclude should be a human one, not an automated one.
- Use of a rigorous validation protocol provides greater confidence in the study sample.
- Web-based survey research is still highly worthwhile, and can achieve high validity.