November 5, 2006

Researching Internet-Based Populations

Wright, K.B. (2005). Researching Internet-based populations: Advantages and disadvantages of online survey research, online questionnaire authoring software packages, and web survey services. Journal of Computer-Mediated Communication, 10(3), article 11.

Wright covers advantages and disadvantages of conducting online survey research. Additionally, he provides a guide to numerous online survey applications, explaining the requirements, benefits, and issues with each.

The article lists three specific advantages of online survey research:

  1. Access to unique populations: The Internet provides access to a broad spectrum of groups, including communities that only exist in cyberspace. Examples include communities devoted to specific virtual issues (cyber-stalking, online stock trading, virtual dating) and support groups for health issues that are stigmatized offline.
  2. Time: Internet-based survey research may save time for researchers, since you don't have to wrangle paper and issues associated with distributing it. Electronic data doesn't require time or funding for manual input, and the data is often more easily manipulated.
  3. Cost: obvious savings on postage, printing, data entry, transcription, recording equipment, travel, and long-distance calls.
Disadvantages discussed are:
  1. Sampling issues: Contacting participants may be an issue, depending on community support of your project. Response rates may vary drastically. Incentives may provoke repeat surveys (see Konstan, et al.) Self-selection can lead to systemic bias, since some individuals are just more likely to participate than others.
  2. Access: Gaining access to a community can be problematic, since members may consider you to be an unwanted intruder and your requests to be spam.

The remainder of the article is taken up with discussion of current web survey software packages and services.

November 4, 2006

The Story of Subject Naught: A Cautionary but Optimistic Tale of Internet Survey Research

Konstan, J.A., Rosser, B.R.S., Ross, M.W., Stanton, J., & Edwards, W.M. (2005). The story of subject naught: A cautionary but optimistic tale of Internet survey research. Journal of Computer-Mediated Communication, 10(2), article 11.

This article documents the verification process of The Men’s Internet Study (MINTS). Subjects were recruited through a banner ad on and offered $20 to complete an online survey. Manual validation was completed for residency, IP addresses, payment addresses, ZIP codes, age, birth dates, and email addresses. Duplicate surveys were weeded out by checking for duplicate IP addresses, email addresses, names, payment info, and e-payment receipts. Finally, surveys were checked for suspiciously quick completion times. These checks led to identification of 119 repeat surveys, 65 of which were completed by “Subject Naught”.

Four lessons were learned about validity threats in web-based survey research:

  1. Validity checking is absolutely essential when conducting Internet studies.
  2. Although automated testing can flag suspicious survey completion patterns, manual review is essential and the final decision to exclude should be a human one, not an automated one.
  3. Use of a rigorous validation protocol provides greater confidence in the study sample.
  4. Web-based survey research is still highly worthwhile, and can achieve high validity.