These are just a few thoughts about one important issue I have had to consider in designing my secondary qualitative research methodology: quality--which is defined generally as "Having a high degree of excellence." Many things, to my mind, are involved in quality research. For example, research should be conducted in an ethical manner by capable researchers; it should begin with careful and intentional conceptualization of concepts that are then linked logically to indicators; and it should seek to impart tentative answers that are relevant and important to the broader world outside of the laboratory walls. But a small slice of this quality pie are the two issues of reliabilityand validity. With these standards, we become sure that the "knowledge" we claim to have observed/tested/discovered is not a one-time fluke and that it is actually what we claim it to be.
Ready, Aim, Fire
The very first methodology text* I ever used had a useful description of reliability and validity:
If you can think of measurement as analogous to hitting the bull's-eye on a target, you'll see that reliability looks like a "tight pattern," regardless of where it hits, since reliability is a function of consistency. Validity, on the other hand, is a function of shots being arranged around the bull's eye (p. 133).
In reality both concepts are a lot more involved than this: The actual arrows in your "measurement quiver" are of different sorts for different analytical needs. But I have always kept that description--and its corresponding concentric circle and dot scatter graphic--in my mind when I think about these important methodological issues. If "quality" in research is partly about "what do we know, and how do we know it," then with reliability we know the same thing over and over again, and with validity we (on average) know in the end what we claim to have sought to know at the start. The best situation is when we know what we set out to know, and we know that with a high degree of consistency.
In the realm of qualitative research, this overall goal is often the same, but the standards for reaching it have often been descibed differently. As with every other aspect of the Great Quantitative-Qualitative Wars, this issue has meant the deployment of many foot soldier scholars and much ink and paper--of course leaving us students in the crossfire. The issue of validity has seemed more battled over than reliability. This is perhaps because threats to reliability may be seen as random, or minor difficulties that can be corrected with more training, better concept/indicator-defining, or other relatively simple fixes. Threats to validity, however, are perhaps seen as more systematic...more fatal...more likely to involve sitting-straight-up-in-the-bed-in-the-middle-of-the-night-with-the-horror-that-your-whole-research-project-is- doomed moments.
Drawing Battle Lines (and Tables)
There seem to be two general approaches to validity (and reliability) in qualitative research. I'll discuss the second approach in a future post. But first, what seems to me to be the standard view: to redefine reliability and validity as they relate to qualitative methodology specifically, quite apart from quantitative methodology. In other words, it is typical to read disccussions of quality issues in qualitative research that assume that there are significant differences between how quality is and should be defined within the two research traditions. A fairly detailed summary is the following:
Morrow, S. L. (2005). Quality and trustworthiness in qualitative research in counseling psychology. Journal of Counseling Psychology, 52, 250-260.
The author provides a categorization of the approaches to--and, more importantly, labels for--quality qualitative work. So, for example, she groups the qualities of credibility, transferability, dependability, and confirmability in the postpositivist paradigmatic camp. These are likely familiar to most students who have taken an introductory qualitative methodology course, corresponding (in some difficult to understand new-math way) to the concepts of validity and reliability--which are taken to be quantitative research-specific concepts and inapporopriate to discussions of qualitative methods.
But Morrow describes other standards. (Or are they other words for the same or similar standards?) For example, under the interpretive-constructivism paradigm are such criteria as fairness and co-construction; and under the critical-ideological rubric is the standard of transgressiveness.
It is not that I find these categories uninformative or uninteresting. I think they are very rich for students of the philosophy of social science research. But I feel such a categorization forces most of us student-researchers to walk even deeper into the political research mine-fields: Now not only are we stopped at checkpoints and demanded at saber point to declare whether or not we are "quantitative researchers" or "qualitative researchers," but we must further state whether we are positivists, garden variety postpositivists, interpretivist-constructivists, or critical-ideologicalists.
Morrow's is not the only categorization system of quality standards in qualitative research. Creswell, one of my favorite scholars and research methodology writers, describes another approach. His books are must-haves, but the following article is a good summary of the issue I am discussing here:
Creswell, J. W., & Miller, D. L. (2000). Determining validity in qualitative inquiry. Theory Into Practice, 39, 124-130.
Overlapping Morrow's system, the authors provide a categorization of specific quality procedures employing the same three paradigms. In this case these camps are labeled "postpositivist or systematic," "constructivist," and "critical." But the authors provide a second dimension with which to group the procedures: "the lens researchers choose to validate their studies" (p. 124). The lens is simply the "viewpoint." The assumption is that, just as a researcher might fall into one of the three paradigm camps, she might also fall into one or more lens camps. These are the lens of the reseracher; lens of study participants; and lens of people external to the study (such as journal reviewers).
I read the description of lenses as meaning: "To whom do you feel you need to prove that your study is valid?" Thus, for example, if you are a postpositivist who feels the need to prove validity to and for yourself, your weapon of choice might be the procedure of triangulation--developing your themes via the convergence among several different types of information.
Again, this is informative and interesting. But again, it mainly alerts me, as an emerging scholar, to the existence of even more partially hidden explosive devices. I found myself intently scrutinizing the table on page 126 of the Creswell & Miller article, considering in turn each of the nine procedures filling out the table that is created after crossing the three paradigms with the three lenses. I felt I could cover my bases by trying to incorporate into my study at least one procedure--or, weapon in the fight for quality--from each row and column. Thus, I underlined "triangulation," "disconfirming evidence," and "researcher reflexivity"--satisfying thricely the lens of the researcher and giving me hits in all three paradigms; I also thought I might be able to build into my my study an audit trail (postpositivist/lens of external people), and "thick, rich description" (constructivist/lens of external people). That just left the participant lens row of the table un-marked-up. As a secondary study, I did not feel I could employ the member checking or researcher-participant collaboration procedures. But as I brainstormed, I thought that "prolonged engagement in the field" might still have been met by my history of participation on the larger research study from which my dissertation data are drawn...
Only later when actually trying to translate these procedures into a coherent methods section narrative in my dissertation proposal did I recognize that I had just added a couple layers of complexity to my study. Better armed may mean marching into battle better prepared. But not if I will be so heavily encumbered that I am unable to march at all.
An Army of One
Right now I have reached a point of aligning myself only with my own research study and my own learning process. I have chosen to be a methodological mercenary, casting aside the flags of specific research paradigms and lenses. I have decided to view the battle as one of knowledge versus ignorance, as opposed to positivists against postpositivists...researcher-lens-ers against journal editor-lensers... I am determined to turn "mine fields" into "mind fields"--vistas riddled with stimulating questions instead of project-deciminating missteps. In so pledging this (non)allegiance, I am leaning more towards the second approach to quality in qualitative research--one in which prolifer-label-ating distinctions between "quantitative quality" and "qualitative quality" are seen as largely unnecessary.
Stay tuned. More on that approach in an upcoming post.
*Babbie, E. (1992). The practice of social research (6th ed.). Belmont, CA: Wadsworth Publishing.