July 2008 Archives

Poem for Wilder Research librarians

| No Comments

At Wilder Research, we promote evidence-based practices that support the foundation's mission:

To promote the social welfare of persons resident or located in the greater Saint Paul metropolitan area by all appropriate means... without regard to, or discrimination on account of, nationality, sex, color, religious scruples or prejudices.

Our library contains journals, books, and other mission-relevant resources. It testifies to our commitment to best practices.

We have three awesome librarians, Heather, Amanda, and Kerry, who maintain the library and provide services, such as literature searches for clients. I used the Dewey Decimal Classification System to write this poem for them on staff appreciation day:

If I was a librarian,
I wouldn't know whether
to classify you as 780 or 811.
When we need literature,
you deliver with 118.
You are our 234,
and without you,
our 003 would break down.
I'm not even being 817!

To which the librarians replied:

Dear Chris,
Thanks for your kind comments. We are happy to hear that you appreciate the 001 and 706 of the library. It has always been our 901 that we are here to help the 128 of Wilder Research in whatever way we can. However, we do note that the 119 of your visits to the library has been low lately. So come on in for some quality 302 when you have a chance.
The Librarians

Namekagon and St. Croix trip

| No Comments

We canoed the Namekagon and St. Croix rivers over Memorial Day weekend. Our friends Stephanie and John and Lori and Carl accompanied Amy and me. We covered about 45 miles over three days, a pace similar to our trip on the Buffalo River.

My favorite part of the trip, besides spending time with our friends, was the spectacular, dynamic views offered by Namekagon. The river travels circuitously between wetlands, sandy, pine-covered hills, and deciduous forest, enabling us to sneak up on wildlife and experience wonderment at each turn. We spotted black bears running through the forest (away from our canoes, thankfully), weasels (otter, pine martin, fisher), and waterfowl (green heron, blue heron, egret, ducks, mallard).

Amy, Lori, and Stephanie (the Cobber Posse)

John, Carl, and me (honorary members of the Posse)

View from our (tick-infested) campsite on the Namekagon
Namekagon Campsite Panoramic.JPG

Our route, approximately

Retrospective pretest posttest


"What is a retrospective pretest posttest?" You are not alone in wondering. I belong to the American Evaluation Association and subscribe to the EVALTALK listserv where members routinely enquire about this method. A retrospective pretest posttest is a type of survey instrument designed to reduce response shift bias.

If training participants, for example, are asked to rate their knowledge, skills, and abilities (KSAs) before and after receiving instruction, then they will often report lower KSAs after the training program. This is known as response shift bias and occurs because training participants tend to overestimate their KSAs beforehand. They become more knowledgeable during the training program and better equipped to gauge their own KSAs. Asking participants about their KSAs retrospectively, after the program, presumably reduces bias. A retrospective pretest posttest is a single survey instrument administered after a program. It contain questions that asks respondents to rate their KSAs in hindsight as well as at the conclusion of the program. Survey questions are nearly identical and evenly spit between "Thinking back before the program..." and "Now, after the program." I should note that in addition to reducing bias, a retrospective pretest posttest, administered once to a cohort, saves resources that would otherwise be spent on administering two or more surveys.

Here's a good explanation posted to EVALTALK by Susan McNamara:
"[T]here are some cases where an after-the-fact measure really works out best. For example, in parent training (especially when court-ordered), the parents come to the first meeting thinking they know everything and that they're perfect parents. If you give them a pretest now, they're going to score themselves very highly. After three months or more of training, they've come to realize that no one is perfect, that there might be better ways of doing things, and perhaps the ways (e.g., of discipline) they were using before weren't the best. Now, they might score themselves lower. If you give them a post-test, it looks like your program isn't working, because their scores are going DOWN. However, if you use a retrospective pre-test ("fill this out now, but think about the attitudes and skills you had before you entered the program") and a regular post-test, you are more likely to get a realistic change."

I would like to research this method further because the scholarly literature has not fully addressed some questions. To what degree are retrospective self-ratings biased? Is there an optimal time to administer a "pretest" in a lagged fashion as to reduce bias (i.e., when does a self-rating shift to becoming unbiased)? Is there an item format that promotes unbiased responses when asked retrospectively (e.g., matrix vs. sequential vs. sectioned)? Should evaluators measure gain scores by supplementing or substituting self-ratings with test items? One could shed light on these questions by simultaneously administering a test measuring KSAs and a survey asking respondents to rate their KSAs throughout a training program or course.

I created the heuristic model below to illustrate how tests and self-ratings might differ over time. The model uses the prediction equation from Dr. Andrew Zieffler's dissertation as a starting point. He measured continuous longitudinal growth of bivariate reasoning ability among students in an educational psychology statistics class. The quadratic growth curve represents actual ability, while the other lines (gray and dotted) represent hypothetical self-rating scenarios that could be measured in a future study of response shift bias.

If one replicated Dr. Zieffler's study with both test items and self-ratings, then the results could help evaluators and statistical educators choose an optimal data collection strategy that minimizes bias and cost. It would provide information for choosing knowledgeably between self-ratings and items that measure one's knowledge, as well as between temporal options (i.e., traditional pre/post, lagged pre/post, retrospective pre/post). Topically, self-ratings of statistical reasoning may help with instructional planning and creating reliable and valid formative assessments with fewer items.

Click on thumbnail to see larger version:

R script
curve(.89856+.32046*x-.0064827*x^2, 0, 30, xlim=c(0,30), ylim=c(0,5.5),
lwd=2, xlab="Teaching session", ylab="Bivariate reasoning")
lines(spline(continuous, n=50, method="natural"), lwd=2, col=8)
prepost=cbind(c(0,30), c(5,4.65))
points(prepost, pch=5, lty=2, lwd=2, col=8)
lines(prepost, lty=2, lwd=2, col=8)
retro=cbind(c(0,30), c(1,4.65))
points(retro, pch=16, lwd=2)
lines(retro, lwd=2, lty=3)
par(ps=9, cex.main=1)
legend("bottomright", c("Predicted mean score (Zieffler, 2006)",
"Continuous self-rating (hypothetical)",
"Before and after self-rating (hypothetical)",
"Retrospective-before and after self-rating (hypothetical) ")
,pch=c(NA,NA,5,16), col=c(1,8,8,1), lty=c(1,1,2,3), lwd=c(2,2,2,2), merge=T)
title(main="Heuristic model:
Bivariate reasoning measured at different times by different methods")