« August 2006 | Main | October 2006 »

September 18, 2006

Making smile sheets work

Reaction evaluations (often called "smile sheets") are the least informative of the four evaluation types I discuss in Step 9 of Creating the One-Shot Library Workshop. But let's face it - when you're whipping out the one-shots, they're often the only kind there’s time to create. If that’s the case for you, here are some tips for making your smile sheets more valuable:

Get beyond “how’d I do??
I’ve seen a lot of smile sheets that are all about the instructor. Instructor-centered design is “so yesterday,? – remember? Try focusing the questions on the learner, not the instructor, to get to what needs to be changed to make the learning better. The following point takes this tip a step further:

Probe for the perceived impact
There is research that shows that people’s rating of relevance has a higher correlation with learning than their rating of learning.

Neil Rackman, in an article in Training Magazine has some interesting things to say about this:

“If people are having a good time, they will very often perceive their learning to be more than it actually is. But high enjoyment is not necessarily related to high learning. If you have a trainer who tells 100 war stories and is very entertaining, that instructor can end up getting tremendous ratings for 'perceived learning,' but two hours later, trainees can't remember a single thing that came out of the session. The danger is that we have some great entertainers in the training ranks who may rate very high on enjoyment but are so busy entertaining the class in order to get good ratings that they're delivering pitiful learning."

The take-away? Ask how relevant the learners thought the workshop was by using a scale from “not at all relevant to me and my [job/course/or other]? to “very relevant.? Skip the questions about how much the learners thought they learned.

Get specific
If you use the same smile sheet for all the workshops your library gives watch out for the time when instructors and designers start to get dismissive - or even just plain bored - with the form and start ignoring the results. If this starts to happen, spend a little time customizing the smile sheet for each workshop. Go back to your Needs Assessment (Step 1) and Learner Analysis (Step 2) to see if there are any potential evaluation questions that would be more specific for your particular workshop. This should help reinvigorate you and others to get back to taking your feedback to heart.

Encourage participation
How often do you practically have to beg to get your learners to stay and fill out the evaluation? Probably often. Jim Kirtpatrick suggests that you take some time to tell the learners why the questions where chosen and what you’ve done with past feedback to improve your workshop. Encourage them to be totally honest with you so that you can provide future learners with a far better workshop.

And finally, recognizing that level 1 “smile sheets? are a limited way to get good feedback, you might want to step it up a notch and use this last tip:

Try a mini focus group
If you have the time – and can gather some $$ or other incentives – see if you can entice some of the learners to stay after the workshop to have a short discussion about the session. When I’ve done this in the past I’ve had great luck by going through each module of the workshop, asking what worked and what didn’t work for each. Usually this type of quick-and-dirty evaluation is worth many times the results I get on my smile sheets.

September 4, 2006

Other resources list now populated

I finally took some time to populate the part of my book site called "Other Resources."

I tried to be reductive and hold back from adding everything that might be useful so this is just a slim "need to know" list. Please take a look and feel free to add other resources in the comments area.