Building Extension's Public Value

Last year I presented to faculty from University of Minnesota's Research and Outreach Centers (ROCs) about communicating the public value of research, outreach and engagement scholarship for promotion and tenure purposes. Since then I've had others ask me about whether and how the public value approach can be useful in documenting scholarly accomplishments.

The guidance I came up with is pretty similar to what I wrote about here: begin with the end in mind. Basically, a scholar who aims early on for public value-level impacts and outcomes, and then evaluates and documents those outcomes, will have built a strong case for her work's public value. Because each step in creating public value involves scholarship--of research, engagement, teaching, and evaluation--the scholar who meticulously documents her contributions should, in the end, be well-positioned to defend her record.

I know, with all of the demands on outreach and engagement faculty, this is easier said than done. I know clients and community-members expect these faculty to engage in activities that are hard to classify as scholarship. But I do think that leading with the end game can help a faculty member prioritize for success.

I also wrote in this blog entry about ways that the public value approach can help close the loop between research and engagement. The research-design-engagement-evaluation loop illustrated in that entry provides a number of opportunities for a scholar focusing on engagement work--in contrast with outreach education--to document her contributions and impact. How did you contribute to (1) the research that underpins the program curriculum, (2) the program design, (3) the engagement itself, (4) the program evaluation? It seems to me that viewing your engagement program in the context of the loop can bring to mind scholarly contributions that you might not have thought to document. Perhaps it can even lead to a more complete promotion and tenure case.

Posted by Laura Kalambokidis at 5:02 PM | Permalink | Comments (0)

At last week's 2013 PILD conference, I heard about a couple of national initiatives that, once developed, should help Extension organizations share impact data that can inform public value messages. In my own comments at the conference, I supported cross-state sharing of ideas, so I was encouraged to hear about these national projects.

First, I think it was NIFA director Dr. Sonny Ramaswamy who mentioned an effort this summer to develop a national portal for impact reporting that will consolidate Extension program impact results.

Second, I believe it was ECOP chair Dr. Daryl Buchholz who highlighted ECOP's Measuring Excellence initiative. It appears that this project is meant to define and demonstrate excellence and to report impacts for Cooperative Extension as a whole. From the website: "Cooperative Extension has advanced from merely reporting inputs and outputs to documenting outcomes and impacts of its programs. However, most of these measures are tied to specific programs. They are not generally assessed or considered at the organizational level." While the "excellence" part of the website is well-developed, the pages having to do with impact are still under construction. I look forward to the work that will populate these pages with resources and guidance for Extension impact teams. Meanwhile, I have to give a shout out for the public value statements on the front page of the website!

By the way, my notes from the PILD keynote talks are a little sparse. If I am wrong about which speaker spoke about which initiative, please correct me in the comments. And if you know more about how the portal or the Measuring Excellence project can strengthen Extension's public value case, please share that, too!

(Photo credit: USDAgov on FlickR)

Posted by Laura Kalambokidis at 4:24 PM | Permalink | Comments (0)

An Extension program creates public value when its positive impact extends beyond the program participants to the greater community. Documenting that a program has created public value, therefore, requires measuring system- or community-level impacts. How often do evaluation studies measure the kinds of impacts that can be classified as public value? In an article in the most recent (April 2012) Journal of Extension, Jeffrey Workman and Scott Scheer try to answer that question. The article, titled "Evidence of Impact: Examination of Evaluation Studies Published in the Journal of Extension," examines program evaluations published in JOE to determine the levels of impact they reached. They considered two impact hierarchies. In Bennett's Hierarchy (from "Up the hierarchy," by C. Bennett in the March 1975 JOE) the highest of seven levels is "end results," which would include such community-level impacts as a stronger economy or improved environmental conditions. In the Logic Model, these kinds of changes in social, economic, civic, and environmental conditions are called "long-term outcomes." Workman and Scheer found that about 5.6 percent of the studies they surveyed reported impacts at these levels.

The authors conclude that "more higher-level evidence of impact is needed." They write that Extension's "ultimate goal is to remain relevant and of value to the public. The strongest method to demonstrate relevancy and public value is to document "true impact" (end results/long-term outcomes)."

Do the authors' findings ring true for you? Are only a small percentage of programs able to demonstrate public value-level impacts? Is it because few programs are achieving that level of impact? Or because the resources have not been available to measure programs' long-term impacts?

Posted by Laura Kalambokidis at 11:45 AM | Permalink | Comments (2)

There has been a lively discussion about communicating the public value of youth development programs on "Youth Development Insight," a blog of the University of Minnesota Extension Center for Youth Development. Community program specialist, Joanna Tzenis argues that youth programs can create public value by having society-level impacts. Examples of two such impacts are building trust among community members and youth becoming agents for change in their communities. You can check out the discussion here.

Posted by Laura Kalambokidis at 4:23 PM | Permalink | Comments (0)

University of Missouri Extension's Stay Strong, Stay Healthy program (SSSH) is a strength training program that leads older adults to feel more active, flexible, and energetic. The SSSH team has created a public value message that they use to generate awareness and support for the program. Here is the message, complete with an estimate of health-related cost savings, excerpted from the program website:

"When you support MU Extension's Stay Strong, Stay Healthy program, participants will increase their physical activity and may improve strength, balance and flexibility, resulting in reduced risk for falls, better overall health and greater independence. These health benefits decrease the likelihood of a participant entering a nursing home, which costs on average $24,455 per year in Missouri. The money saved benefits the community by keeping more discretionary income in circulation locally. It also keeps people actively, independently contributing to society longer."

Do you have a similar program in your state? How do you explain your program's public value?

Posted by Laura Kalambokidis at 4:52 PM | Permalink | Comments (0)

Looking for ways to support your public value message? Spend some time exploring the Evidence Based Living Blog, written by Cornell Cooperative Extension's Karl Pillemer, Associate Dean for Extension and Outreach, and Rhoda Meador, Associate Director of Outreach and Extension in the College of Human Ecology as well as the Associate Director of the Bronfenbrenner Life Course Center. The blog highlights research on the outcomes arising from all kinds of programs and interventions, particularly in the areas of youth development and health and wellness.As the authors say, "The blog is based on one key principle: Now more than ever, people need help separating the good scientific information from the bad. We are all about assessing the scientific evidence on human problems and looking at how to use it every day." Does that sound familiar? Does your extension program make an effort to "separate the good scientific information from the bad"?

Consider adding the Evidence Based Living Blog to your blog reader, so you can see when the authors post about the latest research or media stories on youth behavior and health. Take a stroll through the archives and read the discussions and evidence assembled therein. You may come across ideas for new research projects or findings that you can use to make the case for your own programs. If you find something that you find useful, go ahead and share that in a comment on the Cornell blog or here.

Posted by Laura Kalambokidis at 8:41 PM | Permalink | Comments (0)

Last month a group of about 30 Extension professionals from around the country participated in a train-the-trainer course for "Building Extension's Public Value." One of the participants, Jonathan Ferris of Purdue Extension, shared his ideas for a public value message for Purdue Extension's Venison Workshop. The program teaches participating hunters proper techniques for field dressing deer and safe methods for storing and preserving venison. Educators also update participants about chronic wasting disease in Indiana.

Regarding the evaluation methodology for the venison program, Jonathan reports: "For years, we only asked questions like 'did you pick up some butchering tips,' or 'did you learn something about food safety,' etc. Last year, however, we decided that since we have many return attendees, we would begin asking them if they 1) hunted or fished more as a result of attending our program (we also do fish programs), and 2) do they keep or bring home more fish and game as a result of our programs."

With affirmative responses to those evaluation questions, Jonathan and his colleagues argue that the hunter/fisherman programs create public value by generating hunting and fishing license fees for the state (provided that the program participants hunt and fish in concordance with state regulations). Moreover, wild game and fish are low in saturated fats and sodium, and are generally part of a healthy diet. Sportsmen and women who bring home more wild game and fish and incorporate it into their diets may see improvements in health. When these health improvements lead to lower public health costs, we can see that the Extension programs have generated public value.

Additionally, if the venison team can produce evidence that program participants identify and report animals that show signs of chronic wasting disease, they may be able to make a "natural resource protection" argument, as well.

Do you have hunting and fishing programs in your state? Have you tried to make a case for public funding for such programs? How do you explain the programs' public value?

Posted by Laura Kalambokidis at 4:24 PM | Permalink | Comments (0)

On May 18 in Madison, WI, I will lead a breakout session at the National Extension and Research Administrative Officers' Conference (NERAOC). I will present an overview--and the basic concepts--of the "Building Extension's Public Value" workshop, and talk about how to make a case for funding for outreach, extension, and research. If you are planning to attend the conference, please join me at the 10:15 session.

Posted by Laura Kalambokidis at 4:13 PM | Permalink | Comments (0)

What should an Extension program team have on hand to draft a public value message that secures a skeptical stakeholder's support? Here's my prescription: What's yours?

Posted by Laura Kalambokidis at 8:00 AM | Permalink | Comments (0)

Substantiating the claims that we make about Extension programs' public value is crucial to Extension's credibility. However, we don't always have enough time in a "Building Extension's Public Value" workshop to assemble the documentation (journal articles, program evaluation reports, etc.) to support the claims embedded in a newly drafted public value message. The purpose of the "Research Agenda" workshop module is to list those claims and create a plan for assembling the supporting documents, or even for conducting new program evaluations or research.

Sometimes, a workshop group is torn between wanting to draft a public value message that is persuasive--but, maybe a bit aspirational--and one that contains only claims for which the team has strong supporting evidence. I usually encourage groups to be creative and persuasive during the workshop and worry about the documentation later, but not to publicly use a public value message until they are sure it is defensible. Understandably, this guidance occasionally leads to draft public value messages that include some pretty far-fetched claims.

Cynthia Crawford, Family Financial Education Specialist and County Program Director for University of Missouri Extension in Saline County, MO, has a suggestion for helping workshop groups stay creative while not veering too far off into "aspirational" territory. Cynthia suggests telling teams drafting public value statements that they don't have to have the documentation to substantiate their claims today (during the workshop), but they do have to believe the statements are true today. Cynthia reports that this bit of direction has lead to remarkably strong--and credible--draft public value messages in short amounts of work time.

I will definitely adopt Cynthia's "you have to believe it today" guidance the next time I teach a BEPV workshop. Do you have any other suggestions for helping teams "think big" while staying grounded?

Posted by Laura Kalambokidis at 7:45 AM | Permalink | Comments (0)

While the objective of the "Building Extension's Public Value" workshop is to draft a qualitative message about a program's public value, many of our stakeholders are concerned about programs' financial impacts. For example, county commissioners and state legislators want to know how much a program will cost, and whether it's impacts will reduce strain on the county or state budget. A lot of us, therefore, are eager to quantify the impacts of Extension programs and, wherever possible, convert those impacts into dollars and cents.

Some exciting work is being done on monetizing Extension program impacts. These economic impact briefs from Texas AgriLife Extension are a strong example, and I know there are many more studies.

In future blog entries, I'll write more about ways researchers and program evaluators are quantifying and monetizing Extension program impacts. However, as persuasive as a dollars-and-cents case can be with some stakeholders, I can think of two reasons to proceed with caution as we pursue more financial and fiscal impact studies.

First, Cooperative Extension does not yet have all the resources and tools necessary to estimate the financial and fiscal benefits of all of our programs. To do a credible job, applied economists, program evaluators and others would need to devote many more hours to this effort than are currently available. Data must be collected and analyzed, models built and tested, reports written and vetted. The likely result of pressuring program teams to estimate financial impacts while providing them with inadequate resources is a collection of poor quality analyses that erode Extension's credibility.

Second, some programs' public value lends itself more readily to monetization than others. For example, a program that helps reduce a county's cost of managing its waste can make a strong, straightforward, dollars-and-cents case. On the other hand, methodologies for estimating the fiscal impact of social capital improvements are less well-developed.

Because so many of Extension's stakeholders are concerned about monetary value, I am concerned that those programs whose public value is more easily monetized will rise to the top of the priority list--not because they contribute more public value, but because their value is easier to translate into currency.

The objective of the BEPV workshop is to make strong qualitative cases for all Extension programs that create public value. I hope we can keep doing this, even while we seek the resources necessary to estimate the financial and fiscal impacts of those programs for which that is possible.

Posted by Laura Kalambokidis at 10:57 AM | Permalink | Comments (0)

Many of you use the University of Wisconsin Extension logic model to guide program development and evaluation. Below is my first attempt at mapping the elements of the logic model to a public value message.

The "short-term" or "learning outcomes" in the logic model are a means to achieving the behavior changes and outcomes contained in the public value message. These learning outcomes lead the way to public value--and we must identify and measure them--but they are not the focus of the public value message. A skeptical stakeholder is unlikely to be persuaded of a program's value be hearing that a participant learned or became aware of something. The stakeholder is concerned with what the participant actually did with that knowledge.

What I call "changes" in the public value message are called "intermediate" or "medium term outcomes" in the logic model. What I call "outcomes" are the logic model's "long-term outcomes" or changes in conditions.

It seems to me that public value typically arises from a program's long-term outcomes. In some cases, a program's logic model will already include the outcomes that a stakeholder cares about (public value). In other cases, the public value exercise will tell us which additional outcomes we need to monitor--how we should extend the logic model--in order to substantiate our public value messages.

I believe that the public value approach must work hand in hand with program evaluation: it is through good program evaluation that we are able to make credible statements about our programs' public value.

Posted by Laura Kalambokidis at 4:17 PM | Permalink | Comments (0)

In the previous entry I wrote about two steps that help close the loop between research and Extension (or engagement): conducting research and conducting program evaluation. When I presented this at Purdue's Scholarship of Engagement workshop, a participant asked how I differentiate between research and program evaluation.

For the purpose of the "closing the loop" diagram, I was thinking of research as applied, scholarly investigation, which may or may not have an intended application to an engagement or outreach program. Really, I was envisioning laboratories and experimental plots.

But I also see many kinds of program evaluation as research; certainly evaluation requires the application of research methods (e.g., cost-benefit analysis, survey data analysis, economic impact analysis). However, it may not be the kind of research that subject-matter scholars (e.g., scientists and social scientists) can publish in their own fields' professional journals. So, I think some program evaluation is research and some research is program evaluation. Am I right? Is there more to the distinction than I am making here?

Posted by Laura Kalambokidis at 5:03 PM | Permalink | Comments (0)

When I ask Extension professionals to name Extension's strengths relative to other providers of outreach education, the connection between Extension programs and university research inevitably is the the first item on the list. We build on that key strength when we deliver programs that are based on the best research, and the community's needs inform the research agenda: that is, when we close the loop between research and Extension. I focused on this relationship--substituting "engagement" for "Extension"--at the Purdue Scholarship of Engagement Workshop last week.

Here's how I think an Extension team can close that loop: They (or someone else) conduct research that leads to a discovery (knowledge creation) that could help address a condition of concern in a community (middle left box in the diagram). The team designs their Extension or engagement program with a curriculum that is based on the new knowledge, as well as existing best practices regarding program design and delivery (middle box). If the team is truly "engage" with their community partner, then the partner's needs and strengths will also inform the design of the program. The team conducts their program (middle right box) while also collecting data and observations that can be used to inform the research agenda (top box). This way, what is observed and learned "in the field" makes its way back to the lab to influence the direction of future research. The team also implements their program evaluation plan, which helps them evaluate the impact of the Extension or engagement program (lower right box). The results of the evaluation helps them improve the program design (lower middle box), so greater impact will result next time.

Where does public value come into this scheme? I can think of at least two places: First, in the design phase, the team will plan how they expect the program to create public value. What are the expected impacts and outcomes, and how do they create benefits for sstakeholders who are not the program's direct beneficiaries? Second, in the evaluation phase, team members will assess whether those expected outcomes were generated: whether public value was created.

I can think of a few ways a team can increase their success at closing the loop:

*Form a team that includes researchers, Extension educators, and program evaluators.
*Embed the program evaluation plan into program design.
*Develop and implement a plan for collecting observations and data arising from the Extension or engagement program.
*Keep up to date on relevant research developments.
*Plan for steps to take once the program ends (e.g., analyzing data and revisiting the program design).

Do you think closing the loop between research and engagement is crucial? Can you suggest ways to make it happen more systematically in Extension?

Posted by Laura Kalambokidis at 4:51 PM | Permalink | Comments (0)