University of Minnesota Extension
http://www.extension.umn.edu/
612-624-1222
Menu Menu

Extension > Youth Development Insight > Do outcome evaluations put young people down?

Do outcome evaluations put young people down?

9 Comments

Pamela-Nippolt.jpgIn the winter issue of New Directions for Evaluation, Sarah Zeller-Berkman, director of the Beacons National Strategy Initiative, argues that youth development evaluations reinforce the "status quo" for young people in the United States. She suggests that Western society systematically excludes young people, and that the designs for outcome evaluations play a role in that exclusion. Evaluation studies are largely designed based on assumptions that youth are incomplete and "less than" adults. We do this, she contends, by focusing on individual youth outcomes and ignoring the differences that youth make when they engage with adults, in organizations, and in communities.

The author reviewed 209 evaluations of out-of-school time programs contained in the Harvard Family Research Project database and found that "only a handful of them measured community- or systems-level outcomes, while the majority measured individual gains related to academic achievement and youth-development outcomes." The one-way street for documenting that our youth development programs are making a difference is "fundamentally flawed" Zeller-Berkman concludes. This got my attention.

Youth programs benefit adults, too

A couple of years ago, our center partnered across 11 states to collect data from more 3,000 adults who volunteered in 4-H programs. In this survey (as yet unpublished), adults told us the many ways that they benefit from their involvement with youth. Several themes emerged:

  • Youth-Gardening.jpg increased self confidence for adults
  • improved social skills for adults
  • stronger community connections for adults
  • new learning of subject matter for adults and
  • access to creative outlets through the program for adults.

The adult volunteers wouldn't have gained these benefits without youth participation! To take these benefits into account, Zeller-Berkman urges us to design outcome evaluations that include:

  • the changes that result in the community from partnerships with youth
  • program designs that do this effectively, and
  • strategies written by youth that change adults and communities for the better.


Is it time to re-think the way we approach outcome evaluation in the youth development field? What would you change?

Pamela Larson Nippolt, state faculty and program leader, program evaluation


Zeller-Berkman's article is available to subscribers here.


9 Comments

Cece Gran said:

Thank you for this great post Pam! I have recently been reading some articles on the colonization of knowledge by the West throughout global history and the establishment of the positional superiority of Western knowledge by Linda Smith (1999). She contends that the term "research" is inextricably linked to European imperialism and colonization. In a section of her work she calls "They came, they saw, they measured, they renamed, and they claimed," Smith talks about how Western researchers framed their research questions in terms of "The .... (insert name of indigenous group) problem." A continuing legacy of this type of problematizing of the "other," (and for our purposes children and youth can be inserted as the other), is that many researchers, even those with the best intentions, end up framing their research in ways that put the indigenous individual or group as the locus of the problem and not the many other overarching social and structural issues at play. This colonizing legacy continues today in the way we research and "evaluate" youth, teachers, schools, youth programs, and communities. In its simplest sense, No Child Left Behind for example, can be viewed as putting the locus of the problem at the most local level. Problematizing children, families, schools, teachers, and communities instead of considering and evaluating alternative indigenous ways of thinking and knowing is something I would like to change. Cece Gran, State Faculty, Extension Center for Youth Development, University of Minnesota

Smith, L. T. (1999). Decolonizing methodologies: Research and Indigenous people. New York: Palgrave. Chapters 3 and 4

Joyce Walker, Extension Center for Youth Development said:

Very interesting, Pam! Perhaps our field has spent so much time in recent years tweaking the strategies of how, when and where to run programs that we've forgotten the more fundamental "what and why questions" underlying youth work. When evaluating a youth program, the first question is “What are we doing and why are we doing this?” Is the primary purpose of youth work developing/growing individuals or is it creating and sustaining a just, moral and democratic society? At the 2009 Conference on the History of Community and Youth Work (Durham, UK), Gabriel Eichsteller gave a fascinating talk on social pedagogy, a European term being rediscovered as youth workers and policy makers debate whether nonformal learning (ie. youth work) exists primarily to serve the individual or the society. According to Eichsteller, this educational and philosophical tension in the relationship between the individual and the collective goes back to at least the 1700’s! For more, search “social pedagogy” on Mark Smith’s incredibly useful website www.infed.org .

Pam Larson Nippolt said:

CeCe - Your comments about problematizing the "other" take Zeller-Berkman's point about missing outcomes that reflect youth as contributors to a whole 'nother level. To apply your point within the outcome measurement arena, the movement away from problem-type outcomes for youth (less tobacco use, less obesity, avoiding pregnancy etc.) toward positive, developmental outcomes could be dressing the wolf up in sheep's clothing! Any focus on "outcome" continues to "other-ize" youth. And, geesh, weren't we all youth at one point?

Pam Larson Nippolt

Pam Larson Nippolt said:

Joyce - I could spend hours on the infed.org website!

Youth work as work for individuals or for the collective....We have been working in the youth field with the "throw one star fish back into the ocean and make a difference to that one" metaphor for a very long time, which has been a powerful and effective call to action for volunteers and social change. When we break the unit down to individuals, we can wrap our minds around it, we are more in control (we think), we can begin to make a difference, we can count it! The collective gets messy and complex. We lose the attention of our stakeholders and our message gets muddy. It can seem like one big group hug without anything to show for it. Can we do better at measuring the collective?

Anonymous said:

Thanks for kicking off the blog with such an intriguing piece! Evaluations should go beyond exclusively focusing on individual youth outcomes, if the program does. This would certainly resonate with a “community youth development” approach where youth and adults work in partnership to create community change (while gaining important skills and competencies in the process). That said, while I worry about the emphasis on youth outcomes, a focus on community outcomes is even more complex! I’d vote for a focus on program quality and the practitioner expertise it takes to create it!

- Kate Walker, Extension Center for Youth Development

Anonymous said:

Kate,

The complexity of evaluating community change is, in my opinion, the reason why so many programs focus on outcomes. It seems like we can design a survey or assessment to try to get at outcomes, but investigating "bigger" issues require a sophistication in evaluation that many non-profits and youth organizations don't have. Would you agree, Pam?

Sam Grant, Extension Center for Youth Development

Pam Larson Nippolt said:

Sam - I think that measuring the change in the community is maybe not bigger if the intended outcomes for the program include that youth will benefit adults and the community through their partnerships and leadership. If the program's mission involves youth authoring and directing change through their partnerships and leadership, then we ought to find a way to document those differences.

Maybe the youth development field got ahead of itself with outcome measurement - and Zeller-Berkman is detecting that through her analysis - and we neglected to design outcome evaluations that truly reflect the intentions of the program. Maybe sticking with program quality - as a process within youth development programs - is where we need to be developmentally right now? Back at ya, Kate and Sam and anyone else.....what do you think?

Samantha Grant said:

I don't know if it's so much that we got ahead of ourselves in youth development or if we lost sight of the uniqueness of our field. The No Child Left Behind accountability movement has propelled programs to think that outcomes are the only way. I definitely see the value of outcome assessment- especially if youth are engaged in the process of defining those outcomes- but there is something almost magical about assessing the point of service, be it quality or inquiry. In the gamble of evaluation, where do you think youth development should "put its chips?"

Dale Blyth said:

Thanks Pam for initiating our blog and for starting a great conversation or thread or whatever it is. It is very thought provoking.

While I have a lot of concerns about the excessive focus on too narrow a set of youth outcomes, either very academic or very problem centered, I question Zeller-Berkman's conclusion that it is part of a larger Western effort to support the status quo. Western bias towards evidence and data, yes. Western, especially American, over emphasis on individual achievement yes. Even Western emphasis on encouraging youth to succeed in conventional ways.

Rather, I believe the emphasis on outcome measures of youth is a function, at times distorted, of our funding and value system. In order to justify investments in youth, especially public investments, we require evidence the investment will pay off. We select those things that represent benefits based to too great an extent on a set of outcomes that are costly to society (academic failure, drug use, delinquency). The social pedagogy that Joyce refers to perhaps reflects not only an emphasis on the collective but also a set of values that broadens the individual outcomes that are publically important to support in individuals.

The current debate on what we should hold youth programs accountable for - quality, academic outcomes, prevention of problems, or such things as competence, confidence, connections, engagement and contributions - is very important for the future of our field. Focusing on a broader set of indiviudal outcomes, and measuring them well, is important to the future of youth work. So is the broadening of the measurement of community, adult, and family benefits of youth programs. The later can not replace the importance of the former - but they are an important complement we can and should do better.

Dale Blyth

Leave a comment


Type the characters you see in the picture above.

  • © 2014 Regents of the University of Minnesota. All rights reserved.
  • The University of Minnesota is an equal opportunity educator and employer. Privacy