Youth Development Insight

Soft skills can be hard to measure

2013-02-27T18:00:00Z

If you, like me, evaluate and study youth programs, you should know about a new resource for measuring soft skills outcomes. Soft skills -- communication, relationships and collaboration, critical thinking and decision making, and initiative and self-direction -- can be hard to measure. Youth programs can help young people to acquire these skills, which are important for working and participating in civic life.

The Forum for Youth Investment has published a reviewed report of eight tools to do this. "From soft skills to hard data" reviews eight tools that are both practical and rigorous - offering something for evaluators and program practitioners alike.

]]> The report cites the 2010 Preparing Students for College and Careers policy report that "according to teachers, parents, students and Fortune 1000 executives, the critical components of being college- and career-ready focus more on higher-order thinking and performance skills than knowledge of challenging content."

In my opinion, the concise review of eight measurement tools does four things very well;

it names outcomes that frame the niche of programs designed to build youth learning in community,
it calls on those programs to align their activities with outcomes - an underdeveloped "muscle" of the youth development field,
it lays out the measures in an easy-to-understand guide with details about reliability, validity, and costs associated with the use of the eight measures,
it issues a call to action to advance the field by designing practical studies that are also technically sound, and by improving and advancing the measurement of soft skills.

Have you read the report? Have you used any of these tools? What is your opinion?

-- Pamela Larson Nippolt, evaluation and research specialist

You are welcome to comment on this blog post. We encourage civil discourse, including spirited disagreement. We will delete comments that contain profanity, pornography or hate speech--any remarks that attack or demean people because of their sex, race, ethnic group, etc.--as well as spam.

Build your evaluation muscle to use it effectively in the program

2012-09-26T16:28:37Z

Just when you thought that your youth program was doing well to DO evaluation at all, we evaluators want you to USE it, too! What does it take to make the report, and the entire evaluation process, an integral part of a youth organizations' everyday work?

I've learned that building capacity to use evaluation does not depend on having a lot of fancy bells and whistles. My experiences in the reporting stage of evaluation work with youth-serving organizations have taught me that successful use of evaluation has little to do with slick reports and branded slide presentations. It is more about the right people coming together to roll up their sleeves around the findings and lessons.

]]> Others in the evaluation field have done some thinking about this and are sharing their experiences on evaluation use. Boris Volkov and Jean King provide a capacity-building checklist for those planning evaluations. Their checklist suggests that one of the first places to start to ensure that evaluations get used is at the top of the list -- It is critical that organizational leaders share responsibility for building the organizational "muscle" to use evaluation effectively.

Mary Arnold, an evaluator for Extension and 4-H in Oregon, recommends a four-part framework for building the capacity of Extension educators who work in youth development to use and lead evaluation in youth programs. By starting small with teaching the use of logic models and growing toward large-scale, multi-site evaluation projects, Arnold reports evidence of success in her unit's increased capacity to use, and learn from, evaluation.

Few things worth having come without some elbow grease. We can do it! What does it take for you to be engaged in an evaluation project or process? What do we need to do so that evaluation gets USED in youth organizations?

-- Pamela Larson Nippolt, associate Extension professor and program leader, program evaluation

Who benefits from 4-H volunteering? You might be surprised

2012-05-30T17:00:00Z

You might think that the sole beneficiaries of youth program volunteering would be youth. But you would be mistaken -- the value extends to the community and to the volunteers themselves.

A recent study of 4-H volunteers in the North Central United States documents the types and levels of contributions made by volunteers that benefit youth, their communities, and the volunteers themselves.

More than half a million adults across the US give their time to the 4-H program and Extension.This is a lot of "people power". To put it in context, the YMCA and the American Red Cross -- two of the largest nonprofit organizations in the country -- are each supported by similar-sized corps of volunteers.

]]> Extension 4-H Youth Development, a public organization, is a key actor in the landscape of programs that recruit volunteers to promote the positive development of youth in communities. We can quantify this by turning volunteer hours into dollar signs at the rate of just over $20 per hour. But there is more to it than that.

In this study, my colleagues and I learned, from the responses of more than 3,000 "mostly women living in rural communities" volunteers, that they:

Tend to be college-educated and to stick around for several years of service, particularly when they themselves were 4-H members in their youth.
Tend to spend as much time planning for and communicating plans with youth as they spend actually working with them.
Give more than time -- they donate money, supplies, and mileage on their cars to the 4-H program.
Need training and development as as much as they do a well run volunteer system.

We also learned that volunteers benefit from the relationship, and that their communities do, too. Volunteers told us that they directly benefit from:

Opportunities to be involved with youth learning; in other words, the privilege to partner with young people in community settings
Opportunities for personal growth, becoming better at public speaking or a specific skill
Opportunities for contributing to the 4-H mission and giving back to the organization, being part of something "big"
Becoming better connected and valued as member of their communities

The benefit they most often mentioned was that the 4-H volunteer experience contributed to their own pathway toward becoming a better person. This is both humbling and startling in a "we are all connected" sort of way. It is also incredibly difficult to quantify. This finding sheds new light on youth and adults as partners in youth programs, and their interdependence on one another in the community. Extension and 4-H are strong threads in the fabric of communities, and if we listen closely to 3,000-plus volunteers, one of those brightly-colored, extremely resilient threads is woven by volunteers in partnership with youth. Think about this.

How does your experience with volunteerism compare with the results of this study?

-- Pamela Larson Nippolt, state faculty and program leader, program evaluation

Great expectations are good predictors of science careers

2011-10-19T17:00:20Z

When young people are asked, "What kind of work do you expect to be doing when you are 30 years old?", it turns out that their responses are quite accurate predictions of their college majors.

A 2006 study of young adolescents' career expectations, led by researchers at the University of Virginia, investigated whether 13-year-olds with an expectation for a science-related career obtained science degrees at higher rates than 13-year-olds without this expectation. They do - or at least they did - in a national sample of youth studied during the years 1988 through 2000, and published in 2006.

The study factored in differences in academic achievement, academic characteristics, and demographics, and followed young people living in the U.S. over time. Young people were asked to select one employment option from a list (only one!) and their career expectations were sorted into two groups -- science-related and non-science.

]]> The science-related careers were further separated into "life" sciences and "physical/engineering" sciences. The young people who expected careers at age 30 in the sciences were nearly twice as likely to graduate with a life science college degree, and more than three times as likely to earn a physical/engineering science degree as young people who did not see themselves in science careers. While academic achievement in eighth grade math had a role in predicting later careers in physical/engineering science degrees, math scores were not a predictor for careers in the life sciences!

But expectation dominated, even in the physical sciences. "An average mathematics achiever with a science-related career expectation had a higher probability of earning a baccalaureate degree in the physical sciences or engineering than a high mathematics achiever with a non-science career expectation." In other words, academics matter but they were not the strongest predictor for future engineers.

While initiatives to encourage youth pursuit of science careers may focus attention on eighth grade algebra, these data support that there is more to a future than a good grade. We knew this, of course, but it sure helps make the case for a theory of change when data support what we know.

How can we use this knowledge as we partner with formal educators? This is an important question for our work. 4-H is asking the "expectation question" of youth in a national study in order to compare 4-H youth to youth who are not participating in 4-H. Clearly, how we plan and design nonformal science programs matters, and the stakes are very high.

-- Pamela Larson Nippolt, state faculty and program leader, program evaluation

Valor Publico: Translating youth leadership from Mexico

2011-05-02T16:00:00Z

On a trip to Mexico a couple of weeks ago as a participant in National Extension Leadership Development, I had a chance to see a community health organization that is still going strong 20 years after being founded by a group of youth. As a youth development educator, I was struck by the power of youth when they are engaged as leaders in their community. As an program evaluator, I got to thinking about factors that play into sustaining a program and its overall value to the public over time.

]]> Our hosts, a family of five, warmly oriented our group of three Extension educators, despite our limited Spanish. We quietly sat with the matriarch of the family in the courtyard under the rock "gate," a formation that crowns the nearest mountain and creates an opening to the sky. The eldest daughter Lety greeted us as she returned from the Atekokolli clinic in the village that she helped to found on communal land.

As a child, Lety was at her grandmother's side as she treated community members. Lety learned where to find the healing plants, how to harvest them, and how to use them in treatments. When she was a teen, Lety and 20 other young people in Amatlan started a project they called Atekokolli - the Nahuatl word for the conch shell used in religious ceremonies. "We got together and talked about what we could do to help the community," Lety remembers.

They agreed to create a place where traditional healing is provided to everyone who needs it. Over the years, the group grew and shrank in size and worked together through times of growth and uncertainty to realize the original plan.

Eventually, they secured funding of 35,000 pesos / US$3,000 from a competitive funding source, which allowed them to begin building on a designated site in the middle of town. The considerable time Lety spent at Atokekolli took away from the kitchen, leading to tension with her mother. Despite this and other obstacles, Lety notes that when the project sat dormant for a period of time, it was her mother and her grandmother who told her that she should see it through to completion.

Twenty years after Atekokolli 's inception, Lety leads the clinic with two colleagues. The three founders who continue the vision for the clinic prepare the herbs and plants for use, oversee the temascals (a healing ceremony similar to an American Indian sweat lodge), offer massage, and chiropractic services. Their clinic is busy and clientele draws from a large area. Lety is especially focused on diabetes prevention as obesity and less active lifestyles are taking a toll on the health of community members.

Back at my desk in Minnesota, I am reflecting on the success of this "program." In Extension, we strive to demonstrate the public value of programs for youth. The Atekokolli model lacks the typical trappings of a program, yet accomplished what we hope to accomplish with youth through programs. What did these 20 youth leaders bring to it, and how did the community that they lived in contribute to make Atekekolli so valuable to so many? What examples can we point to of similar youth-led development efforts that have staying power in US communities?

--Pamela Larson Nippolt, assistant Extension professor

Do outcome evaluations put young people down?

2011-02-02T19:21:16Z

In the winter issue of New Directions for Evaluation, Sarah Zeller-Berkman, director of the Beacons National Strategy Initiative, argues that youth development evaluations reinforce the "status quo" for young people in the United States. She suggests that Western society systematically excludes young people, and that the designs for outcome evaluations play a role in that exclusion. Evaluation studies are largely designed based on assumptions that youth are incomplete and "less than" adults. We do this, she contends, by focusing on individual youth outcomes and ignoring the differences that youth make when they engage with adults, in organizations, and in communities.

]]> The author reviewed 209 evaluations of out-of-school time programs contained in the Harvard Family Research Project database and found that "only a handful of them measured community- or systems-level outcomes, while the majority measured individual gains related to academic achievement and youth-development outcomes." The one-way street for documenting that our youth development programs are making a difference is "fundamentally flawed" Zeller-Berkman concludes. This got my attention.

Youth programs benefit adults, too

A couple of years ago, our center partnered across 11 states to collect data from more 3,000 adults who volunteered in 4-H programs. In this survey (as yet unpublished), adults told us the many ways that they benefit from their involvement with youth. Several themes emerged:

increased self confidence for adults
improved social skills for adults
stronger community connections for adults
new learning of subject matter for adults and
access to creative outlets through the program for adults.

The adult volunteers wouldn't have gained these benefits without youth participation! To take these benefits into account, Zeller-Berkman urges us to design outcome evaluations that include:

the changes that result in the community from partnerships with youth
program designs that do this effectively, and
strategies written by youth that change adults and communities for the better.

Is it time to re-think the way we approach outcome evaluation in the youth development field? What would you change?

Pamela Larson Nippolt, state faculty and program leader, program evaluation

Zeller-Berkman's article is available to subscribers here.