Youth Development Insight

Collaborations with schools benefit youth

2012-12-12T18:23:17Z

Recently I was part of a school-community partnership group. We were brought together to create a measurement plan for learning objectives set forth by a local school district. The objectives all focused on building 21st Century skills in youth. There was a heavy emphasis on developing global citizens and cultivating youth interests and talents. Too often these traits are thought of as "soft skills"; however research suggests that soft skills are sometimes the most demanded in the workforce. In fact, National Public Radio ran an article that stated that preschool is one of the best training programs because of the emphasis on soft skills in early childhood. Like preschool teachers, youth workers strive to help youth develop skills that will make them more productive citizens.

]]> Other members of the school-community group were surprised when I said that youth workers think about 21st Century skills all the time. In fact, these are often the goals that drive our program. Their surprise led me to wonder why K-12 teachers don't know this very important part of nonformal learning. Clearly there is a need for more sharing of knowledge. So how can we as nonformal educators collaborate more effectively with schools?

I am not the first person to be concerned about this. AnnMarie Schamper, an educator in Philadelphia, wrote an article in the Spring 2012 edition of Afterschool Matters called Collaboration Between Afterschool Practitioners and In-School Teachers. With a realistic perspective she expressed the need for in-school and out-of-school educators to collaborate and communicate. She offered concrete examples from her practice to serve as a learning tool for others. Collaboration isn't easy, but she put the need for it into perspective, saying, "Collaboration between in-school teachers and after-school practitioners helps both sets of professionals, but the ultimate beneficiaries are the students."

As youth workers, we care deeply about the development of the young people that we work with. Even though it is challenging to find ways to connect with schools, I would argue that it's increasingly important. If we are to build 21st Century learners, we need the help of whole communities, and schools are an important player.

How have you effectively partnered with schools? How have you overcome barriers? What benefits have you seen?

-- Samantha Grant, assistant Extension professor, program evaluation

You are welcome to comment on this blog post. We encourage civil discourse, including spirited disagreement. We will delete comments that contain profanity, pornography or hate speech--any remarks that attack or demean people because of their sex, race, ethnic group, etc.--as well as spam.

What makes an evaluation report compelling? (or not)

2012-08-15T15:32:28Z

Okay, time for true confessions here. How many research or evaluation reports do you have sitting on your desk? You know there was blood, sweat, and tears put into the creation of those documents, but somehow you don't feel compelled to read them. Why not?

I'm willing to guess that the answer is either: A. you don't have time or B. the reports are way too boring. (By the way, reason A is just a disguise for reason B.)

The truth is that many evaluation reports are dull, but there are also great ways to spice them up by focusing the message and using pictures and stories to illustrate points. I have set a goal of unlearning some report-writing habits to make mine more interesting, and thus more likely to spur action.

]]> A new learning experience that I'm taking part in is the American Evaluation Association's eStudy series: An Executive Summary is Not Enough: Effective Reporting Techniques for Evaluators. In it, Kylie Hutchinson sets out to teach practitioners to create meaningful reports.

One of the reasons for ineffective reports may be bad writing habits. In an editorial, Jane Davidson argues that our social science training has taught evaluators some behaviors that need to be unlearned. She notes four ways:

Including models or theories without connection
Leaping to measurement too quickly
Reporting results separately by data source or type
Organizing reports like a master's thesis

In my experience, I've noticed that we train evaluators and researchers to design reports without thinking about our audiences. But if the end user isn't involved in the creation, evaluation data will never be used for change. Davidson states, "Evaluations cannot produce useful answers unless they actually ask useful questions in the first place!" I wanted to stand up and cheer for that line. Asking useful questions requires that an evaluator is in tune with stakeholder needs.

In thinking about this, I realized that as educators, our formal training may have created some behaviors that need to be unlearned. Take for example the youth worker who has training in formal education. What works inside a classroom may not translate to an after-school program or a summer camp. Academic training is important, but we need experience to be good at our jobs.

How about you? Have you read an evaluation report that really drew you in? What was it about that report that made it compelling? Can you see the need for unlearning bad habits?

-- Samantha Grant, assistant Extension professor, program evaluation

Arming parents with the tools to gauge program quality

2012-04-18T17:00:50Z

Out-of-school time providers beware! I'm a parent and know a lot about program quality. Last week as my daughter pirouetted her way into her preschool dance class, I found her dance teacher looking at forms instead of greeting the students. As a youth worker myself, I understand the demands of balancing 20 things at once. But I couldn't help thinking about how this non-greeting affects the learning environment.

I get it that I'm not the typical parent -- I'm the one who grills potential daycare providers on their use of developmentally appropriate practice, because I understand what that is. But I am interested to study more about how the average parent can become a better consumer of learning opportunities for their children. I know that my knowledge has impacted the decisions that I make for my children, and I believe the same would hold true for other parents.

]]> There is a battery of program quality observation tools that demonstrate that not only do we know what makes a high-quality learning environment, but we can observe it. If parents knew more about how to identify quality in youth programs, would they choose differently? Would parents start to demand quality and support efforts to better train youth workers?

In Exploring the Supply and Demand for Community Learning Opportunities in Minnesota, researchers from the University of Minnesota examined more about parents and their ability to find and access youth programs. From this we learned more about:

What is the perceived quality of youth programs?
What do parents and youth want and value in youth programs?
How difficult is it for families to find learning opportunities?

I would love to extend this research by digging into the idea of program quality to learn more about what parents consider to be high quality, and how that influences their decision making. Maybe it's just a pipe dream of mine, but I hope we can create a demand for high-quality programming that comes from all fronts: youth, parents, and youth professionals.

Are you interested in the role parents play in accessing quality learning opportunities for their children? How can we help parents become better consumers of quality?

-- Samantha Grant, assistant Extension professor, program evaluation

Beyond boring data

2012-01-04T18:00:00Z

By now, we are all convinced of the importance of doing evaluation of our programs. I hope we've all begun to collect data to inform our stakeholders and ourselves about how our programs are doing. I have blogged about practical evaluation in youth programs, and the theme of evaluation has been echoed by others in their posts.

Let's assume that you are collecting and analyzing data about your program -- what next? I argue that you must put in as much effort in communicating data as you did in collecting it.

]]> Before making choices about how to package your data, think about:

What data do you have?
Who is the target audience for the data?
What do you want your target audience to know?

In the fall 2011 issue of New Directions for Evaluation, Stephanie Evergreen makes a case for thinking like a graphic designer when communicating data. She states, "Evaluators have a responsibility to make their work as clear and accessible as possible, both to enhance the evaluation's credibility and to encourage the use of evaluation in program change." I agree with her but also think youth workers who do evaluation carry this same responsibility.

Evergreen says that we have a bad habit of making our communication of data boring: "The disconnect lies between our desire to have our findings used and our methods of presenting them." Are you boring your audience with data?

In youth work we have the tendency to be so pleased that we've conducted evaluations that we neglect to think about use and communication. What good are the data if we can't communicate them in a compelling manner? How can we best create communications with users in mind?

Here are some ways to create more compelling communications of data. Compare them with what you do:

Jot down the key messages from your evaluation. Build your presentation around these. Think about how you can make these 2-3 ideas stick.

Ask youth to help. Chances are they will be able to help you get your creative juices flowing. Plus the act of engaging others in discussing your communication methods has to help you break out of your presentation rut.

For an oral presentation, follow the 10-minute rule: if you can't get your point across in 10 minutes, restructure what you're talking about, otherwise your audience will be snoozing.

Think about creating two evaluation reports -- one with more depth for stakeholders who want the details and another short, 1-2 page summary that can be shared widely.

Have you found interesting, engaging ways to share evaluation data? What difference has it made?

--Samantha Grant, assistant Extension professor, program evaluation

Observation should inform program evaluation

2011-09-14T17:00:34Z

Have you ever watched a youth program where everything seemed to be working? As a youth worker, your gut reaction can be a good gauge of when things are "clicking" inside youth programs and when things need improvement. Sometimes with the current pressure to show the outcome and impact of our programs, we lose sight of the skills we develop through experience in youth work - our ability to observe and assess.

Observational methods in evaluation or research are gaining popularity in school and youth settings. In Minnesota 4-H, we have been investing in the Youth Program Quality Assessment. This standardized observational tool allows youth workers to assess safe environments, supportive environments, interaction, and engagement. There are many other tools for assessing youth program quality. Check out The Forum for Youth Investment for a review of tools.

]]> An article in the spring 2011 Afterschool Matters publication takes a look at the Self Assessment of High-Quality Academic Enrichment Practices. Holstead and King detail the growing emphasis of self-assessments inside 21st Century Community Learning Centers. Their article gives a glimpse into aligning self-assessment with standards of program practice and highlights the pros and cons of self-assessment. They note the power of self-assessment for providing information that can build "programs that provide the best possible services to participants."

Pros of self-assessment include: it encourages staff to be reflective, it promotes continual reflection, and it can generate important feedback that staff can use. One of the biggest cons of self-assessment is the risk that in tailoring tools to fit your program, you can lose the reliability and validity of the instrument.

I am a huge proponent of assessing the quality of our learning environments and I strongly believe in observation. Sometimes that means using a standardized tool, like the YPQA, but sometimes it means creating a tool that hones in on what is important in your organization. It can also mean just stopping to watch what is happening inside your program.

So what can observation add to your program?

Observation prompts program staff to slow down and be reflective
Observation takes you to the heart of youth programs - the point of service - where adults and youth come together
Making observation part of your practice helps to build skills in youth workers and encourages a climate of dialogue and improvement

Are you a proponent or practitioner of observation as part of program self-assessment? Why or why not?

-- Samantha Grant, assistant Extension professor, program evaluation

What's the definition of youth work insanity?

2011-06-08T17:04:12Z

How often in programs do we continue to do something because "It's that time of year again"? A quote from Albert Einstein reads, "Insanity: doing the same thing over and over again and expecting different results." I don't think Einstein was thinking about evaluation when he coined this phrase, but it nicely articulates our tendency to continue to offer youth programs without "checking under the hood" periodically.

As an educator for program evaluation, I believe strongly in using evaluation to guide and improve youth programs and to prove their worth to others. I know that others will agree with me on this. But I think we often fail to intentionally build evaluation into our program design and as a result our programs suffer. Jane Powers's research on youth participatory evaluation demonstrates that the act of intentionally engaging youth in the evaluation experience helps to not only build stronger programs but also youth development skills in participating youth.

]]> Youth workers are natural evaluators -- we intuitively modify environments to fit participants' needs. Being intentional about evaluation could include strategies such as:

Checking the "pulse" of a group as they conduct an activity.
Forming an advisory group from a mix of youth workers and youth to plan and critique program offerings.
Embedding reflection into daily practice. The Youth Work Institute has an excellent toolkit on reflection.
Administering a pre-tests and post-tests to youth in a program. The Harvard Family Research Project has some excellent resources on methods and design.
Analyzing evaluation data with the help of youth. Youth will be able to add their own perspectives on understanding data and in making changes to a program.

Evaluation is one natural way to determine if your program is going in the right direction or just going insane. How are you finding ways to embed evaluation in your daily practice? Do you have any tips to make it easier or more natural?

-- Samantha Grant, assistant Extension professor, program evaluation

Who can assess the quality of a youth program?

2011-02-16T18:00:00Z

Can youth and volunteers effectively assess program quality?
Does it matter if adult volunteers or 4-H staff are paired with youth to complete assessments?

Early results from our Minnesota 4-H Quality Improvement Study suggest that youth and volunteers can indeed assess quality and can work with local 4-H clubs to improve their programs. We have also learned that, for the most part, whether youth are paired with adult volunteers or staff made little difference.

]]> Traditionally, program quality is assessed using reliable, trained assessors. Taking that model to scale in a youth organization with more than 130,000 youth would be impossible, so our study sought to discover how to measure quality in a youth and volunteer-led organization.

The study design called for 40 coaches (20 youth and 20 adults) receiving a two-day training on how to observe program quality using the Weikart Center's Youth Program Quality Assessment (YPQA). Participants were grouped into 20 pairs: half of them one youth and one adult volunteer; the other half one youth and one 4-H staff member. Each youth-adult team assessed at least two 4-H clubs for a total of 40 4-H clubs observed.

The goal of the study was to investigate the nuances of using an alternative assessment approach. It was not designed to measure the actual extent of quality improvement attained or compare trained assessors with volunteer assessors. We viewed the process of engaging, as a member of the 4-H program, as a step toward building a quality improvement system.

Results revealed that:
• Youth and adults played an equal role in the observation and initial assessment scoring.
• Youth involvement in the data collection was seen as extremely valuable.
• Observed 4-H clubs were receptive to this opportunity and viewed the assessment process as a positive one.

We at the Extension Center for Youth Development are taking this work to scale in 4-H clubs around Minnesota. The project represents a fine expression of the University of Minnesota's land-grant mission: to bring research and practice expertise together to support local community needs. For more on program quality and how to measure it, visit our website's program quality research page.

Is your youth program finding ways to measure program quality? How are you doing it?

Samantha Grant, assistant Extension professor, program evaluation

Read the Preliminary Minnesota 4-H Quality Improvement Study.