« Pay Me Now Or Pay Me Later | Main | Ding Dong Bell, II »

How Do We Evaluate A University's Performance?

Is the University of Minnesota an Under- or Over-Performer?

Although rankings of universities are certainly a matter for debate, the data available from some of these exercises, especially that of US News, can provide some valuable and thought provoking information for prospective students and their parents. University administrators would do well to think about them and use this information to make rational improvements in the way they operate.

Would you send your son or daughter to an institution that has a much better graduation rate for its students (say 20% better) than would be expected based on the characteristics of an incoming class or would you send them to one that underperformed and actually had a lower graduation rate (4% lower) than would be expected?

One little noticed feature of the 2009 version of US News rankings is an analysis of graduation rates for 2007. After all, it seems reasonable to ask: given an input set of students with certain credentials, how well does an institution do in graduating these students.

Institutions who admit very well-qualified students should have a high graduation rate and those whose standards are somewhat less rigorous might have lower rates.

Thus a new statistic is available described as Overperformance (+) and Underperformance (-).

So to take the example of Harvard: their predicted 2007 graduation rate is 94% and their actual rate is 97%. Their performance score is +3, because students actually graduate at a slightly higher rate than would be expected, based on their entrance characteristics.

According to USNews,

Graduation rate performance " … [is] an indicator of added value [that] shows the effect of the colleges programs and policies on the graduation rate of students after controlling for spending and student characteristics such as the proportion receiving Pell grants and test scores. We measure the difference between the schools six year graduation rate for the class that entered in 2001 and the rate we predict for that class. If the actual graduation rate is higher than the predicted rate, the college is enhancing achievement."

So how does this work out in practice?

Let's first look at the Big Ten Schools in order of the qualifications of incoming students.

All  BigTen.jpg


Now Northwestern is an anomaly because it is the only private school. Also there is not a whole lot of headroom when your actual graduation rate is 90%+. It is noteworthy, however that the U of M is smack in the middle of the BigTen in terms of entrance qualifications.

So let's put Northwestern aside and redo the list based on performance. The results are quite surprising, I think.

Performance Order.jpg

So here we are, once again, in a familiar place at the bottom of the BigTen as far as this graduation statistic goes. Our next closest rival, 7% above us is the University of Iowa. It is noteworthy that there are five schools in the BigTen whose incoming class has a lower predicted graduation rate than Minnesota. So making noises - as our administration has been doing - about the ever increasing quality of our incoming students is clearly not the answer to the questions raised by the data above.

What is it about Penn State, Indiana, and Michigan State that makes them such overperformers compared to Minnesota? In fact all of the schools of the BigTen, except Minnesota, are overperforming in their graduation rates.


Why is this, President Bruininks and Provost Sullivan?

Comments

I've written on your other blog about Penn State's graduation rate. A liberal late drop policy at Penn State, easy grading in general and numerous fail safe majors makes it neigh impossible to flunk out unless one truly tries. There is also a constant drumbeat of propaganda reminding the students of how "special" Penn State is and by extension how special they are. This, I'm guessing, helps in the retention of students by making those who entertain thoughts of transferring feel guilty or like losers for having those thoughts. Penn State is mostly smoke and mirrors.

Thanks for your comments. In a system that rates graduation rates highly, PS does consistently well. From what you say, they may be very adept at gaming the system. I wonder if this is the case at some of the other places that are, apparently, overperforming?

Bill Gleason

Bill,
Being a scientist, combined with belief that students are the most important determinants for their performance, I am highly doubtful of any schools that claim double digit differences. I don't have time to research this but some of these +15-20 does not pass my "smell" test. I agree that all Universities should strive to improve graduation rates, including UMN. In this regard, I would command the U for improving graduation rate from 40's in the 90's to 60's now. I am sure you don't believe that the U's instructors were inferior in the 90's. Perhaps U's numbers are more in line with reality than the Penn State...I know a little about PSU and I agree with veblen.

Finally, there are significant uncertainties about the value and/or validity of measures used in USNews rankings. (Acutally, given your somewhat anti-establishment streak, I am a little surprised that you value USNews rankings so much.) FYI, see following...while it is dated, it seems that the "+/-" could be manipulated.
http://www.eric.ed.gov/ERICWebPortal/custom/portlets/recordDetails/detailmini.jsp?_nfpb=true&_&ERICExtSearch_SearchValue_0=ED432176&ERICExtSearch_SearchType_0=no&accno=ED432176

Keep up the good work.

Thanks for your great comments, Mike.

I am not a big fan of USNews or of their ratings. I will be putting up a post over the summer that pretty much destroys the idea of a ranking system meaning much. It will be based by an analysis done by some Berkeley mathematicians.

They point out - I'm sure facetiously - that at least one defensible rating system would put Penn State as the number one university in the country.

Having said that, the USNews data is thought provoking. OurAdministration here at the U keeps yammering about the quality of the incoming students and racheting it up as if this is the cure to bad graduation rates. Given the numbers above, even if you want to throw out Penn State we are STILL at the bottom of the heap.

Why is this? I have some ideas but won't go into them right now. We have and have had a very serious problem with graduation rates at Minnesota. I say this as someone who has been associated with the U since 1970. Saying that we are making improvements isn't very impressive, since our rate has been so bad in the past.

And ignoring the problem is not going to make it go away.

The U has the potential to be a much better educational institution than it is. I don't think the students are the problem and I don't think the faculty are. That only leave one place left to look.

Best,

Bill

Bill,
Looking forward to it.
BTW, incremental improvements are important. With time, these incremental can add to "something". Who knows, may be UMN will resemble UW or PSU?!

Post a comment

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)