Universities continue to embarrass themselves over rankings

Vandy and Tulane in the current top spots

This appeared on my blog over at Science.

Last week, U.S. News & World Report issued their rankings of the best undergraduate programs in the United States. To counter criticism that the rankings did not adequately recognize universities for supporting low-income students and social mobility, the magazine recently made substantial changes to the formula. As a result, the rankings of many public universities improved dramatically, and a few private universities dropped to a lower position. The reactions from some institutions revealed that they prioritize rankings over other aspects of their missions.

The formula change included discontinuing a long-standing bonus for having classes of fewer than 20 students. When U.S. News added this criterion to the formula, implying that better universities have small classes, many universities launched efforts to limit classes to 19 students and to offer programs like First-Year Seminars, which added new small classes to the school’s roster. I have enjoyed teaching small classes at both the University of North Carolina (UNC) and George Washington (GW) University. Although I agree that small classes have benefits, class sizes should be determined by pedagogical evidence, not rankings. If universities now opt for fewer small classes, the change may indicate that they offered them for the wrong reasons. U.S. News also discontinued the part of the formula that awarded extra points on the basis of the percentage of alums who donate money to the university, which had contributed to phone banks of undergraduates calling to solicit small gifts. If the change in the formula is designed to reward enhancing of social mobility, U.S. News apparently doesn’t think small classes or large numbers of small gifts from alums make a big difference.

In the flurry that followed the release of the rankings, two private universities that dropped in ranking—Vanderbilt and Tulane—made strong statements condemning U.S. News for changing the formula. For decades, most schools have sent out glowing press releases when their scores increased. Higher education pundits have had fun pointing out the karma of loving the rankings when they went up and condemning them when they went down, but the underlying message in the schools’ statements was that public universities do not stand for quality in the way that private schools do. Anyone who has worked in both environments (I was an administrator at both UNC and Washington University in St. Louis) has seen the “state school” bias that exists in private higher education. Private universities are outstanding environments for education and research, but their effect on social mobility is modest given the much greater numbers of students who attend public universities. If the goal is to get more students ahead, recognizing public universities makes sense.

To unpack all of this, I talked to Dominique Baker, a professor of education policy currently in the process of moving to the University of Delaware. Baker is an advocate for educational equity and backs it up with outstanding research. Here’s our conversation:

Holden Thorp: Here's my first question. I don't think either of us are fans of these rankings to begin with. U.S. News made some changes. Did they improve their legitimacy? What do you think about the changes?

Dominique Baker: Yes, that's an interesting question. The big question is: Legitimacy to whom? It’s very clear that some institutions feel that the new rankings have reduced their legitimacy, but it’s not quite clear yet whether their legitimacy has been changed in the eyes of students and their families. And I think for most academics, it has maybe slightly increased their legitimacy or has not changed their thoughts about it at all.

Holden Thorp: Do you think that this will cause more students to go to public universities that are better equipped to provide social mobility?

Dominique Baker: We have good research that shows that there is actually some causal effect to how families and students use these rankings, but the order of magnitude is not necessarily that big. And when you talk to the majority of students, whether that’s through qualitative research or reported features, you see that when people are considering where they want to go to college, there are a lot of things that they factor in. Where is this college located? How far is it from home? What types of programs does it have? And what am I thinking that I’m interested in doing?

Are the rankings a part of that piece? Sure. Are they the number one thing that helps families decide where they’re going to go to college? It’s almost certainly no.

Holden Thorp: I did an informal poll in my class of 16 George Washington University students who are from all over the country. They all acknowledged they looked at rankings, but when I asked them if it was determinative, they said no. Now that wasn’t true when I was at Washington University in St. Louis. I would ask students, “How’d you decide to go to WashU?” And they would say, “It was the highest ranked school I got into.”

Dominique Baker: See, this is that thing: Who are the majority of students in the United States versus who are the people that are attending WashU? Because, as we know, there’s not a significant amount of overlap between those two groups.

Holden Thorp: Yes, I think that’s it. And WashU and GW are both well-resourced private universities, but when you shift focus from a school that has been in the top 10 or in the top of these rankings to one that’s—I don’t even know what GW is—but it’s probably in the 50s or 60s somewhere. Then suddenly it becomes less important.

Why do you think Vanderbilt made such a strong statement?

Dominique Baker: I wish I knew. I will say that I don’t think they’re the only institution that is talking about the rankings in this manner. So it’s not so much that I was shocked by the sentiment that they held. I was quite shocked by their choice to communicate as they did to these two constituency groups, the alums and the faculty.

And it’s especially interesting to me, given that Vanderbilt’s so-called drop seems to me to be within the margin of error. In any given year you’re going to see moves, so moving as few spots as they did is not really that big of a difference when we think about how many institutions, or even how many 4-year institutions, there are in the United States.

It’s not quite clear why they decided to go so far as to say that these measures are “misleading” and “incompetent.”

That’s very loaded language to use, especially without acknowledging that these rankings have never been a capital T truth about what quality is within education.

Holden Thorp: Wake Forest dropped a lot more. I read President Susan Wente’s email, and it was very calm and said, “This is why this happened for those of you who are interested. Wake Forest is still doing the same things that we were doing before.” I thought that was sensible. I mean, it’s understandable that alums would have questions about this.

Dominique Baker: Questions, yes.

Holden Thorp: So it’s good to explain what happened. But Wake Forest, I thought, which in a way got hurt much more by this, if this is a hurt, had a very level-headed statement.

The thing I’m interested in is, okay, so Vanderbilt pulls out of the law school ranking because they say it discourages things that serve the interests of mostly students who don’t have high LSAT scores or choose to enter professions that aren’t all that remunerative compared to going to Skadden Arps and being an associate, and so they’re basically blasting the law school ranking for being too elitist. And then when they drop in the undergrad ranking, they blast it because they’re basically saying it’s not elitist enough.

Dominique Baker: I think that that is a fair assessment. Which is part of the challenge to me. I also follow UVA [University of Virginia] fairly closely, given that I accumulated a couple of degrees from there. And I actually remember that when the law school rankings came out, which UVA Law pulled out of, the main UVA social media did posts and trumpeted their ranking. I actually publicly questioned it and said, “I don’t understand why UVA is celebrating the rankings in a system, which the law school actually pulled itself out of, that is based only on publicly available information.”

So that incident, and this one with Vanderbilt, both show that not everyone at these institutions agrees that the rankings are too elitist, and that it takes a lot to convince the key stakeholders and upper administration that these rankings don’t serve their interests. A number of institutions have been able to convince the law school administration or the medical school administration, but that is not something that they’ve been able to do university-wide.

Holden Thorp: And I guess it’s because the trustees and politicians and alums just care about this so much that they can’t walk away from it. Do you think that’s bigger than what prospective students think about it?

Dominique Baker: I definitely think the trustees matter a lot. I’m certain you have been there as well. I was hearing a lot last week about meetings of trustees who are very upset at U.S. News. Because keep in mind, a number of these trustees over time have green-lit studies from consulting groups to help their institution figure out how they’re going to get a higher ranking. And now they feel like, “Well, look at that money we wasted because they’ve pulled the rug out from under us and changed these metrics. And so we spend all this time, all this money, all this strategizing on figuring out what we do to make our ranking better, and now they just completely changed the formula.” So I do think that that matters a lot.

I also think alums can matter a lot. I often think in a lot of ways on a similar topic, thinking about legacy preferences within admissions, that these can be driven by a lot of alums—and not necessarily recent alums—who have ideas about their institution. But I would also argue that a lot of institutions have not done the work to try to explain to their alums why these rankings don’t matter.

It used to be that when alums donated, that was a part of this ranking. Institutions did not spend a ton of time trying to explain to their alums to say, “Hey, you know that we have a quality education, so forget this ranking. If you want to donate, that’s great, but you know how great we do because you are a product of us.” They didn’t typically say that. What they said was, “Hey, if you give a dollar and the other alum gives a dollar, we’ll boost in the rankings and that matters.”

So institutions have spent decades telling their alums that their ranking is a really clear signal of their quality and strength in academics and all sorts of things in part because they needed the alums to donate more money so the rankings could go up.

Holden Thorp: And then people spend all this money setting up the classes of 19 people or fewer—I actually teach one of these at GW—to get more points on the class size score. I mean, that’s all money that could have been spent on something else. And don’t you think it’s a good thing that they got rid of that metric given that being forced to have a lot of small classes is an impediment to enrolling more students that could benefit? What do you make of all that?

Dominique Baker: I think it depends. I think that we have some good evidence on the K-12 side that smaller classrooms can be helpful. It depends on the situation and the environment. But if you’re going to make smaller classes because you think that the research evidence shows that they are pedagogically a better learning environment, then whether or not U.S. News ranks class size is irrelevant.

Holden Thorp: Yeah, agreed.

Dominique Baker: If you are doing this because it’s the pedagogy that matters, okay, let’s have that conversation. But if you’re doing this just because you want it to make the rankings higher, that was a silly reason.

Holden Thorp: When I was at the University of North Carolina, our ranking was incredibly stable. It was the same every year. So I spent very little time on this. But at WashU, my first big kerfuffle was with the law school because when the law school economic problems started, I said, “Well, let’s just grow the enrollment so that we can pay everybody.” And the law school faculty and the alums just went bananas because they knew that would drop them in the rankings. And I kept saying, “I thought you all were here because you care about justice and stuff like that.” But whoa, did I learn that lesson from them.

Dominique Baker: They were like, “We care about that and…”

Holden Thorp: Yep. On the undergrad side, I was never able to convince them to focus on other things. And eventually I broke down and started showing all those rankings charts just like everybody else, because it’s just relentless.

Do you think anybody could ever make one of these rankings that was good? Or do you think it’s a hopelessly pedantic endeavor to try to rank these universities in the first place?

Dominique Baker: I will say I know a lot of people that I respect and admire who work really hard to try to think about how to create useful rankings. So I don’t rule out the potential that it could happen.

It’s something I personally spend less time on, in large part because I think sometimes when we talk about these rankings, we talk about them like they are just a tabula rasa, as if all of the power is within U.S. News and how they choose to communicate it. And we sort of skip over the cultural part, which is that we all within our society make meaning of these objects. And I definitely think that rankings are a cultural object. And so because of that, I think it’s really, really hard when you make a ranking system to only be thinking about the math and the formula that goes into the ranking because it still matters how the society, the audience takes that information in and makes meaning of it.

Similar to that, I think it’s really important to think carefully about the formulas we use and what values they show about the measures. But I also think we have to keep in mind that people are people and they don’t just receive information as if they’re blank slates.

That’s part of the reason why, as I’m sure you know, U.S. News chief data strategist Bob Morse once said in an interview that when they create the U.S. News formula, they don’t reveal what the formula is before they’ve finished the ranking because they look and if the ranking doesn’t make sense, if it doesn’t have face validity, they tweak it. And that sort of says it all to me. You know that if you put out a ranking, and it doesn’t say that a Harvard, a Princeton, a Stanford, an MIT [Massachusetts Institute of Technology] is the number one institution, that people will look at that and say, “Well, I don’t know if this is a ranking of academic quality,” because it takes work to have a conversation, to work through a societal shift to think about how we have defined and talked about quality previously, what does it mean to change that to think about quality in a different way?

Holden Thorp: For sure. It’s certainly good for their business that they shook this up because they’ll get a lot more clicks this way.

Dominique Baker: They get so many articles from this.