Signals for some or benefit for all: grade inflation in context

Originally published on wonkhe.com

Those in the sector hoping for an easing off when it comes to the bad news stories will doubtless be disappointed to see today’s coverage of centre-right think tank Reform’s report on grade inflation. For fans of the genre it’s a special bit of work — it’s the same old stats, an enormously cynical narrative, and a set of recommendations so daft as to be laughable.

A history lesson

First, we’re transported back to the heady days of britpop, free education and no restriction on taking another degree — “In the mid-1990s there was no detectable ‘grade inflation’ at all”. We’re then yanked into the low fees noughties, where “from 1997 to 2009 the proportion of Firsts almost doubled from 7 to 13 per cent”. And what happened once we stiffed them with £50k graduation debt? “In just seven years since 2010 the proportion of firsts has doubled again from 13 to 26 per cent”. Scandalous!

“Establishing causality is problematic, yet the correlational evidence suggests that when tuition fees rise, so does the proportion of top degree outcomes”. Maybe that big investment means they’re working harder. Maybe more students are working hard to achieve the standard. Maybe teaching has improved, and assessment has become more diverse. Maybe more students are taking resists. After all, “inflation itself must be driven by factors that directly translate into universities awarding higher marks”.

Trouble is, the report then goes on to look at all the other reasons that the sector has cooked up for the miracle. A pro-VC from UEA is mocked for citing improved entry qualifications, though without mentioning the student to staff ratio shift from 18:1 to 13:1 in the rest of his quote. Degree algorithm fiddling is cited, recycling a debunked quote. And without any reference to hard work or student support or assessment techniques, it then finds a handful of academics’ anecdotes to say they’ve been pressured to lower standards. Cue the A-levels chorus of “we worked harder and so did students” from the sector, falling on deaf ears in the press and the think tanks.

Premium mint wafers for all!

The row over grade inflation tell us something about massification. It is probably true that expansion means that more people are better educated than ever before. But it’s probably simultaneously true that that a “first” isn’t as robust a signaller or sorter of standards as it was, either within an HEI or across the sector. It’s like selling After Eights in Poundland — it’s an accepted good that more of society gets to experience that treat, but the elite will bemoan that its signalling power is now diminished, and the press will notice when the box is smaller.

Central to this set of sector growing pains is the use and abuse of metrics. “Grade inflation” is posited in the Reform report (and in TEF) as evidence of gaming — a sign that standards are slipping as providers scramble to climb the tables that have included the percentage of “good honours” as a generator of points for years. Meanwhile a reduction in “drop out” (or “non continuation”) is posited in TEF as a good thing — a sign of increased effort from students and improved performance from providers helping students achieve their objectives.

The trouble is that you can play both of these metrics the opposite way. An improvement in success rates can easily be justified (and in some cases evidenced) through enhanced support and assessment innovation, particularly when targeted at those on the wrong end of achievement gaps in a more diverse study body. And an improvement in drop-out rates can be framed as a sign of slipping standards as providers do all they can to cling on to the funding units they need to get through the demographic dip set to continue into the 2020s.

Almost any outcome metric is susceptible to a cynical suspicion of gaming — fiddling with the inputs or the thresholds instead of working to actually improve performance. If we’re being honest, in any institution it’s almost certainly simultaneously both an unfair accusation and a bit of a truism. And in some cases, what some would regard as gaming is actually the intention of the metric in the first place. Does DfE want providers running humanities courses with poor LEO numbers to improve links with employers, or inject skills content, or ditch the provision in favour of technical and vocational? Or all three?

Without condition

The danger is that we sleepwalk into crisis after crisis. Take tariff discounts. Doing so through results day sleights has long been a practice, but risky — so unconditional offers are now rife throughout the sector. With posh school sixth form tutors now reporting that a critical mass of their seventeen year olds are now holding unconditionals and exhibiting behavioural issues as a result, it’s only a matter of time before this becomes a scandal with a metric in the TEF too. How can selective universities defend selection in theory, with all the inequalities that it’s a proxy for, if they’re abandoning it in practice?

When it comes to wonks, it’s all too easy to argue that the debate about metrics gaming looks at issues through the wrong end of the telescope. Moans about TEF argue that it’s an arbitrary classification that uses a weird algorithm where the right background can get you bumped up if you’re on the border. It doesn’t measure the things that matter, doesn’t compare like with like, and it’s a pretty useless signalling mechanism. I’ve lost count of the critiques of TEF from indignant sector types who rubbish the excellence framework while clinging to the UK’s baffling degree classification system with no hint of irony. Now you know how students feel.

But the reality is that the thrust of government policy of any stripe — that more of us can have what was once the preserve of a few — is unlikely to change, especially when Bob Burgess’ review of the system over a decade ago recognised that the “honours degree… is the core product of the UK higher education system”. Delivering that “product” en masse will always involve economies of scale that cause the actions of competitive actors to be playable both ways. The question in the mess is what — or indeed who — can act as the counterbalance to the cynicism, and that has to mean more than just telling the optimistic side of the story. Someone has to spot that Audi is fiddling the emissions numbers before it happens. DfE often argues that it’s for providers and the sector to show care or restraint, yet the one bit of the Reform report that does resonate is that it’s not at all clear that any of OfS, QAA, Advance HE or UUK are set up, ready or willing to spot, call out and prevent gaming when it’s happening.

A case for quality enhancement

To avoid falling into one of the twin traps of resisting measuring things at all or being caught out by another scandal, we should talk more about innovation in teaching learning and assessment, seek as a sector to understand institutional behaviour and be more open when we’re doing good. One of the miserable things about TEF talk is how much focus there is on the system and the game — but hardly anyone talks about how important, for example, decent assessment and feedback is to learning to students. And this is about more than PR — the culture of competitive secrecy we’ve ended up with (resulting in one provider last year refusing my FOI request for their mental health strategy on the basis of commercial confidentiality) means that it’s genuinely difficult to find out what other providers are doing to shift the needle on their bit of the sector’s continued NSS achilles heel. Reframing this kind of information as “public” — enforcing the expectation buried in E2 of the regulatory framework that “decisions are taken in public” and allowing the collective sector to tell a different tale on metrics and what they tell us about the student experience — will become crucial in the years ahead.

If we were bold enough to recognise that you can cause positive shifts in institutional behaviour without having to fling revealing but faulty metrics data at loan voucher holding applicants, we might get somewhere. Imagine a TEF where the metrics deliberately change every year. Imagine an annual process where ministers, sector leaders and students agree on a set of benchmarked metrics to be judged four years out and there was collective, open, innovation and effort at sector improvement. Imagine if this process were to be repeated at institutional level, where regulators would require corporate and academic governance to set some metric targets locally in response to contexts and distinctive missions. Medals and awards — which will still be prized — might just start to look like rewards for effective leadership, additional student effort and innovation in delivery in the student interest, which is probably what we wanted all along.

What is clear is that the report’s central proposal to fix all of this, a “Designated Assessment Body” for “each subject” that will include “a single, national assessment lasting approximately 3–4 hours for each degree course” is not one that is likely to be adopted any time soon. But without rapidity, humility, subtlety and smarts, the eventual outcome might be much much worse for UK HE.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s