Featured post

Gender in the Merovingian World

Thursday 20 June 2013

Why I don't approve of Rate Your Lecturer

According to some lame-brains who commented upon this piece by Bill Cooke (with which I largely, though not entirely, agree), if you oppose the new Rate Your Lecturer (RYL) site you must be a really terrible lecturer who doesn't care about your teaching and is afraid of bad feedback.

So let's clear that one up first.

Feedback on pretty much all aspects of my teaching has always ranged from very good to excellent.  I care about my teaching.  I don't always get it right (as we'll see) but I do try to do it well and to keep it as fresh and innovative as I can, within the limits of what I feel comfortable with (trying 'cutting edge' teaching techniques if you're just not comfortable with them ends in miserable failure and general mockery, from what I have seen: e.g. I am not comfortable in role play so I don't use it). Student feedback has played an important (if not the only) role in keeping my teaching up to the mark, not least because we have had student feedback on teaching for a long time.  I'll repeatedly come back to that.

I care about my students (well, almost all of them); indeed I actually like them (mostly).  They make me laugh; it's a blessing to have a job that puts you amongst young, lively, intelligent people.  They also drive me up the wall sometimes, because they are smart but they aren't making enough of their abilities and the opportunities they are given.  I sound off about it in depressive moments; I get into trouble for sounding off about it, I get lambasted in the press, but most of those who saw that story (above all my own students) knew that I was ranting at them because I cared about them.  One or two, I admit, I want to punch in the face (and they loom larger in my impression of the student body than they deserve to).  That aside, I am, they tell me, 'a legend'; when it was fashionable to set up lecturer appreciation societies on Facebook (c.2006) I had the largest and most active by far in my dept - maybe across my university.  For what any of all that's worth.

I used to be competitive about being popular with the students and I won at that game in two different institutions with two very different types of students.  On that front, all bets are off.  But enough own-trumpet-blowing on my part.  If I object to RYL it's not because I am a bad/unpopular lecturer.  Hell, bring it on.  10 years ago I'd probably even have got a chilli or two; 5 years ago I'd probably still have cared.  I just want to make sure we've established that if I don't approve it's not because I am frightened of RYL.  OK?

But I do not approve of RYL.  At all.  Why?  Here's why.

The site makes the bold claim that this is 'the only way' that teaching in the UK will improve.  That is nonsense.  But let me make a positive point first.  The categories of RYL are considerably more thoughtful and useful than those on its older US cousin Rate My Professor (RMP), which include 'Easiness'....  Nonetheless the dreaded hotness chillies are apparently coming soon.  One might legitimately ask what the hell the hotness or otherwise of a lecturer has to do with educational quality or review, or furthering any of the avowedly lofty aims of RYL.

But then those lofty aims are only so much smoke and mirrors.

For one thing, anonymous feedback processes already exist and indeed have done for a long time.  It is ludicrous of Michael Bulman, creator and owner of RYL, to suggest that he is creating something new and valuable.  I have had to deal with anonymous student feedback since the mid-1990s, I think.  Those with free text options are the most useful and I have decried before the move to numerical ratings.  The problem with numerical scores is that though they look precise and scientific they are anything but either of those things.  When we give a student a grade on an essay or exam we have a whole series of grade descriptors against which we have to set the mark.  We have second-markers to check, and externals to oversee the process.  Even then it's not an exact science.  There are, however, no grade descriptors for feedback scores and no checks on the appropriateness with which they are awarded.  That's not to say they are useless; far from it.  Free text, as I said, is more valuable because 'Prof. Grumpy speaks too fast/ is always late/ has Powerpoints with the text too small/ has too many Powerpoint slides/ mumbles/ talks too much in seminars/ talks over the students/ is too negative in his comments', or whatever is more useful than '2/5'.  It is a bit as though we gave essays back with nothing other than the mark on them.  Nonetheless, as an internal check on how well teaching is being done numerical feedback is, while crude, valuable.  The problem comes in turning those scores into overall averages and then league tables.  You aren't comparing like with like.  You aren't comparing the responses of the same set of judges.  What an Oxford student, say, rates as 3 out of 5, a Poppleton student might think of as a 4, or vice versa.  There are different expectations to take into account.  An Oxbridge student might, for instance, mark a lecturer down for being too dirigiste, for not allowing her the space to think and be creative; a student in another institution might mark them highly for directing his thinking so clearly.  As ratings within a department, then, numerical scores are very crude; as the basis of comparison between institutions they are meaningless.

These are just not, as they say, robust data.

Those on RYL are even less robust, because there is no way of checking whether the raters are students who have attended the course - even indeed if they are students at all.  There is no index of what proportion of students on a course think the same way about a lecturer, or the extent to which personal motivations (positive or negative) intrude into the assessment.  Previous reactions have focused on the negative, with good reason, but there are equal and opposites.  A friend of mine (who is, by the way, actually a really good teacher) once had a RMP comment, for example, that read something like "♥ My future husband!"  So much for the intellectual value of whatever comment she wrote about his teaching.  Take a hypothetical example.  Suppose a young male lecturer talks a lot about football and is generally 'blokish' in his humour etc.  He might be rated 9 or 10/10 by some of the male students who like his laddishness, but only 1-2/10 by some of the female students who justifiably feel alienated by it (OK - I've made some problematic, stereotypical comments about gender interests of football but stay with me, fill in some other analogy of your own that might work better at yielding the outcome I suggest).  You might reasonably suppose that those two, rather polarised, subsets would be those most likely to post to RYL, and they unsurprisingly yield an average of 5-6 out of 10.  But suppose that, that one unfortunate trait aside, our man's teaching was actually really good and that, had a full sample responded, his grade would average at 8 out of 10 (or conversely that it was pretty poor and would otherwise have got 4 out of 10).  All these issues make the consumer analogy highly problematic.  The problem is not that a lecturer is not a dancing bear (in Bill Cooke's formulation), but that a lecturer is not a standard mass-produced functional commodity, like a kettle reviewed on Amazon.

This yields yet another point, and one that belies RYL's avowed (but entirely disingenuous) aims.  High or low scores have nothing necessarily to do with educational value.  This develops my point about grade descriptors or their lack.  A low score can mean 'this course was too hard for me to coast on', 'I didn't like this course', 'I didn't want to do this course but I had to and I'm still sulking', 'this lecturer made me work harder than I wanted to', 'this lecturer didn't tell me what I needed to write for the exam', 'I thought he was a bit of an arse' or any number of other things which have nothing at all to do with pedagogical quality.  A high score might mean 'this lecturer is cute', 'this lecturer is funny', 'this lecturer gives high marks', 'this course was really easy', 'this lecturer pretty much told us what was on the exam', 'this lecturer brought us chocolate/bought us all a drink at the end of the course' or any number of other things which equally have nothing to do with educational or pedagogical quality.  Again, students need to beware if they are making choices based on RYL, and in my view they shouldn't make their choices on that basis.   In fact I very much doubt that many will (at least in the quality universities), for the simple reason that most are too smart to do so.  That raises the point of what RYL is really for (on which more later).

In any case, as I have argued before, simple numerical data, league tables and competition do not raise standards.  In my view they can (and frequently do) actually lower standards.  Where these things get taken seriously (as all too often, but let me reiterate that RYL won't be taken seriously), they lead to the production of the sorts of things that generate high scores and thus higher league table positions, not necessarily things of higher actual quality.  Hypothetical analogy.  Supposing the research assessment developed its impact agenda and started to award points for readership/books sold (a concrete, objective datum at least, you might say).  That would certainly lead to Universities pressuring lecturers to write best-sellers - not necessarily (or even likely to be) works of higher intellectual quality.  I've talked before about how KIS data lowers pedagogical standards.  If making courses as easy as possible, on undemanding subjects, telling students what to write, setting easy exams, and giving free cake at each class produced high RYL feedback scores (and believe me it almost certainly would), and that led to universities compelling all teachers to make their courses like that would it raise standards?  No of course it wouldn't.  Because what yields high scores is not necessarily what is good.

As I have discussed before I decided to move away from being a 'dancing bear' (in Bill Cooke's terms) to teaching more self-consciously difficult history that I felt would stretch the students intellectually and represent more of what I thought a history degree would do, and to damn my feedback ratings.  As it happens my students responded  well to that, which is an important point.  But my feedback score did drop.  I'm happy to take a 10% cut in approval to do something more rigorous.  How does RYL acknowledge or respond to that in its disingenuously self-proclaimed crusade for better standards?

There is yet a further point.  Another one that I have made before, I admit.  The consumer analogy is fatally flawed.  Take the kettle example I used above.  Someone who buys a kettle, from Amazon or KettlesRUs.com, or wherever, has (usually) a pretty clear idea of what a kettle is and what they want it to do, frequently on the basis of having used kettles in the past.  Someone who buys a CD (if anyone still does), or whatever the young people do these days, by a particular artiste has an idea of what they like about that artiste's work, or that of others in the genre.  Certainly one hopes they at least know what a CD/download is supposed to do.  On that basis they have a good basis upon which to review the product on-line.  This kettle is rubbish; it gives off a funny smell and took an hour to boil a pint of water is perfectly valid, if true.  This CD is not as good as X's last one, or as Y's current album, is equally valid.  On that basis, a review of a course on the basis that a student didn't enjoy it as much as another course she'd taken is also fair enough - but brings us back to the point about comparing like with like.  But there are strict limits to this.  At the end of the day a lecturer knows better than a student what the important elements of a course are, what the students ought to be taking away from a course and what would be the best way of conveying all that - in the way that a kettle-producer doesn't know better than the kettle-consumer, or even in the way that (to some extent, in some ways) an artiste doesn't know better than his fans.  If you think you know what a, say, history degree should involve, how it should be taught, what subjects it should cover and what weighting different areas should be given, then really you ought not to be taking the degree, but teaching it.  If you expect a degree just to give you what you want then why have a degree?  You want a qualification.  But you want it to be worth something too, right?  To that extent you have to trust the professionals to know better than you.  Once you have a degree (or two) then maybe you're in a position to judge a degree course.  Before then, not.  As I have said before, the point of a degree is to produce someone more capable of judging a degree.  If you were knew enough to judge it at the start, or half way through, it would not be worth doing.  Ultimately it is for the lecturers and the many assurance processes they have to go through to get courses approved to determine the intellectual quality of a degree.  Student feedback has an important role in ensuring the quality of its teaching but numerical scores - as I have been at pains to point out, and have doubtless laboured the point as a result - is a blunt and potentially very misleading tool to use to that end.  And, as I have said, it is most unlikely that any university will take any account of RYL anyway.

And then there is RYL's fatuous claim to be trying to 'redress the balance' between teaching and research.  It is, I have to say, a real shame that so many students evidently think that their lecturers derive more credit for their research than for their teaching, or care so much more about their research.  Some do; that's true.  It is a minority these days, and that is one good outcome of the greater degree of quality audit that the profession has been exposed to over the past 20 years (again making the point that RYL has come too late to make any difference).  To be promoted you have to show that you are doing your job to the required standard in all three areas of the lecturer's job: teaching, research and administration.  You then choose two areas in which you think you have done more than could be expected at your level.  Most lecturers go on teaching and research, a few on teaching and admin (largely because most lecturers enjoy teaching and research more than they enjoy admin) but even those who go on research and admin have to show that they've been doing their job properly as a teacher.  In those cases, if they want to be promoted again, even if they still want to emphasise their roles in research and admin they still have to show that they are teaching effectively.  That, at least, is the case in my university and I don't think that that is atypical.  Ten years ago, once you had your senior lecturership you could indeed ignore everything but research to get promoted but that's really not the case any more.  Teaching feedback is an essential part of the promotion process.  Thus RYL's claim to be rewarding good teaching, and for the first time giving students a voice in that, is, in addition to being flawed and factually erroneous for all the reasons already set out, missing the boat by about ten years.

RYL refers to moving away from valuing lecturers simply for 'churning out research'.  Do you really think that that is all that is involved in university level research?  Churning it out?  Try actually doing some, all ye who can't get a 2,000-word undergraduate procedural essay in on time.  Leaving that aside, though, research is what universities are there to do and it is as important a role as teaching.  Do you really want UK HE to stop producing world-leading research?  Maybe more important than that, though, is the fact that active, high-quality research and good teaching are fundamentally linked.  (As an aside this is especially disheartening when set aside the claims of the purveyors of on-line degrees that students should be lectured to by 'star' media dons rather than ordinary lecturers.)  How can I teach you how to be a good historian if I am not a good historian myself?  How can I teach you about how to research if I don't research?  How can I give you cutting edge insights into historical topics if I am not myself actively involved at that cutting edge?  You can't have this both ways.  You can't demand to be taught by the best and then claim that the best spend too much time 'churning out' research (as if that would be what 'the best' did).  Lecturers only have 24 hours in a day like anyone else, and (believe it or not) have lives outside work (well, most do).  The pressures on that time are increasing all the time, frequently in ways that have little or nothing to do with either good teaching or high-quality research.

As a last point on this issue, the very same government policies that have slashed back funding, forcing Universities to charge fees (and at which you should be directing your ire, not at University teaching staff) have also forced Universities to rely ever more heavily on research funding.  And believe me, that is FAR more lucrative and significant in a departmental budget than fees.  RYL won't make any difference to that.  Most lecturers I know would far rather be teaching than filling in grant applications but in the current financial situation that is what they have to do, for very real reasons relating to the very sustainability of UK universities.  Argument about students being the sole consumers and financiers of universities are simply wrong.  Want to change this?  Pressurise the Labour Party to commit to reversing the government university funding policy and vote for them if they do, and insist they make good on that.  And realise that a top-quality education system with has fair access requires state funding, and that in turn requires taxation, especially of the rich.

Thus far I have set out why the claims made by RYL are misleading, why it won't (and indeed can't) do what it claims.  My objection to the site is so far based mostly on its essential fraudulence.  But you might say it's still pretty harmless even so, especially if no one in any position will take any notice of it (and they won't, any more than US universities take any serious notice of RMP).  It is not useful but it is not harmless.

What, then, is the point of RYL?

Michael Bulman, the owner of RYL, clearly, from his statements on the site, knows nothing about British higher education, as all the above makes clear.  But that does not matter to him.  The purpose of his creation of the site is simply to make money.  By creating a site that will get thousands, indeed millions, of hits he can charge for advertising space on it and make a tidy profit.  Let's be absolutely clear about this.

The purpose of RYL is not to improve HE; it is cynically to cash in on the obscene situation where you, the students, have to pay for your higher education.

For that reason alone, if you actually care about UKHE, you, the students, should shun it.

There are however, lastly, very serious moral or ethical issues about sites like this.  They further the pernicious and spreading culture of bullying, cowardly, anonymous abuse.  Bill Cooke (author of the 'Dancing Bears' piece linked to at the top) argues that it will stop lecturers from giving negative feedback on students.  Actually I think that most lecturers have more guts than that.  I have recently finished teaching the worst 'Barbarian Migrations' class that I have taught in 17 years. They weren't stupid (far from it), they weren't unpleasant (even further from it), they weren't even lazy, overall.  But they seemed just to have no commitment to the course and its requirements, showed no initiative, couldn't be bothered to discuss things or talk in class.  I tried with them; I put a lot of thought into the course organisation and seminar set-up; I held a discussion half way through about how I could improve the class.  To no avail.  So I have said as much on their reports.  Most are mature and self-aware enough to take it on the chin.  At least one, though, is puerile enough to write something nasty on RYL as a result.  So be it.  If I get another class like that I'll do the same.  RYL is not going to stop me calling it how it is.  But I should not, nevertheless, have to take public abuse on the internet for doing so, just for doing what I see as my job: abuse that anyone (such as colleagues in other universities) can read without knowing the context behind it.

But I am an old hand who generally (as I said) gets good feedback (even from that 'Barbarians' class).  Let me give you an example that raises more serious issues.

A few years ago we had a young lecturer who was given the task of giving a survey lecture course to the first-years.  This is a serious and daunting task, with a large audience.  Maybe we shouldn't have saddled her with it, though everyone has to start somewhere, and it's the sort of thing I had to do, back in the mists of time, in my first job.  I guess one or two lecturers are brilliant from the off but most aren't.  Even the best are a bit rubbish at first, as they'll admit - even if they tend to be rubbish in an entertaining or endearing way (as I hope may have been the case with me) rather than a boring or incompetent way.  But one thing you often do when you start is to have a script and read it.  It is a prop for confidence and it makes sure you stay to time and on the point.  But it also (usually) means you have far too much to get across.  And so it was.  Unfortunately this lecturer was also up against (in the other survey courses) experienced and popular lecturing old hands.  I had to review the feedback on this course and it was vicious.  Unfair, rude, unhelpful, destructive, sexist.  Indeed I really think that a male lecturer who did the same would not have faced the same abuse.  It was, in short, bullying; bullying a bright young woman starting out on her career.  This was bad enough in internal feedback.  Imagine what it would be like if those students had published all that on the internet for all to see.  Apart from potentially harming employment or career prospects, how would it feel to read all that about yourself?  To adapt the famous quote of (I think) George Bernard Shaw, 90% of lecturers will admit to liking reading gushing praise about themselves from students, and 10% will lie.  Similarly no one apart from the super-thick-skinned likes reading bad feedback or rude comments about themselves, even if 1% of the total return, even if manifestly unfair, even if patently written by someone who never turned up and did no work.  So this sort of thing could easily drive someone with great promise out of the entire educational sector.  Indeed some of the students started a Facebook group that had that abuse about that lecturer on it - which I shut down pronto.  The problem with the attitude that the anonymous, 'drive by' abuse culture fosters is that is is one wherein one does not think about how comments affect the recipient.  It fosters the idea that you have no responsibility.  These are all very bad things.  The fact that it goes on all over the place, from comments strands and Facebook through to Amazon does not make it right or justify the production of yet more.

The Union should provide legal assistance to back action to have lecturers removed from the site, thus making it even more meaningless.  It should also sue Bulman's ass for any even remotely bullying or offensive material.  I doubt it will do either.

But otherwise, by way of response, how about 'Rate Your Students'?  Lecturers could (anonymously of course) name the students on their courses and grade them on their punctuality, attendance, ability to write, manners, reliability, performance in class.  Maybe they could add free text comments on their personalities and defects, and maybe a chilli or two on their general 'hotness'?  There would be every bit as much justification for this.  We educate people who then go out into the wide world and are employed.  Thus the employers of our students are 'stakeholders' (in awful New Labour Speak) in UK Higher Education and deserve candid information to be made freely available about their potential employees that goes beyond anodyne results break-downs and reference-ese. The people who - though they may not pay the fees - pay much else towards students' education need also to be sure that the students are working hard enough.  Those who make the loans deserve to know whether they are backing a sound investment.  And education is a two-way street.  Students deserve to know whether their fellow students in a class are going to be collaborative and supportive or whether they are going to be the sort that sit in seminars never saying a thing, who make no contribution to class discussions or group work - in other words, who piggy-back on everyone else's hard work.  It would be the only way of raising standards among UK students.  Why not?  Additionally, it'd get loads of hits as a site - maybe more than RYL - and generate a decent advertising revenue.  [Get in touch Mr Bulman.]  But can you imagine what the students' response to that would be?!  But why the outrage?  If you're opposed to being publicly, anonymously rated it must be because you're a shit, lazy student who's only at university to drink and get laid, right?