The NFER blog

Evidence for excellence in education

It worked for me

2 Comments

Love it or hate it, Twitter is the place to go if you want a good argument.  Tristram Hunt seems not to be a fan, criticising in yesterday’s Guardian “Twitter-fuelled orthodoxies of left and right, with both sides displaying decreasing interest in evidence-based policymaking”.  One clash last night centred on the merits of a particular teaching intervention Mantle of the Expert. In the blue corner, @andrewolduk led the charge against, with his and his team-mates’ arguments ranging from “it looks mad” through to a more reasoned “where’s the evidence?”  And in the red corner, @debrakidd and others understandably rankled against the first of these arguments (perhaps the shadow education secretary has a point). It’s the response to the second argument I’m particularly interested in though: “It worked for me”.

Now, you might think you already know what I’m going to say next: after all, I work for the National Foundation for Educational Research, and have written before in support of a more evidence-informed teaching profession. And I do indeed treat comments such as “it worked for me” with a measure of suspicion.  However, at the risk of incurring the wrath of colleagues and roughly half of the Twitter community I’d like to make a controversial suggestion….

Perhaps Mantle of the Expert (MoE) did genuinely work for those people making this claim. By which I mean perhaps it really did make a meaningful contribution to improving the lives, learning, and academic performance of the children involved. After all, what reason do we have to think otherwise – surely no one is suggesting conspiracy, mass cover-up, or that we’re falling foul of a crafty online brand ambassador campaign.

So where’s the problem, and how can this be reconciled with my view that teaching would still benefit from greater use of evidence, including RCT evidence, wherever feasible? In short:

  1. Perceptions of effectiveness can sometimes be misleading: objective evidence should be used to support/strengthen such arguments.
  2. Anecdotal success in one setting may be valid, but is not enough to justify large-scale adoption elsewhere (although, nor is failure in one setting always reason enough to rule out success elsewhere – but it does lower the odds).
  3. Simply finding an intervention that works is not enough – not when there may be an easily accessible and even more effective alternative.

I’ll now elaborate a little bit more on each of these three points:

How do you know it worked for you?  Even in the absence of a big shiny RCT demonstrating effectiveness, have you adopted the new approach in a critical, evaluative manner?  It’s not enough to say that children enjoyed it (important as this is); it’s not even necessarily enough to say that a particular outcome improved over time. What you really need to ask is, can you demonstrate that the outcomes of interest improve more than they would have otherwise? Indeed, were you clear before you started what the intended outcome might be? This is where small scale teacher-led enquiry research can come into its own, as I’ve discussed in another recent post.

Does that mean it will work for someone else?  Large-scale quantitative impact studies, including RCTs typically report on the average impact of an intervention. But in reality this hides a range of outcomes across individual children and schools (David Weston gave a good explanation of this in his Research Ed presentation). This variability may be wholly due to chance, in which case implementing an intervention with a zero average effect will always be a gamble. Or it may be associated with some other factor not controlled for in the study (e.g. it worked for all the rural schools and not the urban ones) – in which case there will be schools where the intervention will usually be effective regardless of what the RCT said (and others where it usually won’t). Only by conducting further evaluation into both the processes and the impact of the intervention can this be unpicked, and well-informed decisions made about whether “it worked for your class” translates into “it will work for my class”.

Is good the enemy of great? If there’s any such thing as a universal truth, then “teachers are short of time and resources” must be one. Every moment of time or portion of budget spent on one thing can no longer be spent on something else. It is therefore not enough just to settle on something that seems to be working – we should be striving to find approaches that are the best possible use of the resources we have. Innovation and new ideas are risky – some will succeed (and for this reason the status quo can be pretty risky too) but others will not. The only way to ensure the best for children is for interventions such as MoE to be subjected to scrutiny, and compared with the best available alternatives.

“It worked for me” is fine as a starting point, but not as a final destination.

Author: Ben Durbin

Ben Durbin is Head of Impact at the National Foundation for Educational Research

2 thoughts on “It worked for me

  1. A very good summary of Oldandrewuk’s (and his/her team-mates’) evidence (well lack of) for his/her view.

    You are quite at liberty to treat the “it works for me” with suspicion, but in the end debrakidd places her practice quite clearly in the public domain and is happy to defend it. Debrakidd is accountable via the same sort of structures as Andrewolduk for the results achieved and the levels of engagement of learners in her care.

    I don’t need an RCT to tell me whether MoE is an ultra efficient teaching approach, I can try it for myself as a professional teacher. As I use it I will become more expert in it’s use and will know when it’s use is justified.

    Andrewolduk is clearly free to use his/her “make them sit still and listen to me talking or else” approach, and when he/she shows that this will work with my learners it will be up there with MoE as a tool in my toolbox for use as appropriate . I have used his/her approach myself as approapriate and will continue to do so but I find it limited in application. If he gets results with his approach (which as far as I know he has never divulged in the detail that debrakidd places in the public domain) then he/she should go on using it. As long as his/her learners, their parents and others who expect him/her to perform his/her duties are happy with what he/she does then so be it.

    I do not believe he/she should simply be able to ridicule the approaches of others on the basis that he/she heard similar justifications for Brain Gym, but that is just my view.

    Evidence directly from professional practice will inform my teaching at all times, whether it be from Oldandrewuk or debrakidd. When Oldandrewuk presnts similar evidence to the work that debrakidd has presented I will perhaps use his/her approach more widely. In the meantime I hope Oldandrewuk managed to increase his tweets/blog reads or whicheve statistic is is that he/she uses to judge the work of his/her team.

    I will continue to show an interest in mega-mega-mega analyses, but I will probably still look to debrakidd when I decide how best to approach my professional practice.

    Very interesting post, thank you

  2. Thanks for your comment – I agree that the tone of the debate around MoE wasn’t always especially constructive! That’s a good point regarding the accountability structures already in place for teachers, although my focus wasn’t really on how the performance of individual teachers is judged, but rather on the mechanisms by which practice develops and improves.

    As I’ve hopefully made clear, I’m by no means looking to undermine or underplay the role of professional judgement in shaping practice. The intention of my post is to suggest ways in which the tools and insights offered by research can:
    – support professionals to develop and learn from the approaches they adopt in their class;
    – help differentiate approaches which are more likely to translate into different settings from those which are only working because of the unique set of circumstances of a small number of settings;
    – save time and effort when looking to select between competing approaches to avoid reinvention of the wheel or repetition of others’ past mistakes (as instructive an experience as this can be!).

    My suggestions are meant in the spirit of augmenting and enhancing professional judgement – not replacing it.

Leave a Reply