When I was in college, one of the big assignments in completing an Exercise Science degree was writing some research papers. Few even want to read these things, never mind write them! They are a tedious endeavor.
One professor I had was notoriously hard, but honest. He would have one or two assignments a semester where you basically knew half the class would fail.
He’d then take these as opportunities to show where so many were going wrong, in a class setting. We’d all sing kumbaya, he hoped we’d gotten our reality check, and more or less erase that grade.
On one research paper assignment, he took an entire class to talk about the importance of references. Few in the class thought of, or knew, to reference statements they made in their paper. In one of the most revealing researcher statements of all-time, he, who had been published many times and has a textbook, said,
“Look, just throw a number and a citation in the paper. There are 40 of you. I have like two weeks to grade these things. I can’t go track down every single reference and read the paper to see if it backs up what you said. In fact, I saw some of you cited a journal which I know you do not have access to. The school library doesn’t have it, and it costs like three grand a year to subscribe. I know all you did was read the title and maybe the abstract. But at least those people made some effort.”
I’ve had an orthopedic surgeon as a client for about six years now. It’s been a huge benefit. Truly dumb luck this guy walked in and I was the personal trainer he started with.
We have topics we routinely discuss. Research is one of them. After all our discussions, you can boil down our number takeaway as: read the references. Read a paper, then you read the references of that paper, then you read the references of that paper, and just keep going.
Because nobody does this, you will be ASTOUNDED at what you find.
- You will see the referencers are way more absolute in their statement than the referenced
- You will find researchers cite a study which in no way backs them up
- You will see clear as day that researchers cited a study, only having read the title or the abstract.
- You will see such enormous conflicts of interest you wonder how it’s not written in size 100 bold font, rather than buried in size 4 (or not mentioned at all), so you know what you’re getting into
These are almost always exclusively PhDs. Considered some of the smartest humans have to offer. My orthopedist client went to a top ranked university for his training. He told me they are no less immune to this than anybody else. He had professors spout principles at him he could contradict with his own references (because he’d go read the papers).
Pain Science as an example
I want to preface this by saying everyone has made this mistake, where they didn’t fully vet the research they cite. (I certainly have. Part of me writing this is to act as a self-reminder!) Also, every human research endeavor has this problem. So I’m going to pick on someone and pain science here, but I don’t mean to go after them personally. (I’ve gone after shockwave therapy too.) It’s just that pain science is like Hansel,
On a very well received pain science article, like we’re talking tons of the top names in the field applauding the article, this author, who has a PhD, wrote,
“So education to lower the threat is THE therapy here. We now have some very good evidence to show that just pain physiology education or the top-down approach is enough to lower pain and improve function 5.”
Sometimes a statement just doesn’t sit right with you. I decided to do a little homework. Here was my response,
This study you cite (5) was a research review on pain education.
It was done by an author who sells a book on pain science education.
Five out of the eight studies reviewed were done by this author’s coauthor of that very pain science education book. Five out of the seven randomized controlled trials -the studies we put the most faith in- were done by this coauthor.
These two authors also run an entire business -books, courses, speeches- around pain science education.
Hey, I think Moseley and Butler [the coauthors] have great information. I’m a happy customer of multiple products, and I give some pain science education to every new client, but nobody is immune to conflicts of interest. Worse, this conflict should have been disclosed in the paper. (Whenever a conflict isn’t mentioned, the suspicion radar beeps louder.)
For the other three non-Moseley studies:
–one found the effect of pain education leveled off at three months
–one looked at whether a 30 minute intervention improved immediate understanding of pain science i.e. it asked some questions after the 30 minute session. It had no other follow up. They list this as a major limitation: “This is in fact the most important limitation of the present study: the lack of a follow-up period in which both information retention and changes in cognitions or behavior could be evaluated.”
Plus, it was done in chronic fatigue syndrome patients. Not really applicable to the fitness practitioner’s clientele of a localized painful knee.
-The last study had a sample size of 6. I mean, the other two above had sample sizes of 38 and 46. Not exactly great. But 6?!
-All three studies used Butler and Moseley’s “Explain Book” as the pain science education. In other words, Butler did a review on eight studies assessing whether his own book / business works.
Back to the overall review- it states in the studies it reviewed, pain education was done in conjunction with other treatments. Thus,
“Based on the lack of consistent control groups in the articles reviewed, it is not possible to draw strong conclusions about the influence of the [neuroscience education] content versus individual attention and the acknowledgment that perceived pain may be real.”
Even with all the conflict of interest, your takeaway is stronger than Butler’s!
More succinctly: recent research, from this month, from pain scientists, state,
“studies evaluating pain neuroscience education alone report small effect sizes. (28)”
That (28) referenced is actually the same Butler review.
They found used in conjunction with exercise (they actually used quite a bit of Sahrmann techniques), results are much better.
I don’t mean to sound combative, but this is a consistent issue in the pain science world. The concomitant followers over embellishing, often much more than even the people who’ve come up with the techniques.
Just read and you’ll be fine
It sounds simple and it is- just read. But it is not easy. That single response above took me about four hours to formulate. I had to find the references, find full texts of them, go read each one. That was just to read a single study properly. And that was just to vet a couple sentences of a very long article!
But it is absolutely necessary.
As a final note- a consistent complaint I get through email is “there is so much conflicting information.”
Well, this is one reason there is so much debate in the fitness world, and really any non-hard mathematical science. You don’t need to reference Newton every time you mention physics. The chain of information is much shorter. Math is basically right or wrong. It either works or it doesn’t. It’s quite literally proven that way. It’s as irrefutable as human knowledge gets.
–But even math isn’t fully rigorous!
The longer that chain though, the more noise gets lost in the signal. Research revolving around humans is the ultimate long chain. There are a lot of links to examine to see if it all holds up.
Long story short- at some point, you should get to where whenever you read an abstract and you feel like you know what you’re reading, you get uncomfortable, because you’ve been duped too many times before.
Posted on April 25, 2018