Preface: Why this matters
The following may seem pedantic to some. Here is why I disagree: There has been a lot of discussion about the decline of science in America. A commonly thrown out factor is how media outlets have progressively lessened the amount of science reporters on staff. Another commonly said fact is how funding for scientific research is declining.
There are two ways to look at that graph. First, “funding is declining!” Second,
“Since 1998 funding has…increased.” (By five billion dollars.) Through a myriad of ways science oriented people bitch and moan, “we need more STEM workers,” through a myriad of ways science oriented people are poor communicators, and no better than the people they chastise. “We need to be numbers driven, not opinion driven.” Yet they only look at the graph above in terms conducive to their interests. That is, since 2004, not since 1998.
And no, we don’t need more STEM workers. Or PhDs.
Image credit: Nature.com
(We need more people who know how to put all the research we currently have into practice.)
The fact of the matter is science has many flaws. As I will allude to repeatedly, science is a human endeavor, and humans are quite flawed. This isn’t the issue though. The issue is how often science and its purporters ignore its flaws. Act like science is the end all, be all, can cure all. How they think they can get away with statements like, “We don’t know what will come of our experiments or our work. But give us a shitload of money anyways. It’s in the name of science.”
It’s the difference between NASA and SpaceX. I like Neil DeGrasse Tyson, but ask him why NASA should get more money, and he often goes into these abstract philosophical points. Ask Elon Musk why SpaceX matters and he tells you the same thing, in every single interview. “We need to make humanity multiplanetary in order to increase our odds of survival.”
I really think the “we need more science” people would be a whole lot more effective in their communication if they stopped acting all high and mighty. I really believe it’s one reason people who are not science oriented end up being alienated. It’s how we teach math in this country. Rather than start with what one can do with math, we teach the math and say “figure out on your own what you can do with this.”
“But why am I learning this? What can I do with it?”
“It’s math. It’s important.”
“Oh, ok.” [Hates math the rest of their life.]
Or look at how much of a failure scientists have been in communicating climate change. There is 1) the lack of acknowledgment that scientists have been wrong in the past (and are often wrong) -part of the hubris- and 2) the inability to communicate in terms people care about.
Ask Elon Musk why electric cars matter and it’s the same answer, every single time, “we need sustainable transport.” It’s not climate change, this thing to many that is just too hard to conceptualize. “Why do I care about something 100 years from now? I’ll be dead.” So, Musk instead goes “we will run out of oil so we have to do this anyways.” He talks about the performance of an electric car, like how it accelerates faster than any other car. Its handling. Has two motors. Has autopilot. Saving money by not buying gas. The fact the car doesn’t need oil changes. Lesser brake pad changes. Terms people care about. Climate change is merely a nice side effect.
And he’s always sure to point out, “This would probably happen anyways, but I’m merely hoping to accelerate it by maybe 10 years.” That is, some humility. (This is all an underappreciated aspect of his intellect, as this is carefully calculated.)
I’m going to tackle this topic with some specific points regarding the limits of scientific research.
–
An excerpt from an email:
“After surgery, the doc came in and told me my ACL had been gone ” for a while”. The only assumption I could make was that I tore it in my first “major” knee injury during high school basketball, then subsequently sprained and strained other parts of the knee multiple times. It just shocks me to think I played 2 years of high school bball and 3 years of intramural sports in college with only one acl. Have you ever heard of something like this?”
First, I have heard about this, and I’ve written about it when discussing Adrian Peterson’s ACL recovery.
Second, I’m glad this question was asked because it brings up a point I haven’t addressed on this site.
There are some really annoying people out there who want a research citation for every sentence. These people sit in their Ivory Tower responding to every argument with, “Where’s the research?” If there isn’t a study on it, they don’t believe it, refuse to acknowledge it, or, and this is where my issue comes in with these people, they put not only the idea down, but the person down. “Intellectual bullying” seems to be the trendy term. I prefer “being a pretentious asshole.”
Not only are these people assholes, they’re being moronic. (And don’t realize it. The worst kind.) They act as if scientific research is pristine. That because a study was published, it’s infallible. It’s as if they forget humans are behind research. Humans. Fallible, humans.
Physics is the most stringent of the sciences. In human research, such as biology or medicine, hitting statistical significance of 0.05 is customary. Your results are 95% likely to not have been due to chance. In physics, that number is 0.0000003. There is a one in 3.5 million likelihood your results are due to chance. Physics used to be less strict, where 3-sigma was the standard. (Which is still more strict than any research using humans.) However, too many findings, upon more data, turned out to simply be statistical fluctuations. Physics now calls anything only at 3-sigma as “evidence.” 5-sigma is required for “discovery.”
Despite this, when an experiment is done by some of the most reputed physics researchers in the world, finding one of the most groundbreaking results in recent memory -that of gravitational waves- all at 5-sigma, it’s later found out their results may have only been space dust.
To be clear, in physics terms, everything in psychology, physical therapy, medicine, etc. is only “evidence.” Even at 5-sigma, physicists rarely will say they’re certain of something. Richard Feynman stated “I’m not absolutely sure of anything.”
Of course, sensible people understand research is not perfect. Publication bias, selection bias, statistical manipulation, we know these things exist. Again, humans perform research, and humans are going to screw things up. Unintentionally and intentionally.
However, when we find out mice are significantly influenced by whether the researcher is male or female, the research nazis seem to ignore the, well, research. Even though finding out mice are stressed around a male presence, which changes things like their pain response, calls into question, I don’t know, probably something like every study ever conducted where a researcher was in direct contact with the subject(s).
Again, physics knows this. They know you can’t measure something without influencing the results. As soon as you try to measure something like an electron, you change the properties of the electron. You can’t observe an electron without hitting it with light (how else could you see it?), which is going to do something to it.
This is not a flaw of humans, but of research. Just as humans have limits, so does research.
I believe I’ve come across the following issue only once in a medicine / biology / physical therapy oriented paper. I assume many researchers are aware of this sort of thing, but those, such as clinicians, who obsess over research are not. I believe I came across this on ACL / meniscal repair research (it was maybe a single sentence), which brings us back to the email I received.
Selection bias not only can be performed by the researchers. Selection bias can be a function of the ailment you’re studying.
One of the best ACL papers I’ve come across is A Randomized Trial of Treatment for Acute Anterior Cruciate Ligament Tears.
After detailing what hospital their research took place at, the methods section goes,
“Subjects 18 to 35 years of age who presented to the emergency department with recent knee trauma were screened for eligibility.”
What about those who never presented to the emergency department? Those who twisted their knee, felt weird for a day or two, or maybe a week, but then went on as if nothing ever happened? We know nothing about these people. We don’t know who they are, we don’t know where they are, we don’t know how many of them exist, we don’t know if they exist!
This study claims in the introduction,
We conducted a randomized, controlled trial involving young, active adults with an acute tear of the ACL to determine whether a strategy of structured rehabilitation plus early ACL reconstructive surgery is superior to a strategy of structured rehabilitation with delayed ACL reconstruction offered to subjects who continue to have symptomatic knee instability.”
This is not what this study examined though. This study examined young, active adults with an acute tear of the ACL, who were symptomatic enough to seek the emergency department, thus being eligible to qualify for our study.
All the people who tore their ACL, didn’t seek treatment, and in all likelihood never sought treatment? In this study, it’s as if they don’t exist. Immediately the outcome of this study is biased. Because this study isn’t comparing those who have surgery versus those who do not. It is comparing those who were symptomatic enough to seek a doctor, and the results of them having surgery versus those who did not.
If you think this is minutiae, or semantics, then what do you think science often concerns itself with? Do you not want your bridge engineers to worry about minutiae? This is why some professors out there grade certain subjects, like engineering, with no partial credit. There is no other passing grade for something like a bridge, or a rocket. It works or it doesn’t. Minutiae only matters in the context it’s given, yes, but you can’t disqualify something without first knowing its impact.
So, all the people who tore their ACL, didn’t have surgery, and are fine? They don’t factor in. They would push the results more in favor of not having surgery, but we don’t know how many of them there are. We don’t know what’s unique about them. We can’t account for them.
“What if you studied the percentage of people with an ACL deficiency in the general population, and find out how many of them are doing fine?”
Sure, we could do that. It better be a huge sample size, and it’s going to be insanely expensive to MRI that many people -cost, another limitation of research- but it could be done. But how are you going to differentiate between those who were born without an ACL and those who tore it through their lifetime? How can you know for sure? How do you know when they tore it if the person doesn’t know? How do you know if they adjusted their lifestyle after they tore it if they don’t even know when they tore it? Can you even reasonably answer these questions? I don’t mean is it hard, I mean is it possible?
Let’s look at the guy who emailed me. Let’s say he tore his knee up when he thinks he did. Massive assumption (bad science), but go with it. Then he’s able to go five years of high level activity with minimal issues. (We have to rely on his memory for this. More bad science.) Let’s say one day he tears his knee up to the point he’s ready to get medical treatment, which seems to be what happened.
What if someone else had the same story, but the day before they really tear their knee up, they decide, “Eh, I’m done with this level of activity.” To where the rest of their life is performed at a level of activity that they never tear their knee up badly enough to seek medical treatment. Is this person a coper or non-coper? If we surveyed him (bad science) and MRI’d him, we’d say coper. Realistically, we don’t know. We’re guessing. (Bad science.) He merely may have happened to adjust his lifestyle to where his issue never hit a particular threshold.
And, with ACLs, relatively speaking, this is easy compared to other issues. With other issues, this is not a solvable problem. At least with a torn ACL, you often have solid evidence. Usually, there are some remnants of the ACL remaining. What about something like a meniscal tear? With an ACL, you could conceivably at least get a good idea of how many people are walking around with no ACL and are doing fine, right? Theoretically, MRI a million people. Find out how many don’t have an ACL, survey them on their lifestyle and function, yada, yada. What if the ailment, like a meniscal tear, can heal?
If you do a study comparing acute treatments of meniscal tears, what about all the people who tear their meniscus, don’t seek the ER, then their meniscus heals? If you MRI a million people, you have no idea who those people are. You have no way of knowing if they twisted their knee one day, actually tore their meniscus, but they felt good enough to not go to the ER, then their meniscus heals. Versus those who twisted their knee one day, but that’s all it was.
You only see the people who tear their knee up, then decide to seek medical treatment. And you can’t even technically say, “Ok, we’re at least studying those who had bad enough tears to come to the ER. Or those that are symptomatic enough. How do we handle those people?” Because then you’re ignoring hypochondriacs. Then you’re ignoring the whole brain aspect of what causes pain. You’re ignoring those who, other than near death, don’t go to the ER. Those who hate doctors. Those who can’t afford doctors. Whatever.
We’re not studying the tons of people who tear their knee up and end up fine. These are the ones we’d most like to know about! What are they doing that the more symptomatic people can emulate?
To you, the research nazi, are you accounting for these biases? Because, I’d say with a confidence of five-sigma, you not only aren’t, more often than not, you can’t know how much dust is in your waves.
–
–Ironically, here is a great research paper regarding science communication:
–What’s next for science communication? Promising directions and lingering distractions
–
–
Update 8/5/15- Electric cars do have break pads! But they’re used less, so cost is saved there too.
Posted on December 17, 2014