I wrote a post called A note to research nazis. In it I went over how even science has limitations on what it can accomplish. Such as what it’s able to research.
I’m going to hit on that a bit again here, but what I’m also going to hit on, which I didn’t in the other post, is how often scientific research -we’ll be focusing on nutrition- is masquerading as science. The paper I’m going to reference for this argues it can be to such a degree, it should not even be called science.
This is a HARD hitting paper to the nutrition research community. Like Ronda Rousey I’m-going-to-only punch-you-in-the-face-and-I-don’t-care-what-you-try-to-do-to-me hard.
“I WILL EAT YOUR BABIES BITCH” (Hat Tip Deandra Reynolds.)
I’m partially surprised scientists had the balls to write such a thing. Research papers commonly call out a particular research domain, illustrating some of its shortcomings, to better position why the current research is important. However, it’s quite uncommon, particularly in the health world, for colleagues to call out their industry to such a degree. (I discuss this some more here.) You may genuinely run the risk of exile, or cause significant career detriments.
The other part of me is surprised it’s taken someone this long to write something like this. It’s actually something that was on my writing to-do list, but I’m not sure I can do better than this paper.
–
The broad issues with nutrition research
If you’re someone who has read enough health articles, it doesn’t even have to be the actual research but just articles, you’re probably sick of hearing a new revelatory finding every week. “A new study shows…” how everything causes cancer…How everything cures cancer. Carbs are the devil. No, carbs are angels in the outfield. You should eat every two hours. The three meals a day tradition works well. You’re not fasting? Wait, you are fasting, but it’s not intermittently??? Dude, cavemen did NOT eat that, so you shouldn’t. Who the hell knows what cavemen ate? Protein is bad for your kidneys, but it’s important after a workout. Actually, your kidneys will be fine. Eh, you can wait a few hours after working out. These are the five must take supplements. Fish oil gives systemic pain relief. Antioxidants will help you live forever. Your body makes its own antioxidants, and trying to give it extra is detrimental. Ugh, all supplements are crap. No sir, this supplement works, but your pee may be purple. Hey, that’s the price of glory. Alcohol is bad for your penis, causes cancer, but good for your heart, helps limit other cancers, and well, actually, a lot of people get naked after some alcohol; the penis is grateful for that.
There may be no other domain of research so clueless. It’s embarrassing.
The broad issues with most nutrition research I see are:
1) It’s bad science. Or shouldn’t even be considered science. This is not opinion. I’m saying the research is conducted in a manner that is in opposition to the definition of science. The authors hit on this, rightfully, repeatedly.
2) Certain factors are never controlled, or accounted for. They’re often not even mentioned as potential factors.
3) The more solid the research is, the more likely there is a funding issue.
-> Alan Aragon has a great research review dedicated to nutrition. He recently reviewed a study on meal frequency that seemed very well done, but was in opposition to most of the other research. He gave a bunch of possible reasons for this. He neglected to mention the study was funded by Nutrisytem, who has a stake in meal frequency. Is Nutrisystem going to publish a study saying their entire business isn’t worth the money? Corporations are people -> People have feelings -> Corporations have feelings. (Transitive property son.) We don’t want Nutrisystem to hurt its own feelings. Self-sabotage never gets any of us very far. (Psychological insight bro.)
–
Getting more specific
Because I thought this paper was so solid, I’m going to quote what I highlighted, along with my annotations. I’ll be indenting the quotes, rather than using italics. Bolding is mine.
““80% of Americans (aged >6 y) were not at risk of deficiencies in any of the 7 vitamins” examined via biomarkers (ie, vitamins A, B , B , C, D, E, and folate; emphasis added). In addition, approximately 90% of women of childbearing age (12-49 years) were not at risk for iron deficiency, and folate levels have increased by approximately 50% since the previous national report. As such, most of the US population is not at risk for nutritional deficiencies, and neither do they have nutritional deficiencies and associated diseases.“
The crowd of “we’re overfed but undernourished” is overfed on illusion and undernourished on common sense. I’ve given evidence for this before, only to have some tell me, “I disagree.” As if this were opinion. It’s not. People are not crumbling while walking down the street due to rickets, nor are they looking like upright leopards and or spotted freaks due to scurvy. And if you are low on something, you can pretty much pop a pill and be done with it.
The authors start with the above to illustrate diet is not a major risk factor for disease anymore. I’m going to come back to this later, as I think this statement, and their rationale for it, ends up being…weird. Where they seem to discount the consequences of eating and heart disease, diabetes, etc. For now, the point is malnourishment is at worst a minor problem in the developed world. Therefore, not as much research or money should be put into this area. (A theme of this paper is less resources should be put into nutrition research, period. I’m not sure I’ve ever seen a researcher say their domain deserves less money!)
Despite the achievements of nutritional changes, like in America, the authors are still not happy with the state of nutritional research.
“The genesis of these criticisms is the appalling track record of highly publicized nutrition claims derived from epidemiologic studies (eg, see the studies by Stampfer et al and Rimm et al) that consistently failed to be supported when tested using objective study designs. Young and Karr examined more than 50 nutritional claims from observational studies for a variety of dietary patterns and nutrient supplementation and found that “100% of the observational claims failed to replicate” and that 5 claims were statistically significant “in the opposite direction.”
Like I said, these authors are coming out swinging. Appalling is about as strong a word as a scientist will use in the public domain. You can feel a “Son of a Biscuit!” coming soon.
“Epidemiologic studies suggest that almost any nutrient can be associated with a myriad of outcomes, as observed in Schoenfeld and Ioannidis’ article, “Is Everything We Eat Associated With Cancer?” […] Insofar as the provision of clear and consistent dietary guidelines for the consuming public is a goal of nutrition epidemiology, it has failed in decisively answering the simple question, “What should we eat?”
While the bolding is true, I want to pause here and say there is no answer to “What should we eat?” At least not anymore. Before all the deficiencies had been nearly fully eradicated, then sure, more specific answers to this question were around. Spotted freaks need vitamin C. But not anymore. It’s practically impossible to be malnourished in America, even if you’re in poverty. This is a question with too many potential answers, dealing with too many variables, that there is no specific answer.
The question to ask is, How should we eat? The implication being to move away from specific nutrients and move towards specific behaviors. One thing I think we can take away from all this crappy research is so many different routes have seemed fine. Low carbs, high carbs, low fat, high fat, vegan, not vegan. One of the amazing aspects of the body is its ability to adapt nutritionally. In the 21st century, when trying to lose weight, you have to be doing something truly insane, for a pretty long period of time, to cause bodily harm.
“Five decades of controversy surrounding basic dietary guidelines and nutrition recommendations is a public acknowledgement of a failed research paradigm.”
I don’t think you can criticize something more strongly in a scientific paper. I can’t recall ever reading a scientific paper where I felt so strongly the authors were actually humans and not robots i.e. they have an opinion, and are outright stating it. Good for them. But I do wonder if a whisper fight broke out after this was published.
I want to reiterate though, while the problems with nutrition research are significant, I do think there is an element of “well, there will always be controversy because so many routes are ok!”
–
First big issue (getting more specific)
“Memory-based dietary assessment methods (M-BMs) (eg, interviews, questionnaires, and surveys) are the dominant data collection protocols in national nutrition surveillance and government-funded epidemiologic nutrition and obesity research.”
This is the first major argument by the authors, and it’s one I couldn’t agree with more. SO much nutrition research revolves around asking people what they ate. This makes as much sense as the time I went off-roading in a golf cart. Yet,
“Importantly, M-BM data are used to inform national nutritional policy and dietary guidelines.”
it’s how we make recommendations!
“Although decades of unequivocal evidence demonstrate that the indirect, proxy estimates derived from M-BMs bear little relation to actual energy or nutrient consumption,”
Whoa, whoa, whoa. People don’t accurately remember, or report, how much they ate? I know, I had the same reaction.
I can’t find the paper, but I know one study, in nauseating detail, found people underestimate their calorie intake by 30-50%, which is a common range in other studies. That’s an enormous miscalculation.
“We assert that the explanatory and predictive failure of epidemiologic nutrition research is explained by its reliance on M-BMs, and, as such, the uncritical faith in the validity and value of M-BMs has wasted significant resources and constitutes the single greatest impediment to actual scientific progress in the fields of obesity and nutrition research.“
I agree and disagree with this statement. Resources have been wasted, memory based research is certainly an impediment, but the witch-hunt of what’s the one thing we should not eat to prevent weight gain, what’s the one thing we eat too much of causing weight gain, the continued pursuit to demonize everything but our own eating behavior (not ingredients), the avoidance of figuring out how to get people to eat less overall, is the single greatest impediment in my mind. Why we have this obsession with the minutiae of our eating, I don’t understand. This is not only the general population. Much of the science and reporting on the science revolves around this too. The authors mentioned there is no need to go hunting for more deficiencies. We’ve found them, we’ve cured them. Yet we continue the hunt for the weight gaining ingredient, which we’ve also already found. (Calories (energy).)
–
“the subjective (ie, private, not publicly accessible) mental phenomena (ie, memories) from which M-BM data are derived are not subject to independent observation, quantification, falsification, or verification; as such, M-BM data are pseudoscientific and inadmissible in scientific research.“
In order for something to be considered science, it needs to be falsifiable. Unless you record someone’s life, you can’t disprove what they say they’ve eaten. It’s not falsifiable. (Even if you did record their life, how do you know they aren’t eating differently now that they’re being observed?) Any nutritional study using this technique then is not science. Considering most research is published in scientific journals, much of nutrition research should not be published in these journals. Or they should be put in the “this is not science” section.
“determined that creation science was not a science because it was not falsifiable and, therefore, could not be taught as science in Arkansas public schools.”
I’m not looking to start a religious debate here. I thought the above was interesting as I didn’t know it.
I will say, many times the never ending arguments are about things that are not falsifiable. They are beliefs people have. Beliefs that can’t be questioned because there is no way to disprove them. Science is getting better at finding ways to assess many beliefs though. In the interim, it doesn’t mean the belief is wrong. It could very well be accurate. But there needs to be an acknowledgment of what it is- a belief. An opinion. Not science.
–
Second specific issue
“the failure to accurately and objectively measure and control for physical activity (PA), cardiorespiratory fitness (CRF), and other obvious confounders annuls inferences regarding diet-health relationships.”
My annotation after reading this sentence was, “YES! It’s all inadequate!” Because I can take this even further. Physical activity is typically defined as exercise. Meaning it’s planned activity. But physical activity is much more than that. From a categorical perspective, we define having a standing desk as not exercise in the same way sitting at your desk is not exercise. It’s not planned activity like going to the gym or going for a walk is. Same thing with walking the stairs vs taking the elevator, or when you park further away from the grocery store (sometimes because you have to). These things are never measured in this type of research.
Taking it even further, something like twitching your leg is not measured.
Or remember the person in school who was always tapping their thumb on the table? Or moving their pencil? “Just sit still!”
Twitching the leg, and these other movements, are categorized as NEAT: Non-Exercise Activity Thermogenesis. It’s activity, but a different kind. A kind we practically never measure. Done for 16+ hours everyday, it can add up to matter.
This is important because some people, after overeating, may be more likely to twitch their leg all day to help make up for that overconsumption. They don’t exercise more but they ARE more physically active. The “antsy” person if you will. If you don’t measure this -energy expenditure- how can you accurately say what’s going on from the energy ingestion side of things? Some people lose weight on a diet where some don’t, why? “Well, uh, some people respond to carbs differently.” “Some people can eat whatever they want #HighMetabolism.” Sure, maybe. But what about physical activity? Were some people, for some reason or another, more active after eating that certain way? What if “high metabolism” is primarily increased physical activity (but not exercise)? Guess what? You have no idea because you didn’t measure it!
-> Yes, I understand this is hard to study. (Though it’s becoming easier with how many different forms of movement sensors there are now, like your phone.) But I already discussed this element of research in the nazi post. That it has limitations. It’s worth mentioning again, but that’s not the point of this post. The point is this is never mentioned as a limitation of a nutritional study. When a limitation of your research is not realized and or not acknowledged, one that’s pretty obvious, it’s bad science.
–
“In other words, nutrition researchers designate numeric values to whatever the respondents are willing or able to recall about what they think (or would like the researcher to think) he or she consumed during the study period. Given the indirect, pseudoquantitative (ie, number-generating) nature of M-BMs and the fact that the respondents’ reports of their memories are subject to intentional and unintentional distorting factors (eg, perceptual, encoding, and retrieval errors; social desirability; false memories; and omissions), it is not surprising that most conclusions drawn from these number-generating protocols have not been supported when subjected to rigorous objective examination.“
While I think we all understand people don’t accurately remember what they eat, I don’t think we all understand how poor our memories generally are.
“Given that “[a] cross the 39-year history of the NHANES, [self-reported energy intake] data on the majority of respondents (67.3% of women and 58.7% of men) were not physiologically plausible”, we concluded that these data are not valid for any inferences regarding EI [energy intake] and the etiology of the obesity epidemic. A recent editorial in the British Medical Journal concurred and stated that the NHANES dietary data are “incompatible with life.””
BIGGEST POINT OF THIS POST AND PAPER: women lie more than men.
In all seriousness, the majority of people misreport their intake. It depends on the report, but based on the authors numbers, it’s just about always above 50% of responders, and sometimes near 90%. It’s unlikely there is one reason why. Whether faulty memory, lying, genuinely not knowing how many calories are in things, a woman’s propensity to manipulate, it’s probably a combination of everything. I think overall, overweight people know they eat too much. BUT, they probably don’t realize how much they overeat.
“underreporting increased to more than 70% for the entire NHANES sample and to approximately 77% and 85% for obese men and women, respectively.“
(I hope you see which gender is winning.) The heavier you are, the more your memory fails you, or the more you lie, or the more you don’t know how much you’re eating.
So, we have some idea of how many people underreport, but by how much?
“Men underreported EI [Energy Intake] 12% to 14% using the average of two 24HRs and 31% to 36% using FFQs. Women underreported by 16% to 20% using the average of two 24HRs and by 34% to 38% using FFQs.”
These numbers are a bit lower than the 30-50% I mentioned earlier, but they’re in the neighborhood. We’ll call it a third, give or take. If you think you eat 3000 calories, there is a good chance you eat ~4000.
“In 2015, a multinational report demonstrated that misreporting “in five populations of the African diaspora” was substantial, with the South African cohort exhibiting an astounding 52.1% underreporting of dietary EI. With respect to age, Forrestal found in children and adolescents that misreporting “appeared to be more common than it is among adults.”
I thought this was worth quoting as it illustrates this is not only Americans, or a specific culture we’re talking about. It seems to be a human phenomenon. We’re all a bunch of overweight, dementia like, deceivers.
I also noted here the age remarks make me think this could be more pure ignorance than deception. That as people get older, they have a better sense of how many calories are in something through experience, so they get better at not underreporting.
But then I read this:
“both macronutrient and micronutrient composition are significantly altered in underreporters, with reported fat and carbohydrate consumption often lower and reported protein, fruit, and vegetable intakes higher.”
Either people want others to think they eat better than they really do, or people are truly fooling themselves into how well they eat, or some combination.
“This simple fact renders energy adjustments fallacious and demonstrates that the assumption that M-BM data can be used to examine patterns of diet or dietary composition is not logically valid.”
I feel like this is how people at MIT insult one another.
- Me and friends at a bar- [for the 26th time tonight] “dude, you’re an idiot. You’re buying my next drink. And none of that mixed drink crap.”
- MIT bar- “Well, er, now, you know your argument is fallacious and demonstrates your logically invalid approach to the expanding universe. Can I get another Mai Tai please?”
When discussing the fallibility of human memory even more:
“Perhaps the most salient example of the fallibility of memory and recall (and misplaced confidence) is that false reporting (ie, inaccurate eyewitness testimony) was a key factor in approximately 75% of the first 100 cases of individuals exonerated by DNA evidence after conviction for crimes that they did not commit.”
Tying the criminal justice system to eating, judges have been found to act differently right before a lunch break compared to right after. Hangry judges are not who you want to have your case in front of.
“Nevertheless, one more notable example is warranted. Immediately on leaving a restaurant, Kronenfeld et al had participants report on the attire of the waitstaff and the restaurants’ choice of music. Participants demonstrated much greater agreement on what the waiters were wearing compared with the waitresses’ attire. The interesting finding was that these restaurants had an all-female waitstaff (ie, there were no waiters in the restaurants). Participants also provided much greater detail on the music from restaurants that were not playing music than from restaurants that were.”
This only reaffirms my belief humans have done some amazing things due to the few pounds between our ears, but on balance we are not that smart.
The authors mention despite how obviously faulty our memories are, the reliance on human memory hasn’t waned. It’s still used in nutritional research as much as ever. Studies like the above make one question how much we should be using our memories for a lot of things.
“Freeman et al demonstrated a 52% error rate in recalling social interactions,”
For years I’ve tried to come around to the “don’t bother counting calories” crowd. I understand it’s a pain, I understand it’s been around for a long time, so it’s not as new or sexy. For years now, short of eating the same thing everyday and adjusting from there, I have only come across more and more evidence illustrating the need to write things down and track it. At least temporarily. The more overweight you are, the more true this is. There is no way you can rely on memory for this stuff.
The authors go on at length about the way food questionnaires are conducted are even more conducive to causing memory issues. Due to things like using commonly associated words. Ask someone what they ate for breakfast, and it’s easy to misremember things when commonly associated breakfast foods are all listed as possible answers.
“Although the terms science and research are used interchangeably, they are not synonymous.”
More reason to not be enamored with, or tie every opinion you have, to research. It’s often opinion as much as any op-ed article is.
“Nevertheless, faith and belief are basic tenets of religion, not science.”
I liked this quote because faith and belief are what many eating paradigms are. Or how many go about their eating habits. They go based on their opinion, and because they say “it works for me,” or “it worked for this person I know,” you can’t falsify that. You can give rationale for why that’s not a good way to go about things, but often that ingrains the person even more in their belief. “They don’t get me. I know something they don’t. I miss Oprah.”
Put another way, you don’t see people arguing over whether gravity exists. Or whether the earth spins. But you see never ending debate on what people should eat.
“Yet, most nutrition research does not measure any form of energy expenditure or objectively quantify PA.”
“For example, PA, CRF [cardiorespiratory fitness], and exercise are not even listed on the National Institutes of Health’s spreadsheet of categorical spending of nearly 250 classifications through 2016. This is unfortunate given that 80% of Americans are not at risk for most nutritional deficiencies, but 95% of Americans are at risk for PA deficiency (ie, inactivity or high sedentary behavior) and do not meet the federal recommendations of 30 minutes per day of moderate to vigorous PA.”
This is as much of a wake up call as one can get when it comes to research resources. I’m going to sound like some Donald Trump cool-aid drinker (aka cyanide), but with research, you have a very tough time justifying needing more money when you’re not currently allocating the money in proportion to what’s impacting society. As of now, the nutrition world is on the “keep gettin’ dem checks” research all-star team.
The authors close their discussion with,
“we assert that research efforts and funding of M-BMs and diet-health research are misdirected and argue that those resources would be better targeted to the most prevalent disease of deficiency of the 21st century: inactivity (ie, a lack of PA and exercise and high levels of sedentary behavior).”
Can you say walk off?
–
Let’s get weird
The final conclusion is something I’m not sure about. I think it may be more a ground rule double than a home run.
The authors seem to gloss over eating behavior and the consequences of too much caloric ingestion, in favor of focusing on the other end, caloric expenditure. This seems primarily based on one paper, by one author, who was a co-author on the paper I’ve discussed here. This paper is weird but very interesting. It’s something I may go into in another post, but not here.
What I want to mention here is this conclusion oozes the “WHAT’S THE ONE THING WE NEED TO ADDRESS?” mantra I mentioned earlier. Where now we’re saying screw nutrition, it’s all physical activity.
The fact of the matter is places like America are approaching 70% overweight rate, with more than a third obese. If you’re obese, in a modern economy, gooooooooood luck generating enough physical activity to overcome that, without any dietary modification. Exercise can cause some to gain weight!
I was with the authors on nearly everything, but this point I really can’t get on board with. Scientists seem forever enamored with the simplicity of classical mechanics. Force? That’s only mass and acceleration. Velocity? Just distance and time. Often in classical mechanics, one variable is squared, so you can say while there are multiple variables, that squared variable is exponentially more important.
After 100 years plus of research, I’m not sure we’re ever going to simplify the body to this degree. “Move more and eat less” is about as simple as it’s going to get equation wise, but the variables within that equation are endless.
Humans are messy science. We’re hard science. We often qualify as pseudoscience. Physics doesn’t change over time (at least it doesn’t appear that way), but humans do. Culture, technology, biology, will all be different in 50 years. But F will still equal MA. We’ll need to perform new nutritional research, to find why people are eating the way they are, and what they’re eating. It’s tough enough. Let’s not make it tougher by obsessing over whatever the new trans fat equivalent is (that one bad thing), nor by still asking people “What did you eat last month?”
–
Posted on August 31, 2015