'Statistically Significant' Doesn't Mean 'Right'

[Jeff Serven]

Something I have found funny of recent is people posting studies they don't read and assuming the claim is absolute fact. Or getting creative and using a sentence out of a book completely out of context. Ive been meaning to write a blog about this for a while but someone else did it for me!

http://www.bloombergview.com/articles/2016-03-18/-statistically-significant-doesn-t-mean-right

What are your thoughts? Do we really 'know what we know"? And, all of those other philosophical questions? 

This question is a bit of a slippery slope.. 

 

[Joachim Nagler]

I agree with this article. 

There's way too much focus on the p-value as an absolute measure in the scientific community. But as the article stated, it's just a probability of being wrong. For example, the comparison of two groups gives a p-value of 4%. This could mean there is a difference. It could also mean that the sample size was too small and that the value would change with more data points. 
And even quite a few scientists don't realize that. Just today I had to explain the concept to a colleague. But even if the scientist realizes it, often the journals won't accept papers that are done without a p-value calculation. 

So, it's a useful tool but one has to know it's meanings and limitations.

Note, I'm just scratching the surface of the rabbit hole that statistics can be and if the cases get more complicated I have to refer to the mathematicians in my institute.

Also, there's too much of a focus on positive results. Negative results are often not publishable, even though they are just as useful! So a lot of scientists only publish their best results. Maybe they tried 100 times and got it right once and published that. No wonder a lot of the studies are not reproducible!

[Toni Laukkavaara]

Its all about bang for the buck in studies. They're almost always done for mainstream purposes

[David Beckerman]

I work in the health profession so approach this idea from that perspective. We are often faced with news articles that report "X study shows Y causes Z disease". Much the same as nutrition it seems! It requires knowledge already in the area to fully understand the evidence. 

 

There are times when research cannot give us the definitive answer so it is left to the individual to interpret them in their own way. What I remember is that it is rare to have a sentinel stand alone study that can change current practice. It's more of an integration of all the research that we can find that helps me form an opinion. To be honest I don't put much weight on p value I am more leaning towards sensitivity and specificity.  

[Eduardo Paul]

On 18/03/2016 at 11:49 AM, Jeff Serven said:

Something I have found funny of recent is people posting studies they don't read and assuming the claim is absolute fact.

Way too common, especially if it supports previous beliefs...

About the statistics, most people don't realize how hard it is to draw conclusions from statistical data, especially in areas like biological or social sciences. They usually have a huge amount of variables, making it impossible to fully control the experiment. On top of that, the effects are usually not that big, making it hard to tell a real effect from a random result.

And this is true not just for the general population, but for the scientists themselves too. I'm working on my PhD. in Physics, and I can say how little statistics most physicists know (when compared with what you would expect them to know). Being the hardest of sciences, one would expect physicists to really be able to handle statistics, but that's not quite true. Apart from a few areas such as particle physics and cosmology, I would say most physicist's knowledge of statistics leaves a lot to be desired. Anyway, I digress...

On 18/03/2016 at 11:49 AM, Jeff Serven said:

What are your thoughts? Do we really 'know what we know"? And, all of those other philosophical questions?

In many instances, I think we do. These conflicts between studies, and trouble designing experiments, analyzing data, and reproducing results are part of the natural scientific process. In any area there are studies with pretty solid evidence. The major problem is just that so many times the media gets ahead of itself publishing preliminary or not so solid evidence as the new truth. And then when proper results arrive contradicting the first ideas, scientists get blamed for "not being able to make their minds..."

[Jeff Serven]

Eduardo,

Great post! Thank you.

Is there any indications or tell tells that someone without a PhD is physics can look for to read between the lines? You mention variables and control of the experiments, is that a place to start? 

I guess what Im asking is; what are a few key red flags in your personal experience?

Cheers,

Jeff

[Douglas Wadle]

So true, and a big part of why untrained persons (and, sorry to say, some scientists/physicians as well) do not understand medical science, or even science in general for that matter.  A number of excellent posts above with good posts.  One additional factor I wanted to raise is the difference between statistically significant, and clinically significant.  This is related in some ways to the difference between relative risk and absolute risk.  For instance, one intervention may decrease the risk of heart attack by 50%.  The p-value is < 0.0005.  This is a highly statistically significant finding, and in a prospective randomized trial should be taken seriously.  However, that 50% reduction may be lowering an event from 1.5% to 1%.  Is that absolute risk reduction of 0.5% clinically significant?  Probably not.  So that is not something I would take seriously, even though it is statistically significant and a good study.  

[Julien Le Nestour]

Being deeply involved in the academic field, this is completely true and I'd say most studies are most probably wrong on a number of dimensions, in every field. There are numerous examples but a good one on nutrition on how the sausage gets made between flawed research, to publication in (seemingly) respected academic journals and finally to worldwide PR in mainstream media is the following: http://io9.gizmodo.com/i-fooled-millions-into-thinking-chocolate-helps-weight-1707251800

Always fun to read such stuff except when you realize that the majority of all scientific research efforts (time, funding, etc.) is just wasted due to institutional misalignment of incentives...

[Julien Le Nestour]

6 hours ago, Jeff Serven said:

Is there any indications or tell tells that someone without a PhD is physics can look for to read between the lines? You mention variables and control of the experiments, is that a place to start? 

I guess what Im asking is; what are a few key red flags in your personal experience?

• small sample sizes for studies: if it seems small for your untrained eyes, it's probably too small scientifically as well, even though statistically it's ok
• no replication: nowadays, I don't take most studies seriously unless they've been replicated somewhat. Main issue here is that researchers in every field have almost no incentives to try and replicate exiting studies by opposition to doing new stuff and getting funding for it
• for studies on humans: how were participants recruited? are they just students in 1 field in 1 single US university? or they recruited off Amazon mechanical turk (and thus have all incentives to click as fast as possible on buttons) as in many psychology studies? most studies do not come close to having a representative sample to infer anything remotely general, yet most do.
• for studies on humans: what are the incentives for participants, assuming they are not willing participants (ie assume they don't care about doing the right thing for "Science")? For example, most economics / finance experimental studies ask students that are paid 5$ to do the study to play economic games that mimic real-life challenging decisions. Most will choose randomly if all that's at stake is science and they get the $5 at the end. Good studies will give them $100 if they get the game right: good design, but costly, so not a lot of people do.
• for studies at home on nutrition: assume people will forget to report things, lie if they don't follow the instructions, and generally will just do as they usually do if not actively forced to follow the study's instructions. I don't blame them, why should they make extra efforts in place of the scientists? So unless it's in a setting actively monitored 24/7, these studies are collecting deeply flawed data.

Sorry for the rant! That was just off the top of my head and I could go on all day long ;-) I won't even go into how an economic study out of Harvard and MIT, which has shaped policies worldwide since 2008, was based on an analysis that was run in excel on the wrong cells and was caught only in passing by a grad student looking to replicate it and brave enough to go public with his findings (very rare)...

[Bryant Wilson]

I work in medicine as well, and have to try to make sense of these sorts of studies, which can very difficult from a professional perspective, much less from a laymen's.  I don't believe you can look at the front page of a study and draw meaningful conclusions from it.  In truth, no part of the study is irrelevant.  To understand a study, you have to know the basic scientific issue with enough familiarity to "know what is unknown" (to quote nobody in particular), so that you know what question is trying to be investigated (in medical journals this is usually introduced briefly in the beginning background section).  The methods are crucial to understand, as well.  Who is being studied?  Is there a high, low, or mixed prevelance/risk of the trait/disease/intervention in this population?  What intervention is being studied?  What are the inclusion criteria?  What are the exclusion criteria?  What is being measured?  Is it en vivo or ex vivo?  Is it a true outcome or a surrogate measure of an outcome (where there are assumptions about it's relation to the true outcome that may be incorrect, incompletely understood, or utterly theoretical, unproven and quite possibly wrong)?  Then insert the statistical analysis issues here, as detailed in many previous posts, as well.  And whether the study is actually relevant to any random internet researcher becomes more and more debatable.  One thing I think is often overlooked is how many studies are set up not to be acted on by individuals, but to show a relationship to other researchers that justifies investing more time and money into future studies that may or may not be actionable.

If I could get one thing across to my patients or friends who like to do quick Google searches, it would be to not overestimate the importance of the "conclusion" or the p-value or the number needed to treat or any other individual statistic.  The conclusions of most studies do not answer grand and broad questions.  They "conclude" only the limited question posed in the thesis of the study.  They usually only answer a small question in a limited setting that may or may not be relevant to any individuals situation.  And how the conclusion fits into the grand scheme of things is usually highly debatable.  One look at the editorial section of most scientific journals would probably surprise many people who see these articles as much more definitive than they really are. 

[Jeff Serven]

Bryant, Douglas and Julien,

Very informative posts. Thank you!

Lets go to the flip side of the coin. Just for the sake of being thorough. Now that we have a good amount of information on how to asses a study. What is the good in taking the time to read these studies in full?

I think my first indication that studies weren't the end all be all was when I read the Nurses Heath Study. At face value it sounds great that the cohort was licensed nurses. You instantly think wow this is going to be accurate. But, then you look at the details such as diet; I cant remember if they were suppose to report what they ate for the past 5 day or the past year but either way is insanely inaccurate. After I read that I said to myself this has to be grossly inaccurate. I found some research (and later my personal experience corroborates) that basically said women wear rose tinted glasses when they look at the food they eat. Women significantly underestimate the frequency and amount of junk food they eat, the younger the woman the darker the tint (men have a different set of problems). So going back the the Nurses Heath Study - how could I trust any conclusion they come to in regards to nutrition. I could go on forever about the NHS...

On the flip side this study was the spark that got me thinking I could get anyone lean on any style of diet (high carb, low carb, etc). The study is not the end all be all its just something that really sparked the idea in my head that different styles of diets (vegetarian, paleo, low fat, etc) are just a different game and just need different rules to succeed. I used that methodology for a vey long time with people as to not interrupt the natural habitat. After a few years I realized the further away from Thrive (yes, Im extremely bias) the shorter they kept the results we had originally obtained. So, now Ive changed my tune slightly but at the time this study basically showed me that there is no one perfect diet, that 'how' people do is more important than 'what' people do. Note: this is my opinion, none of the above is actually in the study. 

http://jama.jamanetwork.com/article.aspx?articleid=205916

Anyway, I don't just want to bag on studies and research. I am not saying that is what anyone is doing. I just want to shift gears a bit and look at the glass half full now. Since you guys have such good info I want to extract all that I can. 

[Eduardo Paul]

Wow! This is a really nice discussion! Bryant, Douglas and Julien, did excellent jobs at summarizing many important points.

Another one I'd like to add is the difference in study designs.

There are the observational studies, in which a large group of people is basically observed living their lives as usual, and then you try to draw correlations among some variables, like the diet they followed and risk of dying from heart disease. The Nurses Health Study Jeff just cited is one example.

The observational studies are important but they lack the capacity to show any causal relationship between the variables being study. That is because of the huge number of variables that you are simply not taking into account that can bias your results. Just like that famous story of the number of ice-creams people in Australia take being correlated to shark attacks. The association may be real, but it doesn't mean one thing causes the other directly. In this example the common cause is the fact that, when it's hot, people tend to have more ice-creams and go more frequently to the beaches.

And then there are the controlled studies, in which you get a group of people randomly divided among two or more subgroup and then assign to each group one kind of intervention (or no intervention at all). It could be like comparing a low-carb to a high-carb diet. In these controlled studies if you see a correlation it's more likely that the intervention caused the effect.

So when you see some study, knowing whether it was observational or controlled is key.

Of course, there also the problem with how the media portray the studies, but that was pretty well discussed in the link Julien provided.

@Jeff Serven It's really disappointing when we start to realize those kind of things. But not all is lost. There are good researches pieces out there. We just have to educate ourselves to able to tell them from the garbage. I'll take a look at the paper you linked. It looks interesting.

[Douglas Wadle]

There is a ton of literature out there, so the question as to when to read the full study becomes very appropriate.  For me, I glance at the title first, is it even relevant to me or anything I care about.  If so, I read the abstract, is it something that just confirms what I already do, or is it something that will change my practice and philosophy about something.  If the latter, then it's time to read and dissect the study to see if it is as important as it seems in the abstract.  Then, if it does appear to be something big, I will read the editorials/letters about it the following week and see what the closest experts in that field are saying, to see if there was something I might have missed.  Only at that point will I be able to say whether I should change what I'm doing: maybe it is interesting, but more follow up research is needed; maybe it is not clinically useful for my specific interests; maybe I should start rethinking how I am doing something.  That is the process I use, and I find it to be an efficient use of my time, but also allows me to keep up with the firehose of information coming our way.

[Jeff Serven]

Yes, great discussion. Eduardo, Bryant, Douglas and Julien thank you. 

 

[Bryant Wilson]

1 hour ago, Douglas Wadle said:

There is a ton of literature out there, so the question as to when to read the full study becomes very appropriate.  For me, I glance at the title first, is it even relevant to me or anything I care about.  If so, I read the abstract, is it something that just confirms what I already do, or is it something that will change my practice and philosophy about something.  If the latter, then it's time to read and dissect the study to see if it is as important as it seems in the abstract.  Then, if it does appear to be something big, I will read the editorials/letters about it the following week and see what the closest experts in that field are saying, to see if there was something I might have missed.  Only at that point will I be able to say whether I should change what I'm doing: maybe it is interesting, but more follow up research is needed; maybe it is not clinically useful for my specific interests; maybe I should start rethinking how I am doing something.  That is the process I use, and I find it to be an efficient use of my time, but also allows me to keep up with the firehose of information coming our way.

I could write my own approach, but I would basically plagiarize the above:)  

You can only do this with so many topics before it would become overwhelming.  So if a topic is outside your scope, or if it doesn't motivate you to really research it in detail (both the basic science and the evolving literature), I find it more useful to search out some experts whose opinion you trust, who have a track record of success, and for the most part take their advice with perhaps just a touch of skepticism because, of course, everyone is wrong at least some of the time;)