This week, Su and Jonathan discuss how to evaluate medical studies. When a “breakthrough” hits the news,
can you tell if it really is one? Some tips.
Su Robotti: Hi, I’m Su Robotti and I’m the founder of MedShadow.
Jonathan Block: Hi and I’m Jonathan Block, I’m the content manager here at MedShadow.
SR: Just recently, Jonathan was at a health journalist conference, where one of the topics we found very interesting and that is, how do you know that you can trust the content of an article in a newspaper, in a magazine? How does the consumer know? And Jonathan, what did you learn about medical studies?
JB: Well, it was very interesting because the panel that I was listening to, one of the editors at healthnewsreview.org which were very fond of here at MedShadow was talking about this. And it’s true, you are inundated with all of these articles that seem to say, “Oh this drug does this or this supplement does this,” and you really have to read into the article a little bit to find out whether you can actually believe what’s being said. And one of the first things you want to know is if the article is based on a study, how exactly it was powered. Did it actually have a substantial number of people who were involved in this study? If there was a study of looking at a particular medication, was there a control group, as in a placebo control group, that sort of things. So these are some of the basics right off the bat.
SR: But one of the key indicators that you maybe you should think twice about an article is when the headlines says, “Breakthrough,” or you know, “New Discovery,” or “Cure,. “You know, there aren’t that many breakthroughs in medicine. Medicine is a science of increments. You learn a little bit more and a little bit more in these medical studies.
JB: Right, it’s interesting that you bring that up because one of the artist in the actual article that I believe it was on nbcnews.com that the editor from HealthNewsReview brought up was something that had to deal with — it said, cure for kidney stones discovered on roller coaster, which of course sounds very, very interesting, and–
SR: I would click that article —
JB: You definitely would, and it was actually very interesting to study that basically these researchers, they took — kind of made — a kidney out of some kind of plastic that’s supposed to replicate what at an actual kidney is like they put in substance that’s very similar to a kidney, brought it with them on a roller coaster in Orlando, which was where the Association of Healthcare Journalists conference was, and they noticed that after a few times the kidney stone — the fake kidney stone inside this fake liver — actually started moving further down as if that’s a way that you could potentially get rid of kidney stones is to go on roller coaster a few times. And the way that the article was positioned had made it sound like that was a cure. In fact, it was actually used through in the headline. And you really can’t say that something is a cure when you bring a plastic kidney on a roller coaster.
SR: Well, by the logic of that study, a pregnant woman could get right through labor and delivery very quickly by riding a roller coaster, but I’m not sure her obstetrician would suggest that.
JB: No, obviously, you wouldn’t do that at all. One of the things mentioned during this particular panel was that you have to look at the perceived cost of a particular intervention. Does the article actually address that.
Along similar lines, if you read an article that seems to tell all of the benefits that were seen in a particular trial, you would want to also know, well what are some of the drug’s, what are some of the side effects, which of course, our viewers are very interested in. And what was going on that — how does this particular treatment or study, whatever was examined, and how does it deal with current options that are out there, because you could have a great new, you know, drug that comes out, but if there are so many different side effects or if it’s prohibitively expensive, how useful is it really.
So the bottom line is that when you’re reading one of these articles about a study, you should really have a bunch of questions in your mind as you are reading it so that you can really determine whether this is an actual breakthrough or whether it’s some kind of a study that’s come out and perhaps whoever wrote a news release on it is trying to kind of spin the results.
SR: The core basis of any science is that it is reproducible. Any experiment should be able to done again in a science quest by students or by other scientists in another way. If it can’t be reproduced, then possibly something was going oddly in the experiment, but it does mean it can’t be taken as a scientific fact if you can’t reproduce it. So many of these studies that we see are—it’s the first time that it’s been studied, so it hasn’t been, nobody has tried to duplicate it yet. Or it’s an outliers study. Nine studies or 99 studies showed X, but one study showed Y, so you do want to figure out, if you can, is this study that seems like a breakthrough study, is it really just an outliers’ study, and it hasn’t been reproduced yet, and it goes against what 99 other studies said. All things to keep in mind as you look at these wonderful breakthrough world of medicine articles.
JB: Right. The other thing that’s also important to mention is that, which is what I like to say, is follow the money. What I mean by that is whenever I write a story, for example for the news section of our website, I want to see who’s funding it. Now if it’s funded by, for example, the National Institutes of Health or a university, I find that a little bit more plausible. But when I look at medical studies that I’m going write about for MedShadow, and I think that there’s something that says this was funded by a drug company or perhaps some sort of an association that maybe has something to benefit out of a positive outcome, first of all, I’m going to mention that in the article. And then secondly, I’m going to look at the study results even more critically.
SR: Right. Scientists who were being paid by a company that might benefit by the results of that study doesn’t mean that the results of the study are invalid or useless, but they must be seen in context and with those parameters around it.
JB: Exactly.
SR: Any wise last words, because I think we’ve covered everything?
JB: No. I mean, just as I was mentioning throughout our conversation today is that, you know, really when you’re reading these articles, don’t just take what’s written in the headline or on the first paragraph, at face value. Really read through the entire article, and if there’s a link to the actual study or the primary source, I would encourage you to go and click on it and just make sure that you have a full picture of the study because frequently in an article, if you’re only getting a portion about what the study found, you might be missing out a lot of other important information that might actually undermine what is in the article you’re reading itself.
SR: Good, thanks Jonathan, we’ll have to send you to more conferences—
JB: I hope so, thank you.
SR: And please, go to our website, at www.medshadow.org for more information on the risks and the benefits of the medicines that you take.
