Why Didn’t the New Miracle Drug Work for Me? Statistical, Practical, and Clinical Significance

Updated on March 21, 2019
Natalie Frank profile image

Natalie Frank, a Ph.D. in Clinical Psychology, specializes in pediatrics, behavioral health and stress and coping in everyday life.

In recent years, there has been a large increase in the number of medications that are available to treat different types of psychological disorders. Yet many medications that are developed and tested, don’t make it to market as they are not found to be effective. It is assumed by many that drugs that become available to the public must successfully treat the disorder which they target. This may or may not be the case. True, they effectively treat the disorder in certain people. You just may not be one of them.

“How can this be?” you may be asking yourself. “I mean, I know not every medication helps everyone, but it has to help most people or it wouldn’t be on the market, right? There are studies that say it works. That proves that it effective, doesn’t it?”

The confusion here, lies in what research results prove and what they have to prove in order to be published in top journals and subsequently reported through media outlets that will disseminate the findings to the public. This is a matter of the difference between what is statistically significant and what is clinically or practically significant.

Types of Significance and the Results of Research Studies

Statistical Significance

Statistical significance focuses on the probability that an observed difference is not due to chance. The question answered by this type of significance is “Do two groups differ to the point that you can say the results are due to the treatment you imposed and not due to random external variable or chance? It isn’t concerned with the types of differences observed or whether or not they are meaningful to those being treated. It merely indicates that there are or aren’t differences that have resulted in the measured outcomes that weren’t random.

Practical Significance

This type of significance is concerned with the size of the observed differences. This is a response to a problem with statistical significance. That is that statistical significance is heavily influenced by the sample size and can find even small differences between the group receiving the treatment and the control group if the sample is large enough. The problem with this is that it simply speaks to whether or not there are or aren’t differences between the groups not whether the difference are large enough to potentially impact people treated with the medication.

So say a research study is examining whether a medication decreases the symptoms of depression and the results show that the treatment group averaged a 3 point decrease on a self report rating scale while the placebo group averaged a 1 point decrease which has been shown to be statistically significant. This says that the decrease observed in the treatment group is significantly different from the decrease observed in the control group and that the difference is a real difference not due to chance.

However, while statistically significant, this may not be much of a noticeable difference for those receiving the treatment. Additional research may suggest that there needs to be a minimum of a 10 point decrease in ratings before people report a noticeable decrease in their depression.

Practical significance is concerned with whether or not there is a big enough difference in scores for there to be a noticeable difference for those treated. So just because something is statistically significant doesn’t mean it is a large enough difference to merit further study or to indicate effects are enough to impact patients.

Clinical Significance

While practical significance begins to address whether a new medication will decrease symptom severity enough be noticeable by those being treated, there is still a problem. Consider the example given above. Say that a study is conducted to examine whether a medication is effective for improving the symptoms of depression in hospitalized individuals. A 10 point average difference in scores is set as the cutoff for concluding that it is. The results show a 10 point decrease in symptoms for the the treatment group but not for the control group receiving the placebo.

However, what if the treated individuals still require hospitalization after the intervention? The difference between groups is found to be statistically significant so it is determined that is due to the treatment not to chance. The difference is also found to be practically significant because there is a 10 point difference between the groups which is considered large enough to be a noticeable difference.

But while those treated may report a noticeable difference in symptoms that doesn’t indicate that the differences would be considered clinically meaningful based on criteria set by health care practitioners. Clinical practitioners are not concerned with the size of the change that results from the treatment in the absence of other indicators of improvement. They are interested in whether patients will be able to live normal lives following treatment.

Thus, if all the treated patients are still hospitalized after treatment it wouldn’t matter if there was a 10 point, 20 point, or 50 point difference. They aren’t going to be able to live normal lives if they are still suffering from symptoms that are severe enough to warrant continued hospitalization. The ultimate indicator of clinical significance is whether or not those treated are still distinguishable from those without depression following treatment. Results are said to be clinically significant when the two groups cannot be differentiated based on their symptoms.

Conclusions

While each of these types of significance answer important questions, it is the last one, clinical significance, which is most relevant for those who are being treated. However, it can be argued that requiring there be no differences in symptoms between the treated group and those without a disorder for clinical significance to exist is too strict of a criterion by which to judge efficacy.

Ultimately, it is those who are suffering from a disorder who should determine what makes a meaningful difference in their lives. A medication may decrease symptoms to the point that those treated consider it an improvement in their quality of life, although they continue to experience some symptoms. It is not the place for clinical practitioners to decide that this isn’t significant because it didn’t result in the complete alleviation of all symptoms and this assumption may prevent a treatment from being used when it could help.

At the end of the day, it is up to the person being treated to decide what makes a big enough difference in their life to warrant taking a particular medication. For some a small decrease in symptoms may be enough for them to determine that taking the medication is worth it. For others, this same difference may not be big enough to justify it. Obviously, there are other factors that affect this decision such as side effects, length of treatment, and addictive properties among others. Yet it is not up to a clinical provider to make decisions for a patient. They should, instead, make decisions collaboratively with a patient to ensure the best possible outcomes

The Take Away

Just because a medication has been proven to be extremely effective for treating a certain problem does not mean it will automatically be effective for you. While many people may benefit from certain drugs, it’s important to keep in mind that no matter what criteria were used to determine efficacy, not everyone in any study was helped by it. Depending on what type of significance was used to evaluate the outcomes, even if it was found to make a difference for the majority of those in the treatment group, the difference may or may not have been meaningful to those actually treated.

Unfortunately, studies attempting to determine exactly what characteristics distinguish between those who were helped by a drug and those who weren’t, or what exactly constitutes a “meaningful change” in the minds of those being treated usually aren’t feasible. This is because there can be any number of possible characteristics that play a role in determining this and there must be enough people with each characteristic to be able to discover differences. It is difficult to recruit enough subjects with different characteristics for these types of studies and the expense and logistics of doing so are usually prohibitive.

What all this means is that while it is natural to get your hopes up when you discover there is a medication that can help you feel better physically and/or emotionally, if it doesn’t work don’t lose hope. There are any number of reasons this might be the case including the manner in which efficacy was determined and what led your practitioner to prescribe it.

As in any field, some health care providers are better than others. Some are better at listening to your specific experiences and symptoms and evaluating exactly what is going with you to determine the best course of action. Others may simply focus on the big picture, such as what general disorder you seem to be suffering from and then prescribe whatever is believed to be the best drug for the overall problem instead of based on your specific circumstances.

Keep in mind that today there are often numerous medications available to treat different disorders. It may take time to determine which one is right for you. But if you are patient and have a prescribing provider who works with you instead of dictating a course of action for you, together you should be able to find a treatment that significantly improves your quality of life. It is important to remain optimistic about the options available, including non-traditional routes of treatment, until you find what it is that will help you live a more satisfying, more productive and happier life.

This content is for informational purposes only and does not substitute for formal and individualized diagnosis, prognosis, treatment, prescription, and/or dietary advice from a licensed medical professional. Do not stop or alter your current course of treatment. If pregnant or nursing, consult with a qualified provider on an individual basis. Seek immediate help if you are experiencing a medical emergency.

Questions & Answers

    Comments

      0 of 8192 characters used
      Post Comment
      • profile image

        Miebakagh57 

        7 months ago

        Hi, Natalie, thanks for sharing.

      • profile image

        Lorna Lamon 

        8 months ago

        Very interesting and informative article. It's always a good idea to research - finding the right provider is also key. Thank you for sharing.

      working

      This website uses cookies

      As a user in the EEA, your approval is needed on a few things. To provide a better website experience, healthproadvice.com uses cookies (and other similar technologies) and may collect, process, and share personal data. Please choose which areas of our service you consent to our doing so.

      For more information on managing or withdrawing consents and how we handle data, visit our Privacy Policy at: https://healthproadvice.com/privacy-policy#gdpr

      Show Details
      Necessary
      HubPages Device IDThis is used to identify particular browsers or devices when the access the service, and is used for security reasons.
      LoginThis is necessary to sign in to the HubPages Service.
      Google RecaptchaThis is used to prevent bots and spam. (Privacy Policy)
      AkismetThis is used to detect comment spam. (Privacy Policy)
      HubPages Google AnalyticsThis is used to provide data on traffic to our website, all personally identifyable data is anonymized. (Privacy Policy)
      HubPages Traffic PixelThis is used to collect data on traffic to articles and other pages on our site. Unless you are signed in to a HubPages account, all personally identifiable information is anonymized.
      Amazon Web ServicesThis is a cloud services platform that we used to host our service. (Privacy Policy)
      CloudflareThis is a cloud CDN service that we use to efficiently deliver files required for our service to operate such as javascript, cascading style sheets, images, and videos. (Privacy Policy)
      Google Hosted LibrariesJavascript software libraries such as jQuery are loaded at endpoints on the googleapis.com or gstatic.com domains, for performance and efficiency reasons. (Privacy Policy)
      Features
      Google Custom SearchThis is feature allows you to search the site. (Privacy Policy)
      Google MapsSome articles have Google Maps embedded in them. (Privacy Policy)
      Google ChartsThis is used to display charts and graphs on articles and the author center. (Privacy Policy)
      Google AdSense Host APIThis service allows you to sign up for or associate a Google AdSense account with HubPages, so that you can earn money from ads on your articles. No data is shared unless you engage with this feature. (Privacy Policy)
      Google YouTubeSome articles have YouTube videos embedded in them. (Privacy Policy)
      VimeoSome articles have Vimeo videos embedded in them. (Privacy Policy)
      PaypalThis is used for a registered author who enrolls in the HubPages Earnings program and requests to be paid via PayPal. No data is shared with Paypal unless you engage with this feature. (Privacy Policy)
      Facebook LoginYou can use this to streamline signing up for, or signing in to your Hubpages account. No data is shared with Facebook unless you engage with this feature. (Privacy Policy)
      MavenThis supports the Maven widget and search functionality. (Privacy Policy)
      Marketing
      Google AdSenseThis is an ad network. (Privacy Policy)
      Google DoubleClickGoogle provides ad serving technology and runs an ad network. (Privacy Policy)
      Index ExchangeThis is an ad network. (Privacy Policy)
      SovrnThis is an ad network. (Privacy Policy)
      Facebook AdsThis is an ad network. (Privacy Policy)
      Amazon Unified Ad MarketplaceThis is an ad network. (Privacy Policy)
      AppNexusThis is an ad network. (Privacy Policy)
      OpenxThis is an ad network. (Privacy Policy)
      Rubicon ProjectThis is an ad network. (Privacy Policy)
      TripleLiftThis is an ad network. (Privacy Policy)
      Say MediaWe partner with Say Media to deliver ad campaigns on our sites. (Privacy Policy)
      Remarketing PixelsWe may use remarketing pixels from advertising networks such as Google AdWords, Bing Ads, and Facebook in order to advertise the HubPages Service to people that have visited our sites.
      Conversion Tracking PixelsWe may use conversion tracking pixels from advertising networks such as Google AdWords, Bing Ads, and Facebook in order to identify when an advertisement has successfully resulted in the desired action, such as signing up for the HubPages Service or publishing an article on the HubPages Service.
      Statistics
      Author Google AnalyticsThis is used to provide traffic data and reports to the authors of articles on the HubPages Service. (Privacy Policy)
      ComscoreComScore is a media measurement and analytics company providing marketing data and analytics to enterprises, media and advertising agencies, and publishers. Non-consent will result in ComScore only processing obfuscated personal data. (Privacy Policy)
      Amazon Tracking PixelSome articles display amazon products as part of the Amazon Affiliate program, this pixel provides traffic statistics for those products (Privacy Policy)
      ClickscoThis is a data management platform studying reader behavior (Privacy Policy)