Why Didn’t the New Miracle Drug Work for Me? Statistical, Practical, and Clinical Significance
In recent years, there has been a large increase in the number of medications that are available to treat different types of psychological disorders. Yet many medications that are developed and tested, don’t make it to market as they are not found to be effective. It is assumed by many that drugs that become available to the public must successfully treat the disorder which they target. This may or may not be the case. True, they effectively treat the disorder in certain people. You just may not be one of them.
“How can this be?” you may be asking yourself. “I mean, I know not every medication helps everyone, but it has to help most people or it wouldn’t be on the market, right? There are studies that say it works. That proves that it effective, doesn’t it?”
The confusion here, lies in what research results prove and what they have to prove in order to be published in top journals and subsequently reported through media outlets that will disseminate the findings to the public. This is a matter of the difference between what is statistically significant and what is clinically or practically significant.
Types of Significance and the Results of Research Studies
Statistical significance focuses on the probability that an observed difference is not due to chance. The question answered by this type of significance is “Do two groups differ to the point that you can say the results are due to the treatment you imposed and not due to random external variable or chance? It isn’t concerned with the types of differences observed or whether or not they are meaningful to those being treated. It merely indicates that there are or aren’t differences that have resulted in the measured outcomes that weren’t random.
This type of significance is concerned with the size of the observed differences. This is a response to a problem with statistical significance. That is that statistical significance is heavily influenced by the sample size and can find even small differences between the group receiving the treatment and the control group if the sample is large enough. The problem with this is that it simply speaks to whether or not there are or aren’t differences between the groups not whether the difference are large enough to potentially impact people treated with the medication.
So say a research study is examining whether a medication decreases the symptoms of depression and the results show that the treatment group averaged a 3 point decrease on a self report rating scale while the placebo group averaged a 1 point decrease which has been shown to be statistically significant. This says that the decrease observed in the treatment group is significantly different from the decrease observed in the control group and that the difference is a real difference not due to chance.
However, while statistically significant, this may not be much of a noticeable difference for those receiving the treatment. Additional research may suggest that there needs to be a minimum of a 10 point decrease in ratings before people report a noticeable decrease in their depression.
Practical significance is concerned with whether or not there is a big enough difference in scores for there to be a noticeable difference for those treated. So just because something is statistically significant doesn’t mean it is a large enough difference to merit further study or to indicate effects are enough to impact patients.
While practical significance begins to address whether a new medication will decrease symptom severity enough be noticeable by those being treated, there is still a problem. Consider the example given above. Say that a study is conducted to examine whether a medication is effective for improving the symptoms of depression in hospitalized individuals. A 10 point average difference in scores is set as the cutoff for concluding that it is. The results show a 10 point decrease in symptoms for the the treatment group but not for the control group receiving the placebo.
However, what if the treated individuals still require hospitalization after the intervention? The difference between groups is found to be statistically significant so it is determined that is due to the treatment not to chance. The difference is also found to be practically significant because there is a 10 point difference between the groups which is considered large enough to be a noticeable difference.
But while those treated may report a noticeable difference in symptoms that doesn’t indicate that the differences would be considered clinically meaningful based on criteria set by health care practitioners. Clinical practitioners are not concerned with the size of the change that results from the treatment in the absence of other indicators of improvement. They are interested in whether patients will be able to live normal lives following treatment.
Thus, if all the treated patients are still hospitalized after treatment it wouldn’t matter if there was a 10 point, 20 point, or 50 point difference. They aren’t going to be able to live normal lives if they are still suffering from symptoms that are severe enough to warrant continued hospitalization. The ultimate indicator of clinical significance is whether or not those treated are still distinguishable from those without depression following treatment. Results are said to be clinically significant when the two groups cannot be differentiated based on their symptoms.
While each of these types of significance answer important questions, it is the last one, clinical significance, which is most relevant for those who are being treated. However, it can be argued that requiring there be no differences in symptoms between the treated group and those without a disorder for clinical significance to exist is too strict of a criterion by which to judge efficacy.
Ultimately, it is those who are suffering from a disorder who should determine what makes a meaningful difference in their lives. A medication may decrease symptoms to the point that those treated consider it an improvement in their quality of life, although they continue to experience some symptoms. It is not the place for clinical practitioners to decide that this isn’t significant because it didn’t result in the complete alleviation of all symptoms and this assumption may prevent a treatment from being used when it could help.
At the end of the day, it is up to the person being treated to decide what makes a big enough difference in their life to warrant taking a particular medication. For some a small decrease in symptoms may be enough for them to determine that taking the medication is worth it. For others, this same difference may not be big enough to justify it. Obviously, there are other factors that affect this decision such as side effects, length of treatment, and addictive properties among others. Yet it is not up to a clinical provider to make decisions for a patient. They should, instead, make decisions collaboratively with a patient to ensure the best possible outcomes
The Take Away
Just because a medication has been proven to be extremely effective for treating a certain problem does not mean it will automatically be effective for you. While many people may benefit from certain drugs, it’s important to keep in mind that no matter what criteria were used to determine efficacy, not everyone in any study was helped by it. Depending on what type of significance was used to evaluate the outcomes, even if it was found to make a difference for the majority of those in the treatment group, the difference may or may not have been meaningful to those actually treated.
Unfortunately, studies attempting to determine exactly what characteristics distinguish between those who were helped by a drug and those who weren’t, or what exactly constitutes a “meaningful change” in the minds of those being treated usually aren’t feasible. This is because there can be any number of possible characteristics that play a role in determining this and there must be enough people with each characteristic to be able to discover differences. It is difficult to recruit enough subjects with different characteristics for these types of studies and the expense and logistics of doing so are usually prohibitive.
What all this means is that while it is natural to get your hopes up when you discover there is a medication that can help you feel better physically and/or emotionally, if it doesn’t work don’t lose hope. There are any number of reasons this might be the case including the manner in which efficacy was determined and what led your practitioner to prescribe it.
As in any field, some health care providers are better than others. Some are better at listening to your specific experiences and symptoms and evaluating exactly what is going with you to determine the best course of action. Others may simply focus on the big picture, such as what general disorder you seem to be suffering from and then prescribe whatever is believed to be the best drug for the overall problem instead of based on your specific circumstances.
Keep in mind that today there are often numerous medications available to treat different disorders. It may take time to determine which one is right for you. But if you are patient and have a prescribing provider who works with you instead of dictating a course of action for you, together you should be able to find a treatment that significantly improves your quality of life. It is important to remain optimistic about the options available, including non-traditional routes of treatment, until you find what it is that will help you live a more satisfying, more productive and happier life.
This content is for informational purposes only and does not substitute for formal and individualized diagnosis, prognosis, treatment, prescription, and/or dietary advice from a licensed medical professional. Do not stop or alter your current course of treatment. If pregnant or nursing, consult with a qualified provider on an individual basis. Seek immediate help if you are experiencing a medical emergency.