Clinical significance: Difference between revisions
m Quick-adding category Medical statistics (using HotCat) |
m Cleaned up using AutoEd |
||
Line 1: | Line 1: | ||
Clinical significance was first proposed by Jacobson, Follette, and Revenstorf <ref> |
Clinical significance was first proposed by Jacobson, Follette, and Revenstorf <ref> |
||
Jacobson, N.S., Follette, W.C., and Revenstorf, D. (1984). Psychotherapy outcome research: Methods for reporting variability and evaluating clinical significance. Behavior Therapy, 15(4).</ref> as a way to answer the question, is a therapy or treatment effective enough such that a client does not meet the criteria for a diagnosis? Jacobson and Truax later defined clinical significance as “the extent to which therapy moves someone outside the range of the dysfunctional population or within the range of the functional population.”<ref>Jacobson, N., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59(1), 12-19.</ref> They proposed two components of this index of change: the status of a patient or client after therapy has been completed, and “how much change has occurred during the course of therapy.” <ref> Jacobson, N., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59(1), 12-19.</ref> |
Jacobson, N.S., Follette, W.C., and Revenstorf, D. (1984). Psychotherapy outcome research: Methods for reporting variability and evaluating clinical significance. Behavior Therapy, 15(4).</ref> as a way to answer the question, is a therapy or treatment effective enough such that a client does not meet the criteria for a diagnosis? Jacobson and Truax later defined clinical significance as “the extent to which therapy moves someone outside the range of the dysfunctional population or within the range of the functional population.”<ref>Jacobson, N., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59(1), 12-19.</ref> They proposed two components of this index of change: the status of a patient or client after therapy has been completed, and “how much change has occurred during the course of therapy.” <ref> Jacobson, N., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59(1), 12-19.</ref> |
||
Clinical significance is also a consideration when interpreting the results of the [[psychological assessment]] of an individual. |
Clinical significance is also a consideration when interpreting the results of the [[psychological assessment]] of an individual. Frequently, there will be a difference of scores or subscores that is [[Statistical significance|statistically significant]], unlikely to have occurred purely by chance. However, not all of those statistically significant differences are clinically significant, in that they do not either explain existing information about the client, or provide useful direction for intervention. Differences that are small in magnitude typically lack practical relevance and are unlikely to be clinically significant. Differences that are common in the population are also unlikely to be clinically significant, because they may simply reflect a level of normal human variation. Additionally, clinicians look for information in the assessment data and the client's history that corroborates the relevance of the statistical difference, to establish the connection between performance on the specific test and the individual's more general functioning.<ref>Sattler JM (2008). Assessment of children: Cognitive foundations (5/e). San Diego: Sattler Publications. ISBN 978-0-9702671-6-0</ref><ref>Kaufman AS & Lichtenberger EO (2005). Assessing adolescent and adult intelligence. NY: Wiley. ISBN 0471735531</ref> |
||
==Comparison to other calculations of significance== |
==Comparison to other calculations of significance== |
||
Line 13: | Line 13: | ||
Clinical significance and practical significance are often erroneously used interchangeably<ref> Peterson, L. (2008). "Clinical" Significance: "Clinical" Significance and "Practical" Significance are NOT the Same Things. Online Submission, Paper presented at the Annual Meeting of the Southwest Educational Research Association (New Orleans, LA, Feb 7, 2008).</ref>. Practical significance yields quantified information about the importance of a finding. Effect size is one type of practical significance<ref>Peterson, L. (2008). "Clinical" Significance: "Clinical" Significance and "Practical" Significance are NOT the Same Things. Online Submission, Paper presented at the Annual Meeting of the Southwest Educational Research Association (New Orleans, LA, Feb 7, 2008).</ref>. It quantifies the extent to which a sample diverges from expectations<ref>Vacha-Hasse, T., Nilsson, J.E., Reetz, D.R., Lance, T.S., & Thompson, B. (2000). Reporting practices and APA editorial policies regarding statistical significance and effect size. Theory & Psychology, 10, 413-425.</ref>. In terms of testing clinical treatments, practical significance answers the question, ''how effective'' is the treatment, or how much change does the treatment cause? Effect size can provide important information about the results of a study, and are now recommended for inclusion in addition to statistical significance, but effect sizes have their own sources of bias, are subject to change based on population variability of the dependent variable, and tend to focus on group effects, not individual changes<ref>Wilkinson, L., & APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594- 604. </ref><ref>Cohen, J. (1997). The earth is round (p < .05). The American Psychologist, 49 (12), 997-1003.</ref><ref> Peterson, L. (2008). "Clinical" Significance: "Clinical" Significance and "Practical" Significance are NOT the Same Things. Online Submission, Paper presented at the Annual Meeting of the Southwest Educational Research Association (New Orleans, LA, Feb 7, 2008).</ref>. |
Clinical significance and practical significance are often erroneously used interchangeably<ref> Peterson, L. (2008). "Clinical" Significance: "Clinical" Significance and "Practical" Significance are NOT the Same Things. Online Submission, Paper presented at the Annual Meeting of the Southwest Educational Research Association (New Orleans, LA, Feb 7, 2008).</ref>. Practical significance yields quantified information about the importance of a finding. Effect size is one type of practical significance<ref>Peterson, L. (2008). "Clinical" Significance: "Clinical" Significance and "Practical" Significance are NOT the Same Things. Online Submission, Paper presented at the Annual Meeting of the Southwest Educational Research Association (New Orleans, LA, Feb 7, 2008).</ref>. It quantifies the extent to which a sample diverges from expectations<ref>Vacha-Hasse, T., Nilsson, J.E., Reetz, D.R., Lance, T.S., & Thompson, B. (2000). Reporting practices and APA editorial policies regarding statistical significance and effect size. Theory & Psychology, 10, 413-425.</ref>. In terms of testing clinical treatments, practical significance answers the question, ''how effective'' is the treatment, or how much change does the treatment cause? Effect size can provide important information about the results of a study, and are now recommended for inclusion in addition to statistical significance, but effect sizes have their own sources of bias, are subject to change based on population variability of the dependent variable, and tend to focus on group effects, not individual changes<ref>Wilkinson, L., & APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594- 604. </ref><ref>Cohen, J. (1997). The earth is round (p < .05). The American Psychologist, 49 (12), 997-1003.</ref><ref> Peterson, L. (2008). "Clinical" Significance: "Clinical" Significance and "Practical" Significance are NOT the Same Things. Online Submission, Paper presented at the Annual Meeting of the Southwest Educational Research Association (New Orleans, LA, Feb 7, 2008).</ref>. |
||
In contrast, clinical significance yields information on whether a treatment was effective enough to change a patient’s diagnostic label. In terms of clinical treatment studies, clinical significance answers the question, is a treatment effective enough to cause the patient to be normal? |
In contrast, clinical significance yields information on whether a treatment was effective enough to change a patient’s diagnostic label. In terms of clinical treatment studies, clinical significance answers the question, is a treatment effective enough to cause the patient to be normal? |
||
For example, a treatment might significantly change depressive symptoms (significance), the change could be a large decrease in depressive symptoms (practical significance- effect size), and 40% of the patients no longer met the diagnostic criteria for depression (clinical significance). It is very possible to have a treatment that yields a significant difference and medium or large effect sizes, but does not move a patient from dysfunctional to functional. |
For example, a treatment might significantly change depressive symptoms (significance), the change could be a large decrease in depressive symptoms (practical significance- effect size), and 40% of the patients no longer met the diagnostic criteria for depression (clinical significance). It is very possible to have a treatment that yields a significant difference and medium or large effect sizes, but does not move a patient from dysfunctional to functional. |
Revision as of 21:51, 19 July 2009
Clinical significance was first proposed by Jacobson, Follette, and Revenstorf [1] as a way to answer the question, is a therapy or treatment effective enough such that a client does not meet the criteria for a diagnosis? Jacobson and Truax later defined clinical significance as “the extent to which therapy moves someone outside the range of the dysfunctional population or within the range of the functional population.”[2] They proposed two components of this index of change: the status of a patient or client after therapy has been completed, and “how much change has occurred during the course of therapy.” [3]
Clinical significance is also a consideration when interpreting the results of the psychological assessment of an individual. Frequently, there will be a difference of scores or subscores that is statistically significant, unlikely to have occurred purely by chance. However, not all of those statistically significant differences are clinically significant, in that they do not either explain existing information about the client, or provide useful direction for intervention. Differences that are small in magnitude typically lack practical relevance and are unlikely to be clinically significant. Differences that are common in the population are also unlikely to be clinically significant, because they may simply reflect a level of normal human variation. Additionally, clinicians look for information in the assessment data and the client's history that corroborates the relevance of the statistical difference, to establish the connection between performance on the specific test and the individual's more general functioning.[4][5]
Comparison to other calculations of significance
Statistical significance
Statistical significance tends to be used in the context of null hypothesis significance testing (NHST). NHST answers the question, if a hypothesis that an effect is zero in the population is true (the null hypothesis), what is the probability of obtaining data that indicate the effect is not zero[6]? NHST is often misunderstood in several ways: that the p-value is the probability that the null hypothesis is false; that it is related to probability of replication; and that if the null hypothesis is rejected, the proposed alternative hypothesis should be accepted. Given the nature of NHST, and its common misuse, statistical significance does not yield information about magnitude of effect, practical significance, nor clinical significance [7]. NHST only yields information about if results are likely given one assumption about the population[8]. In terms of testing clinical treatments, statistical significance can only indicate an answer to this question: if a treatment is actualy ineffective, how likely it is that the statistical test of the treatment would erroneously indicate that the treatment is effective?
Practical significance and effect size
Clinical significance and practical significance are often erroneously used interchangeably[9]. Practical significance yields quantified information about the importance of a finding. Effect size is one type of practical significance[10]. It quantifies the extent to which a sample diverges from expectations[11]. In terms of testing clinical treatments, practical significance answers the question, how effective is the treatment, or how much change does the treatment cause? Effect size can provide important information about the results of a study, and are now recommended for inclusion in addition to statistical significance, but effect sizes have their own sources of bias, are subject to change based on population variability of the dependent variable, and tend to focus on group effects, not individual changes[12][13][14].
In contrast, clinical significance yields information on whether a treatment was effective enough to change a patient’s diagnostic label. In terms of clinical treatment studies, clinical significance answers the question, is a treatment effective enough to cause the patient to be normal?
For example, a treatment might significantly change depressive symptoms (significance), the change could be a large decrease in depressive symptoms (practical significance- effect size), and 40% of the patients no longer met the diagnostic criteria for depression (clinical significance). It is very possible to have a treatment that yields a significant difference and medium or large effect sizes, but does not move a patient from dysfunctional to functional.
Calculation of clinical significance
Just as there are many ways to calculate statistical significance and practical significance, there are a variety of ways to calculate clinical significance. Five common methods are the Jacobson-Truax method, the Gulliksen-Lord-Novick method, the Edwards-Nunnally method, the Hageman-Arrindell method, and hierarchical linear modeling[15].
Jacobson-Truax
Jacobson-Truax is common method of calculating clinical significance. It involves calculating a Reliability Change Index (RCI)[16]. The RCI equals the difference between a participant’s pre-test and post-test scores, divided by the standard error of the difference. Cutoff scores are established for placing participants into one of four categories- recovered, improved, unchanged, or deteriorated- depending on the directionality of the RCI and whether the cutoff score was met.
Gulliksen-Lord-Novick
The Gulliksen-Lord-Novick method[17] is similar to Jacobson-Truax, except that it takes into account regression to the mean. This is done by subtracting the pre-test and post-test scores from a population mean, and dividing by the standard deviation of the population [18].
Edwards-Nunnally
The Edwards-Nunnally method[19] of calculating clinical significance is a more stringent alternative to the Jacobson-Truax method[20]. Reliability scores are used to bring the pre-test scores closer to the mean, and then a confidence interval is developed for this adjusted pre-test score. Confidence intervals are used when calculating the change from pre-test to post-test, so greater actual change in scores is necessary to show clinical significance, compared to the Jacobson-Truax method.
Hageman-Arrindel
The Hageman-Arrindel[21] calculation of clinical significance involves indices of group change and of individual change. The reliability of change indicates whether a patient has improved, stayed the same, or deteriorated. A second index, the clinical significance of change, indicates four categories similar to those used by Jacobson-Truax: deteriorated, not reliably changed, improved but not recovered, and recovered.
Hierarchical Linear Modeling (HLM)
HLM involves growth curve analysis instead of pre-test post-test comparisons, so three data points are needed from each patient, instead of only two data points (pre-test and post-test) [22]. A computer program, such as Hierarchical Linear and Nonlinear Modeling[23] is used to calculate change estimates for each participant. HLM also allows for analysis of growth curve models of dyads and groups.
References
- ^ Jacobson, N.S., Follette, W.C., and Revenstorf, D. (1984). Psychotherapy outcome research: Methods for reporting variability and evaluating clinical significance. Behavior Therapy, 15(4).
- ^ Jacobson, N., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59(1), 12-19.
- ^ Jacobson, N., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59(1), 12-19.
- ^ Sattler JM (2008). Assessment of children: Cognitive foundations (5/e). San Diego: Sattler Publications. ISBN 978-0-9702671-6-0
- ^ Kaufman AS & Lichtenberger EO (2005). Assessing adolescent and adult intelligence. NY: Wiley. ISBN 0471735531
- ^ Cohen, J. (1997). The earth is round (p < .05). The American Psychologist, 49 (12), 997-1003.
- ^ Haase, R.F., Ellis, M.V., Ladany, N. (1989).Multiple Criteria for Evaluating the Magnitude of Experimental Effects. Journal of Counseling Psychology, 36(4), 511-516.
- ^ "Clinical" Significance: "Clinical" Significance and "Practical" Significance are NOT the Same Things. Online Submission, Paper presented at the Annual Meeting of the Southwest Educational Research Association (New Orleans, LA, Feb 7, 2008).
- ^ Peterson, L. (2008). "Clinical" Significance: "Clinical" Significance and "Practical" Significance are NOT the Same Things. Online Submission, Paper presented at the Annual Meeting of the Southwest Educational Research Association (New Orleans, LA, Feb 7, 2008).
- ^ Peterson, L. (2008). "Clinical" Significance: "Clinical" Significance and "Practical" Significance are NOT the Same Things. Online Submission, Paper presented at the Annual Meeting of the Southwest Educational Research Association (New Orleans, LA, Feb 7, 2008).
- ^ Vacha-Hasse, T., Nilsson, J.E., Reetz, D.R., Lance, T.S., & Thompson, B. (2000). Reporting practices and APA editorial policies regarding statistical significance and effect size. Theory & Psychology, 10, 413-425.
- ^ Wilkinson, L., & APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594- 604.
- ^ Cohen, J. (1997). The earth is round (p < .05). The American Psychologist, 49 (12), 997-1003.
- ^ Peterson, L. (2008). "Clinical" Significance: "Clinical" Significance and "Practical" Significance are NOT the Same Things. Online Submission, Paper presented at the Annual Meeting of the Southwest Educational Research Association (New Orleans, LA, Feb 7, 2008).
- ^ Peterson, L. (2008). "Clinical" Significance: "Clinical" Significance and "Practical" Significance are NOT the Same Things. Online Submission, Paper presented at the Annual Meeting of the Southwest Educational Research Association (New Orleans, LA, Feb 7, 2008).
- ^ Jacobson, N., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59(1), 12-19.
- ^ Hsu, L. M. (1999). A comparison of three methods of identifying reliable and clinically significant client changes: commentary on Hageman and Arrindell. Behaviour Research and Therapy, 37, 1195-1202.
- ^ Peterson, L. (2008). "Clinical" Significance: "Clinical" Significance and "Practical" Significance are NOT the Same Things. Online Submission, Paper presented at the Annual Meeting of the Southwest Educational Research Association (New Orleans, LA, Feb 7, 2008).
- ^ Speer, D. C., & Greenbaum, P. E. (1995). Five methods for computing significant individual client change and improvement rates: Support for an individual growth curve approach. Journal of Consulting and Clinical Psychology, 63, 1044-1048.
- ^ Peterson, L. (2008). "Clinical" Significance: "Clinical" Significance and "Practical" Significance are NOT the Same Things. Online Submission, Paper presented at the Annual Meeting of the Southwest Educational Research.
- ^ Hageman, W. J., & Arrindell, W. A. (1999). Establishing clinically significant change: increment of precision and the distinction between individual and group level of analysis. Behaviour Research and Therapy, 37, 1169-1193.
- ^ Peterson, L. (2008). "Clinical" Significance: "Clinical" Significance and "Practical" Significance are NOT the Same Things. Online Submission, Paper presented at the Annual Meeting of the Southwest Educational Research.
- ^ http://www.ssicentral.com/hlm/index.html