This study is motivated by the potential problem of using observational data to draw inferences about treatment outcomes when experimental data are not available. We compare two statistical approaches, ordinary least-squares (OLS) and instrumental variables (IV) regression analysis, to estimate the outcomes (three-year post-treatment survival) of three treatments for early stage breast cancer in elderly women: mastectomy (MST), breast conserving surgery with radiation therapy (BCSRT), and breast conserving surgery only (BCSO). The primary data source was Medicare claims for a national random sample of 2907 women (age 67 or older) with localized breast cancer who were treated between 1992 and 1994. Contrary to randomized clinical trial (RCT) results, analysis with the observational data found highly significant differences in survival among the three treatment alternatives: 79.2% survival for BCSO, 85.3% for MST, and 93.0% for BCSRT. Using OLS to control for the effects of observable characteristics narrowed the estimated survival rate differences, which remained statistically significant. In contrast, the IV analysis estimated survival rate differences that were not significantly different from 0. However, the IV-point estimates of the treatment effects were quantitatively larger than the OLS estimates, unstable, and not significantly different from the OLS results. In addition, both sets of estimates were in the same quantitative range as the RCT results.We conclude that unadjusted observational data on health outcomes of alternative treatments for localized breast cancer should not be used for cost-effectiveness studies. Our comparisons suggest that whether one places greater confidence in the OLS or the IV results depends on at least three factors: (1) the extent of observable health information that can be used as controls in OLS estimation, (2) the outcomes of statistical tests of the validity of the instrumental variable method, and (3) the similarity of the OLS and IV estimates. In this particular analysis, the OLS estimates appear to be preferable because of the instability of the IV estimates.
Copyright 2002 John Wiley & Sons, Ltd.