Showing 1–5 of 5 results for author: Fernández-Loría, C

Causal Fine-Tuning and Effect Calibration of Non-Causal Predictive Models

Authors: Carlos Fernández-Loría, Yanfang Hou, Foster Provost, Jennifer Hill

Abstract: This paper proposes techniques to enhance the performance of non-causal models for causal inference using data from randomized experiments. In domains like advertising, customer retention, and precision medicine, non-causal models that predict outcomes under no intervention are often used to score individuals and rank them according to the expected effectiveness of an intervention (e.g, an ad, a r… ▽ More This paper proposes techniques to enhance the performance of non-causal models for causal inference using data from randomized experiments. In domains like advertising, customer retention, and precision medicine, non-causal models that predict outcomes under no intervention are often used to score individuals and rank them according to the expected effectiveness of an intervention (e.g, an ad, a retention incentive, a nudge). However, these scores may not perfectly correspond to intervention effects due to the inherent non-causal nature of the models. To address this limitation, we propose causal fine-tuning and effect calibration, two techniques that leverage experimental data to refine the output of non-causal models for different causal tasks, including effect estimation, effect ordering, and effect classification. They are underpinned by two key advantages. First, they can effectively integrate the predictive capabilities of general non-causal models with the requirements of a causal task in a specific context, allowing decision makers to support diverse causal applications with a "foundational" scoring model. Second, through simulations and an empirical example, we demonstrate that they can outperform the alternative of building a causal-effect model from scratch, particularly when the available experimental data is limited and the non-causal scores already capture substantial information about the relative sizes of causal effects. Overall, this research underscores the practical advantages of combining experimental data with non-causal models to support causal applications. △ Less

Submitted 13 June, 2024; originally announced June 2024.

Inferring Effect Ordering Without Causal Effect Estimation

Authors: Carlos Fernández-Loría, Jorge Loría

Abstract: Predictive models are often employed to guide interventions across various domains, such as advertising, customer retention, and personalized medicine. These models often do not estimate the actual effects of interventions but serve as proxies, suggesting potential effectiveness based on predicted outcomes. Our paper addresses the critical question of when and how these predictive models can be in… ▽ More Predictive models are often employed to guide interventions across various domains, such as advertising, customer retention, and personalized medicine. These models often do not estimate the actual effects of interventions but serve as proxies, suggesting potential effectiveness based on predicted outcomes. Our paper addresses the critical question of when and how these predictive models can be interpreted causally, specifically focusing on using the models for inferring effect ordering rather than precise effect sizes. We formalize two assumptions, full latent mediation and latent monotonicity, that are jointly sufficient for inferring effect ordering without direct causal effect estimation. We explore the utility of these assumptions in assessing the feasibility of proxies for inferring effect ordering in scenarios where there is no data on how individuals behave when intervened or no data on the primary outcome of interest. Additionally, we provide practical guidelines for practitioners to make their own assessments about proxies. Our findings reveal not only when it is possible to reasonably infer effect ordering from proxies, but also conditions under which modeling these proxies can outperform direct effect estimation. This study underscores the importance of broadening causal inference to encompass alternative causal interpretations beyond effect estimation, offering a foundation for future research to enhance decision-making processes when direct effect estimation is not feasible. △ Less

Submitted 15 August, 2024; v1 submitted 24 June, 2022; originally announced June 2022.

Causal Decision Making and Causal Effect Estimation Are Not the Same... and Why It Matters

Authors: Carlos Fernández-Loría, Foster Provost

Abstract: Causal decision making (CDM) based on machine learning has become a routine part of business. Businesses algorithmically target offers, incentives, and recommendations to affect consumer behavior. Recently, we have seen an acceleration of research related to CDM and causal effect estimation (CEE) using machine-learned models. This article highlights an important perspective: CDM is not the same as… ▽ More Causal decision making (CDM) based on machine learning has become a routine part of business. Businesses algorithmically target offers, incentives, and recommendations to affect consumer behavior. Recently, we have seen an acceleration of research related to CDM and causal effect estimation (CEE) using machine-learned models. This article highlights an important perspective: CDM is not the same as CEE, and counterintuitively, accurate CEE is not necessary for accurate CDM. Our experience is that this is not well understood by practitioners or most researchers. Technically, the estimand of interest is different, and this has important implications both for modeling and for the use of statistical models for CDM. We draw on prior research to highlight three implications. (1) We should consider carefully the objective function of the causal machine learning, and if possible, optimize for accurate treatment assignment rather than for accurate effect-size estimation. (2) Confounding does not have the same effect on CDM as it does on CEE. The upshot is that for supporting CDM it may be just as good or even better to learn with confounded data as with unconfounded data. Finally, (3) causal statistical modeling may not be necessary to support CDM because a proxy target for statistical modeling might do as well or better. This third observation helps to explain at least one broad common CDM practice that seems wrong at first blush: the widespread use of non-causal models for targeting interventions. The last two implications are particularly important in practice, as acquiring (unconfounded) data on all counterfactuals can be costly and often impracticable. These observations open substantial research ground. We hope to facilitate research in this area by pointing to related articles from multiple contributing fields, including two dozen articles published the last three to four years. △ Less

Submitted 30 September, 2021; v1 submitted 8 April, 2021; originally announced April 2021.

A Comparison of Methods for Treatment Assignment with an Application to Playlist Generation

Authors: Carlos Fernández-Loría, Foster Provost, Jesse Anderton, Benjamin Carterette, Praveen Chandar

Abstract: This study presents a systematic comparison of methods for individual treatment assignment, a general problem that arises in many applications and has received significant attention from economists, computer scientists, and social scientists. We group the various methods proposed in the literature into three general classes of algorithms (or metalearners): learning models to predict outcomes (the… ▽ More This study presents a systematic comparison of methods for individual treatment assignment, a general problem that arises in many applications and has received significant attention from economists, computer scientists, and social scientists. We group the various methods proposed in the literature into three general classes of algorithms (or metalearners): learning models to predict outcomes (the O-learner), learning models to predict causal effects (the E-learner), and learning models to predict optimal treatment assignments (the A-learner). We compare the metalearners in terms of (1) their level of generality and (2) the objective function they use to learn models from data; we then discuss the implications that these characteristics have for modeling and decision making. Notably, we demonstrate analytically and empirically that optimizing for the prediction of outcomes or causal effects is not the same as optimizing for treatment assignments, suggesting that in general the A-learner should lead to better treatment assignments than the other metalearners. We demonstrate the practical implications of our findings in the context of choosing, for each user, the best algorithm for playlist generation in order to optimize engagement. This is the first comparison of the three different metalearners on a real-world application at scale (based on more than half a billion individual treatment assignments). In addition to supporting our analytical findings, the results show how large A/B tests can provide substantial value for learning treatment assignment policies, rather than simply choosing the variant that performs best on average. △ Less

Submitted 30 April, 2022; v1 submitted 24 April, 2020; originally announced April 2020.

Explaining Data-Driven Decisions made by AI Systems: The Counterfactual Approach

Authors: Carlos Fernández-Loría, Foster Provost, Xintian Han

Abstract: We examine counterfactual explanations for explaining the decisions made by model-based AI systems. The counterfactual approach we consider defines an explanation as a set of the system's data inputs that causally drives the decision (i.e., changing the inputs in the set changes the decision) and is irreducible (i.e., changing any subset of the inputs does not change the decision). We (1) demonstr… ▽ More We examine counterfactual explanations for explaining the decisions made by model-based AI systems. The counterfactual approach we consider defines an explanation as a set of the system's data inputs that causally drives the decision (i.e., changing the inputs in the set changes the decision) and is irreducible (i.e., changing any subset of the inputs does not change the decision). We (1) demonstrate how this framework may be used to provide explanations for decisions made by general, data-driven AI systems that may incorporate features with arbitrary data types and multiple predictive models, and (2) propose a heuristic procedure to find the most useful explanations depending on the context. We then contrast counterfactual explanations with methods that explain model predictions by weighting features according to their importance (e.g., SHAP, LIME) and present two fundamental reasons why we should carefully consider whether importance-weight explanations are well-suited to explain system decisions. Specifically, we show that (i) features that have a large importance weight for a model prediction may not affect the corresponding decision, and (ii) importance weights are insufficient to communicate whether and how features influence decisions. We demonstrate this with several concise examples and three detailed case studies that compare the counterfactual approach with SHAP to illustrate various conditions under which counterfactual explanations explain data-driven decisions better than importance weights. △ Less

Submitted 13 October, 2021; v1 submitted 21 January, 2020; originally announced January 2020.