-
Weighted trajectory analysis and application to clinical outcome assessment
Authors:
Utkarsh Chauhan,
Kaiqiong Zhao,
John Walker,
John R. Mackey
Abstract:
The Kaplan-Meier estimator (KM) is widely used in medical research to estimate the survival function from lifetime data. KM is a powerful tool to evaluate clinical trials due to simple computational requirements, a logrank hypothesis test, and the ability to censor patients. However, KM has several constraints and fails to generalize to ordinal variables of interest such as toxicity and ECOG perfo…
▽ More
The Kaplan-Meier estimator (KM) is widely used in medical research to estimate the survival function from lifetime data. KM is a powerful tool to evaluate clinical trials due to simple computational requirements, a logrank hypothesis test, and the ability to censor patients. However, KM has several constraints and fails to generalize to ordinal variables of interest such as toxicity and ECOG performance. We devised Weighted Trajectory Analysis (WTA) to combine the advantages of KM with the ability to compare treatment groups for ordinal variables and fluctuating outcomes. To assess statistical significance, we developed a new hypothesis test analogous to the logrank test. We demonstrate the functionality of WTA through 1000-fold clinical trial simulations of unique stochastic models of chemotherapy toxicity and schizophrenia progression. At several increments of sample size and hazard ratio, we compare the performance of WTA to both KM and Generalized Estimating Equations (GEE). WTA generally required half the sample size to achieve comparable power to KM; advantages over GEE include its robust non-parametric approach and summary plot. We also apply WTA to real clinical data: the toxicity outcomes of melanoma patients receiving immunotherapy and the disease progression of patients with metastatic breast cancer receiving ramucirumab. The application of WTA demonstrates that using traditional methods such as percent incidence and KM can lead to both Type I and II errors by failing to model illness trajectory. This article outlines a novel method for clinical outcome assessment that extends the advantages of Kaplan-Meier estimates to ordinal outcome variables.
△ Less
Submitted 7 June, 2023;
originally announced June 2023.
-
Tracking the State and Behavior of People in Response to COVID-1 19 Through the Fusion of Multiple Longitudinal Data Streams
Authors:
Mohamed Amine Bouzaghrane,
Hassan Obeid,
Drake Hayes,
Minnie Chen,
Meiqing Li,
Madeleine Parker,
Daniel A. Rodríguez,
Daniel G. Chatman,
Karen Trapenberg Frick,
Raja Sengupta,
Joan Walker
Abstract:
The changing nature of the COVID-19 pandemic has highlighted the importance of comprehensively considering its impacts and considering changes over time. Most COVID-19 related research addresses narrowly focused research questions and is therefore limited in addressing the complexities created by the interrelated impacts of the pandemic. Such research generally makes use of only one of either 1) a…
▽ More
The changing nature of the COVID-19 pandemic has highlighted the importance of comprehensively considering its impacts and considering changes over time. Most COVID-19 related research addresses narrowly focused research questions and is therefore limited in addressing the complexities created by the interrelated impacts of the pandemic. Such research generally makes use of only one of either 1) actively collected data such as surveys, or 2) passively collected data. While a few studies make use of both actively and passively collected data, only one other study collects it longitudinally. Here we describe a rich panel dataset of active and passive data from U.S. residents collected between August 2020 and July 2021. Active data includes a repeated survey measuring travel behavior, compliance with COVID-19 mandates, physical health, economic well-being, vaccination status, and other factors. Passively collected data consists of all locations visited by study participants, taken from smartphone GPS data. We also closely tracked COVID-19 policies across counties of residence throughout the study period. Such a dataset allows important research questions to be answered; for example, to determine the factors underlying the heterogeneous behavioral responses to COVID-19 restrictions imposed by local governments. Better information about such responses is critical to our ability to understand the societal and economic impacts of this and future pandemics. The development of this data infrastructure can also help researchers explore new frontiers in behavioral science. The article explains how this approach fills gaps in COVID-19 related data collection; describes the study design and data collection procedures; presents key demographic characteristics of study participants; and shows how fusing different data streams helps uncover behavioral insights.
△ Less
Submitted 1 October, 2022; v1 submitted 23 September, 2022;
originally announced September 2022.
-
Representation Learning via Invariant Causal Mechanisms
Authors:
Jovana Mitrovic,
Brian McWilliams,
Jacob Walker,
Lars Buesing,
Charles Blundell
Abstract:
Self-supervised learning has emerged as a strategy to reduce the reliance on costly supervised signal by pretraining representations only using unlabeled data. These methods combine heuristic proxy classification tasks with data augmentations and have achieved significant success, but our theoretical understanding of this success remains limited. In this paper we analyze self-supervised representa…
▽ More
Self-supervised learning has emerged as a strategy to reduce the reliance on costly supervised signal by pretraining representations only using unlabeled data. These methods combine heuristic proxy classification tasks with data augmentations and have achieved significant success, but our theoretical understanding of this success remains limited. In this paper we analyze self-supervised representation learning using a causal framework. We show how data augmentations can be more effectively utilized through explicit invariance constraints on the proxy classifiers employed during pretraining. Based on this, we propose a novel self-supervised objective, Representation Learning via Invariant Causal Mechanisms (ReLIC), that enforces invariant prediction of proxy targets across augmentations through an invariance regularizer which yields improved generalization guarantees. Further, using causality we generalize contrastive learning, a particular kind of self-supervised method, and provide an alternative theoretical explanation for the success of these methods. Empirically, ReLIC significantly outperforms competing methods in terms of robustness and out-of-distribution generalization on ImageNet, while also significantly outperforming these methods on Atari achieving above human-level performance on $51$ out of $57$ games.
△ Less
Submitted 15 October, 2020;
originally announced October 2020.
-
Deep learning for prediction of population health costs
Authors:
Philipp Drewe-Boss,
Dirk Enders,
Jochen Walker,
Uwe Ohler
Abstract:
Accurate prediction of healthcare costs is important for optimally managing health costs. However, methods leveraging the medical richness from data such as health insurance claims or electronic health records are missing. Here, we developed a deep neural network to predict future cost from health insurance claims records. We applied the deep network and a ridge regression model to a sample of 1.4…
▽ More
Accurate prediction of healthcare costs is important for optimally managing health costs. However, methods leveraging the medical richness from data such as health insurance claims or electronic health records are missing. Here, we developed a deep neural network to predict future cost from health insurance claims records. We applied the deep network and a ridge regression model to a sample of 1.4 million German insurants to predict total one-year health care costs. Both methods were compared to Morbi-RSA models with various performance measures and were also used to predict patients with a change in costs and to identify relevant codes for this prediction. We showed that the neural network outperformed the ridge regression as well as all Morbi-RSA models for cost prediction. Further, the neural network was superior to ridge regression in predicting patients with cost change and identified more specific codes. In summary, we showed that our deep neural network can leverage the full complexity of the patient records and outperforms standard approaches. We suggest that the better performance is due to the ability to incorporate complex interactions in the model and that the model might also be used for predicting other health phenotypes.
△ Less
Submitted 6 March, 2020;
originally announced March 2020.
-
Calibration of inexact computer models with heteroscedastic errors
Authors:
Chih-Li Sung,
Beau David Barber,
Berkley J. Walker
Abstract:
Computer models are commonly used to represent a wide range of real systems, but they often involve some unknown parameters. Estimating the parameters by collecting physical data becomes essential in many scientific fields, ranging from engineering to biology. However, most of the existing methods are developed under the assumption that the physical data contains homoscedastic measurement errors.…
▽ More
Computer models are commonly used to represent a wide range of real systems, but they often involve some unknown parameters. Estimating the parameters by collecting physical data becomes essential in many scientific fields, ranging from engineering to biology. However, most of the existing methods are developed under the assumption that the physical data contains homoscedastic measurement errors. Motivated by an experiment of plant relative growth rates where replicates are available, we propose a new calibration method for inexact computer models with heteroscedastic measurement errors. Asymptotic properties of the parameter estimators are derived, and a goodness-of-fit test is developed to detect the presence of heteroscedasticity. Numerical examples and empirical studies demonstrate that the proposed method not only yields accurate parameter estimation, but it also provides accurate predictions for physical data in the presence of both heteroscedasticity and model misspecification.
△ Less
Submitted 26 May, 2020; v1 submitted 25 October, 2019;
originally announced October 2019.
-
Time to Die: Death Prediction in Dota 2 using Deep Learning
Authors:
Adam Katona,
Ryan Spick,
Victoria Hodge,
Simon Demediuk,
Florian Block,
Anders Drachen,
James Alfred Walker
Abstract:
Esports have become major international sports with hundreds of millions of spectators. Esports games generate massive amounts of telemetry data. Using these to predict the outcome of esports matches has received considerable attention, but micro-predictions, which seek to predict events inside a match, is as yet unknown territory. Micro-predictions are however of perennial interest across esports…
▽ More
Esports have become major international sports with hundreds of millions of spectators. Esports games generate massive amounts of telemetry data. Using these to predict the outcome of esports matches has received considerable attention, but micro-predictions, which seek to predict events inside a match, is as yet unknown territory. Micro-predictions are however of perennial interest across esports commentators and audience, because they provide the ability to observe events that might otherwise be missed: esports games are highly complex with fast-moving action where the balance of a game can change in the span of seconds, and where events can happen in multiple areas of the playing field at the same time. Such events can happen rapidly, and it is easy for commentators and viewers alike to miss an event and only observe the following impact of events. In Dota 2, a player hero being killed by the opposing team is a key event of interest to commentators and audience. We present a deep learning network with shared weights which provides accurate death predictions within a five-second window. The network is trained on a vast selection of Dota 2 gameplay features and professional/semi-professional level match dataset. Even though death events are rare within a game (1\% of the data), the model achieves 0.377 precision with 0.725 recall on test data when prompted to predict which of any of the 10 players of either team will die within 5 seconds. An example of the system applied to a Dota 2 match is presented. This model enables real-time micro-predictions of kills in Dota 2, one of the most played esports titles in the world, giving commentators and viewers time to move their attention to these key events.
△ Less
Submitted 21 May, 2019;
originally announced June 2019.
-
Machine Learning Meets Microeconomics: The Case of Decision Trees and Discrete Choice
Authors:
Timothy Brathwaite,
Akshay Vij,
Joan L. Walker
Abstract:
We provide a microeconomic framework for decision trees: a popular machine learning method. Specifically, we show how decision trees represent a non-compensatory decision protocol known as disjunctions-of-conjunctions and how this protocol generalizes many of the non-compensatory rules used in the discrete choice literature so far. Additionally, we show how existing decision tree variants address…
▽ More
We provide a microeconomic framework for decision trees: a popular machine learning method. Specifically, we show how decision trees represent a non-compensatory decision protocol known as disjunctions-of-conjunctions and how this protocol generalizes many of the non-compensatory rules used in the discrete choice literature so far. Additionally, we show how existing decision tree variants address many economic concerns that choice modelers might have. Beyond theoretical interpretations, we contribute to the existing literature of two-stage, semi-compensatory modeling and to the existing decision tree literature. In particular, we formulate the first bayesian model tree, thereby allowing for uncertainty in the estimated non-compensatory rules as well as for context-dependent preference heterogeneity in one's second-stage choice model. Using an application of bicycle mode choice in the San Francisco Bay Area, we estimate our bayesian model tree, and we find that it is over 1,000 times more likely to be closer to the true data-generating process than a multinomial logit model (MNL). Qualitatively, our bayesian model tree automatically finds the effect of bicycle infrastructure investment to be moderated by travel distance, socio-demographics and topography, and our model identifies diminishing returns from bike lane investments. These qualitative differences lead to bayesian model tree forecasts that directly align with the observed bicycle mode shares in regions with abundant bicycle infrastructure such as Davis, CA and the Netherlands. In comparison, MNL's forecasts are overly optimistic.
△ Less
Submitted 13 November, 2017;
originally announced November 2017.
-
Consilience: A Holistic Measure of Goodness-of-Fit
Authors:
William H. Neill,
Ray H. Kamps,
Scott J. Walker,
Hsin-i Wu,
T. Scott Brandes,
Delbert M. Gatlin III,
Tiffany L. Hopper,
Robert R. Vega
Abstract:
We describe an apparently new measure of multivariate goodness-of-fit between sets of quantitative results from a model (simulation, analytical, or multiple regression), paired with those observed under corresponding conditions from the system being modeled. Our approach returns a single, integrative measure, even though it can accommodate complex systems that produce responses of M types. For eac…
▽ More
We describe an apparently new measure of multivariate goodness-of-fit between sets of quantitative results from a model (simulation, analytical, or multiple regression), paired with those observed under corresponding conditions from the system being modeled. Our approach returns a single, integrative measure, even though it can accommodate complex systems that produce responses of M types. For each response-type, the goodness-of-fit measure, which we label "Consilience" (C), is maximally 1, for perfect fit; near 0 for the large-sample case (number of pairs, N, more than about 25) in which the modeled series is a random sample from a quasi-normal distribution with the same mean and variance as that of the observed series (null model); and, less than 0, toward minus-infinity, for progressively worse fit. In addition, lack-of-fit for each response-type can be apportioned between systematic and non-systematic (unexplained) components of error. Finally, for statistical assessment of models relative to the equivalent null model, we offer provisional estimates of critical C vs. N, and of critical joint-C vs. N and M, at various levels of Pr(type-I error). Application of our proposed methodology requires only MS Excel (2003 or later); we provide Excel XLS and XLSX templates that afford semi-automatic computation for systems involving up to M = 5 response types, each represented by up to N = 1000 observed-and-modeled result pairs. N need not be equal, nor response pairs in complete overlap, over M.
△ Less
Submitted 20 October, 2018; v1 submitted 22 October, 2017;
originally announced October 2017.
-
Modeling and Forecasting the Evolution of Preferences over Time: A Hidden Markov Model of Travel Behavior
Authors:
Feras El Zarwi,
Akshay Vij,
Joan Walker
Abstract:
Literature suggests that preferences, as denoted by taste parameters and consideration sets, may evolve over time in response to changes in demographic and situational variables, psychological, sociological and biological constructs, and available alternatives and their attributes. However, existing representations typically overlook the influence of past experiences on present preferences. This s…
▽ More
Literature suggests that preferences, as denoted by taste parameters and consideration sets, may evolve over time in response to changes in demographic and situational variables, psychological, sociological and biological constructs, and available alternatives and their attributes. However, existing representations typically overlook the influence of past experiences on present preferences. This study develops, applies and tests a hidden Markov model with a discrete choice kernel to model and forecast the evolution of individual preferences and behaviors over long-range forecasting horizons. The hidden states denote different preferences i.e. modes considered in the choice set, and sensitivity to level-of-service attributes. The evolutionary path of those hidden states is hypothesized to be a first-order Markov process. The framework is applied to study the evolution of travel mode preferences, or modality styles, over time, in response to a major change in the public transportation system. We use longitudinal travel diary from Santiago, Chile. The dataset consists of four one-week pseudo travel diaries collected before and after the introduction of Transantiago, a complete redesign of the public transportation system in the city. Our model identifies four modality styles in the population: drivers, bus users, bus-metro users, and auto-metro users. The modality styles differ in terms of the travel modes that they consider and their sensitivity to level-of-service attributes. At the population level, there are significant shifts in the distribution of individuals across modality styles before and after the change in the system, but the distribution is relatively stable in the periods after the change. Finally, a comparison between the proposed dynamic framework and comparable static frameworks reveals differences in aggregate forecasts for different policy scenarios.
△ Less
Submitted 28 July, 2017;
originally announced July 2017.
-
A Discrete Choice Framework for Modeling and Forecasting The Adoption and Diffusion of New Transportation Services
Authors:
Feras El Zarwi,
Akshay Vij,
Joan Walker
Abstract:
Current travel demand models are unable to predict long-range trends in travel behavior as they do not entail a mechanism that projects membership and market share of new modes of transport (Uber, Lyft, etc). We propose integrating discrete choice and technology adoption models to address the aforementioned issue. In order to do so, we build on the formulation of discrete mixture models and specif…
▽ More
Current travel demand models are unable to predict long-range trends in travel behavior as they do not entail a mechanism that projects membership and market share of new modes of transport (Uber, Lyft, etc). We propose integrating discrete choice and technology adoption models to address the aforementioned issue. In order to do so, we build on the formulation of discrete mixture models and specifically Latent Class Choice Models (LCCMs), which were integrated with a network effect model. The network effect model quantifies the impact of the spatial/network effect of the new technology on the utility of adoption. We adopted a confirmatory approach to estimating our dynamic LCCM based on findings from the technology diffusion literature that focus on defining two distinct types of adopters: innovator/early adopters and imitators. LCCMs allow for heterogeneity in the utility of adoption for the various market segments i.e. innovators/early adopters, imitators and non-adopters. We make use of revealed preference (RP) time series data from a one-way carsharing system in a major city in the United States to estimate model parameters. The data entails a complete set of member enrollment for the carsharing service for a time period of 2.5 years after being launched. Consistent with the technology diffusion literature, our model identifies three latent classes whose utility of adoption have a well-defined set of preferences that are significant and behaviorally consistent. The technology adoption model predicts the probability that a certain individual will adopt the service at a certain time period, and is explained by social influences, network effect, socio-demographics and level-of-service attributes. Finally, the model was calibrated and then used to forecast adoption of the carsharing system for potential investment strategy scenarios.
△ Less
Submitted 23 July, 2017;
originally announced July 2017.
-
Causal Inference in Travel Demand Modeling (and the lack thereof)
Authors:
Timothy Brathwaite,
Joan Walker
Abstract:
This paper is about the general disconnect that we see, both in practice and in literature, between the disciplines of travel demand modeling and causal inference. In this paper, we assert that travel demand modeling should be one of the many fields that focuses on the production of valid causal inferences, and we hypothesize about reasons for the current disconnect between the two bodies of resea…
▽ More
This paper is about the general disconnect that we see, both in practice and in literature, between the disciplines of travel demand modeling and causal inference. In this paper, we assert that travel demand modeling should be one of the many fields that focuses on the production of valid causal inferences, and we hypothesize about reasons for the current disconnect between the two bodies of research. Furthermore, we explore the potential benefits of uniting these two disciplines. We consider what travel demand modeling can gain from greater incorporation of techniques and perspectives from the causal inference literatures, and we briefly discuss what the causal inference literature might gain from the work of travel demand modelers. In this paper, we do not attempt to "solve" issues related to the drawing of causal inferences from travel demand models. Instead, we hope to spark a larger discussion both within and between the travel demand modeling and causal inference literatures. In particular, we hope to incite discussion about the necessity of drawing causal inferences in travel demand applications and the methods by which one might credibly do so.
△ Less
Submitted 7 December, 2017; v1 submitted 22 June, 2017;
originally announced June 2017.
-
Asymmetric, Closed-Form, Finite-Parameter Models of Multinomial Choice
Authors:
Timothy Brathwaite,
Joan L. Walker
Abstract:
In transportation, the number of observations associated with one discrete outcome is often greatly different from the number of observations associated with another discrete outcome. This situation is known as class-imbalance. In statistics, one hypothesized explanation for class imbalance is the existence of data generating processes that are characterized by asymmetric (as opposed to typically…
▽ More
In transportation, the number of observations associated with one discrete outcome is often greatly different from the number of observations associated with another discrete outcome. This situation is known as class-imbalance. In statistics, one hypothesized explanation for class imbalance is the existence of data generating processes that are characterized by asymmetric (as opposed to typically symmetric) probability functions. Despite being a valid hypothesis for class-imbalanced choice situations, few simple models exist for testing this explanation in transportation settings---settings that are inherently multinomial. Our paper fills this gap. As such, it should be of interest to transportation scholars and practitioners alike.
Overall, we addressed the following questions: "how can one construct asymmetric, closed-form, finite-parameter models of multinomial choice" and "how do such models compare against commonly used symmetric models?" To do so, we (1) introduced a new class of closed-form, finite-parameter, multinomial choice models that we call "logit-type models," (2) introduced a procedure for using our logit-type models to extend existing binary choice models to the multinomial setting, and (3) introduced a procedure for creating new binary choice models (both symmetric and asymmetric). Together, our contributions allow us to create new asymmetric, multinomial choice models by creating multinomial extensions of asymmetric, binary choice models that already exist or that we create ourselves. We demonstrated our methods by developing four new asymmetric, multinomial choice models. We found that most of our asymmetric models dominated the multinomial logit (MNL) model in terms of in-sample and out-of-sample log-likelihoods. Moreover, on our two empirical applications, we also found practical differences between the MNL model and our new asymmetric models.
△ Less
Submitted 8 February, 2018; v1 submitted 19 June, 2016;
originally announced June 2016.
