-
A Large-Scale Neutral Comparison Study of Survival Models on Low-Dimensional Data
Authors:
Lukas Burk,
John Zobolas,
Bernd Bischl,
Andreas Bender,
Marvin N. Wright,
Raphael Sonabend
Abstract:
This work presents the first large-scale neutral benchmark experiment focused on single-event, right-censored, low-dimensional survival data. Benchmark experiments are essential in methodological research to scientifically compare new and existing model classes through proper empirical evaluation. Existing benchmarks in the survival literature are often narrow in scope, focusing, for example, on h…
▽ More
This work presents the first large-scale neutral benchmark experiment focused on single-event, right-censored, low-dimensional survival data. Benchmark experiments are essential in methodological research to scientifically compare new and existing model classes through proper empirical evaluation. Existing benchmarks in the survival literature are often narrow in scope, focusing, for example, on high-dimensional data. Additionally, they may lack appropriate tuning or evaluation procedures, or are qualitative reviews, rather than quantitative comparisons. This comprehensive study aims to fill the gap by neutrally evaluating a broad range of methods and providing generalizable conclusions. We benchmark 18 models, ranging from classical statistical approaches to many common machine learning methods, on 32 publicly available datasets. The benchmark tunes for both a discrimination measure and a proper scoring rule to assess performance in different settings. Evaluating on 8 survival metrics, we assess discrimination, calibration, and overall predictive performance of the tested models. Using discrimination measures, we find that no method significantly outperforms the Cox model. However, (tuned) Accelerated Failure Time models were able to achieve significantly better results with respect to overall predictive performance as measured by the right-censored log-likelihood. Machine learning methods that performed comparably well include Oblique Random Survival Forests under discrimination, and Cox-based likelihood-boosting under overall predictive performance. We conclude that for predictive purposes in the standard survival analysis setting of low-dimensional, right-censored data, the Cox Proportional Hazards model remains a simple and robust method, sufficient for practitioners.
△ Less
Submitted 6 June, 2024;
originally announced June 2024.
-
Navigating Cultural Diversity: Barriers and Potentials in Multicultural Agile Software Development Teams
Authors:
Daniel Welsch,
Luisa Burk,
David Mötefindt,
Michael Neumann
Abstract:
Context: Social aspects are of high importance for being successful using agile methods in software development. People are influenced by their cultural imprint, as the underlying cultural values are guiding us in how we think and act. Thus, one may assume that in multicultural agile software development teams, cultural characteristics influence the result in terms of quality of the team work and…
▽ More
Context: Social aspects are of high importance for being successful using agile methods in software development. People are influenced by their cultural imprint, as the underlying cultural values are guiding us in how we think and act. Thus, one may assume that in multicultural agile software development teams, cultural characteristics influence the result in terms of quality of the team work and consequently, the product to be delivered. Objective: We aim to identify barriers and potentials that may arise in multicultural agile software development teams to provide valuable strategies for both researchers and practitioners faced with barriers or unrealized potentials of cultural diversity. Method: The study is designed as a single-case study with two units of analysis using a mixed-method design consisting quantitative and qualitative methods. Results: First, our results suggest that the cultural characteristics at the team level need to be analyzed individually in intercultural teams, Second, we identified key potentials regarding cultural characteristics providing key potentials such as a individual team subculture that fits agile values like open communication. Third, we derived strategies supporting the potentials of cultural diversity in agile software development teams. Conclusion: Our findings show, that a deeper understanding of cultural influences in multicultural agile software development teams is needed. Based on the results, we already prepare future work to validate the results in other industries.
△ Less
Submitted 18 November, 2023;
originally announced November 2023.
-
Examining properness in the external validation of survival models with squared and logarithmic losses
Authors:
Raphael Sonabend,
John Zobolas,
Philipp Kopper,
Lukas Burk,
Andreas Bender
Abstract:
Scoring rules promote rational and honest decision-making, which is becoming increasingly important for automated procedures in `auto-ML'. In this paper we survey common squared and logarithmic scoring rules for survival analysis and determine which losses are proper and improper. We prove that commonly utilised squared and logarithmic scoring rules that are claimed to be proper are in fact improp…
▽ More
Scoring rules promote rational and honest decision-making, which is becoming increasingly important for automated procedures in `auto-ML'. In this paper we survey common squared and logarithmic scoring rules for survival analysis and determine which losses are proper and improper. We prove that commonly utilised squared and logarithmic scoring rules that are claimed to be proper are in fact improper, such as the Integrated Survival Brier Score (ISBS). We further prove that under a strict set of assumptions a class of scoring rules is strictly proper for, what we term, `approximate' survival losses. Despite the difference in properness, experiments in simulated and real-world datasets show there is no major difference between improper and proper versions of the widely-used ISBS, ensuring that we can reasonably trust previous experiments utilizing the original score for evaluation purposes. We still advocate for the use of proper scoring rules, as even minor differences between losses can have important implications in automated processes such as model tuning. We hope our findings encourage further research into the properties of survival measures so that robust and honest evaluation of survival models can be achieved.
△ Less
Submitted 3 June, 2024; v1 submitted 10 December, 2022;
originally announced December 2022.
-
Flexible Group Fairness Metrics for Survival Analysis
Authors:
Raphael Sonabend,
Florian Pfisterer,
Alan Mishler,
Moritz Schauer,
Lukas Burk,
Sumantrak Mukherjee,
Sebastian Vollmer
Abstract:
Algorithmic fairness is an increasingly important field concerned with detecting and mitigating biases in machine learning models. There has been a wealth of literature for algorithmic fairness in regression and classification however there has been little exploration of the field for survival analysis. Survival analysis is the prediction task in which one attempts to predict the probability of an…
▽ More
Algorithmic fairness is an increasingly important field concerned with detecting and mitigating biases in machine learning models. There has been a wealth of literature for algorithmic fairness in regression and classification however there has been little exploration of the field for survival analysis. Survival analysis is the prediction task in which one attempts to predict the probability of an event occurring over time. Survival predictions are particularly important in sensitive settings such as when utilising machine learning for diagnosis and prognosis of patients. In this paper we explore how to utilise existing survival metrics to measure bias with group fairness metrics. We explore this in an empirical experiment with 29 survival datasets and 8 measures. We find that measures of discrimination are able to capture bias well whereas there is less clarity with measures of calibration and scoring rules. We suggest further areas for research including prediction-based fairness metrics for distribution predictions.
△ Less
Submitted 22 July, 2022; v1 submitted 26 May, 2022;
originally announced June 2022.
-
Is Patent Law Technology Specific?
Authors:
Dan L. Burk,
Mark A. Lemley
Abstract:
Although patent law purports to cover all manner of technologies, we have noticed recent divergence in the standards applied to biotechnology and to software patents: the Federal Circuit has applied a very permissive standard of obviousness in biotechnology, but a highly restrictive disclosure requirement. The opposite holds true for software patents, which seems to us exactly contrary to sound…
▽ More
Although patent law purports to cover all manner of technologies, we have noticed recent divergence in the standards applied to biotechnology and to software patents: the Federal Circuit has applied a very permissive standard of obviousness in biotechnology, but a highly restrictive disclosure requirement. The opposite holds true for software patents, which seems to us exactly contrary to sound policy for either industry. These patent standards are grounded in the legal fiction of the "person having ordinary skill in the art" or PHOSITA. We discuss the appropriateness of the PHOSITA standard, concluding that it properly lends flexibility to the patent system. We then discuss the difficulty of applying this standard in different industries, offering suggestions as to how it might be modified to avoid the problems seen in biotechnology and software patents.
△ Less
Submitted 26 September, 2001; v1 submitted 25 September, 2001;
originally announced September 2001.
