-
Transfer Learning in Information Criteria-based Feature Selection
Authors:
Shaohan Chen,
Nikolaos V. Sahinidis,
Chuanhou Gao
Abstract:
This paper investigates the effectiveness of transfer learning based on Mallows' Cp. We propose a procedure that combines transfer learning with Mallows' Cp (TLCp) and prove that it outperforms the conventional Mallows' Cp criterion in terms of accuracy and stability. Our theoretical results indicate that, for any sample size in the target domain, the proposed TLCp estimator performs better than t…
▽ More
This paper investigates the effectiveness of transfer learning based on Mallows' Cp. We propose a procedure that combines transfer learning with Mallows' Cp (TLCp) and prove that it outperforms the conventional Mallows' Cp criterion in terms of accuracy and stability. Our theoretical results indicate that, for any sample size in the target domain, the proposed TLCp estimator performs better than the Cp estimator by the mean squared error (MSE) metric in the case of orthogonal predictors, provided that i) the dissimilarity between the tasks from source domain and target domain is small, and ii) the procedure parameters (complexity penalties) are tuned according to certain explicit rules. Moreover, we show that our transfer learning framework can be extended to other feature selection criteria, such as the Bayesian information criterion. By analyzing the solution of the orthogonalized Cp, we identify an estimator that asymptotically approximates the solution of the Cp criterion in the case of non-orthogonal predictors. Similar results are obtained for the non-orthogonal TLCp. Finally, simulation studies and applications with real data demonstrate the usefulness of the TLCp scheme.
△ Less
Submitted 29 May, 2022; v1 submitted 6 July, 2021;
originally announced July 2021.
-
A Discussion on Practical Considerations with Sparse Regression Methodologies
Authors:
Owais Sarwar,
Benjamin Sauk,
Nikolaos V. Sahinidis
Abstract:
Sparse linear regression is a vast field and there are many different algorithms available to build models. Two new papers published in Statistical Science study the comparative performance of several sparse regression methodologies, including the lasso and subset selection. Comprehensive empirical analyses allow the researchers to demonstrate the relative merits of each estimator and provide guid…
▽ More
Sparse linear regression is a vast field and there are many different algorithms available to build models. Two new papers published in Statistical Science study the comparative performance of several sparse regression methodologies, including the lasso and subset selection. Comprehensive empirical analyses allow the researchers to demonstrate the relative merits of each estimator and provide guidance to practitioners. In this discussion, we summarize and compare the two studies and we examine points of agreement and divergence, aiming to provide clarity and value to users. The authors have started a highly constructive dialogue, our goal is to continue it.
△ Less
Submitted 8 February, 2021; v1 submitted 18 November, 2020;
originally announced November 2020.
-
OR-Gym: A Reinforcement Learning Library for Operations Research Problems
Authors:
Christian D. Hubbs,
Hector D. Perez,
Owais Sarwar,
Nikolaos V. Sahinidis,
Ignacio E. Grossmann,
John M. Wassick
Abstract:
Reinforcement learning (RL) has been widely applied to game-playing and surpassed the best human-level performance in many domains, yet there are few use-cases in industrial or commercial settings. We introduce OR-Gym, an open-source library for developing reinforcement learning algorithms to address operations research problems. In this paper, we apply reinforcement learning to the knapsack, mult…
▽ More
Reinforcement learning (RL) has been widely applied to game-playing and surpassed the best human-level performance in many domains, yet there are few use-cases in industrial or commercial settings. We introduce OR-Gym, an open-source library for developing reinforcement learning algorithms to address operations research problems. In this paper, we apply reinforcement learning to the knapsack, multi-dimensional bin packing, multi-echelon supply chain, and multi-period asset allocation model problems, as well as benchmark the RL solutions against MILP and heuristic models. These problems are used in logistics, finance, engineering, and are common in many business operation settings. We develop environments based on prototypical models in the literature and implement various optimization and heuristic models in order to benchmark the RL results. By re-framing a series of classic optimization problems as RL tasks, we seek to provide a new tool for the operations research community, while also opening those in the RL community to many of the problems and challenges in the OR field.
△ Less
Submitted 17 October, 2020; v1 submitted 14 August, 2020;
originally announced August 2020.
-
Domain reduction techniques for global NLP and MINLP optimization
Authors:
Yash Puranik,
Nikolaos V. Sahinidis
Abstract:
Optimization solvers routinely utilize presolve techniques, including model simplification, reformulation and domain reduction techniques. Domain reduction techniques are especially important in speeding up convergence to the global optimum for challenging nonconvex nonlinear programming (NLP) and mixed-integer nonlinear programming (MINLP) optimization problems. In this work, we survey the variou…
▽ More
Optimization solvers routinely utilize presolve techniques, including model simplification, reformulation and domain reduction techniques. Domain reduction techniques are especially important in speeding up convergence to the global optimum for challenging nonconvex nonlinear programming (NLP) and mixed-integer nonlinear programming (MINLP) optimization problems. In this work, we survey the various techniques used for domain reduction of NLP and MINLP optimization problems. We also present a computational analysis of the impact of these techniques on the performance of various widely available global solvers on a collection of 1740 test problems.
△ Less
Submitted 26 June, 2017;
originally announced June 2017.
-
Simulation optimization: A review of algorithms and applications
Authors:
Satyajith Amaran,
Nikolaos V. Sahinidis,
Bikram Sharda,
Scott J. Bury
Abstract:
Simulation Optimization (SO) refers to the optimization of an objective function subject to constraints, both of which can be evaluated through a stochastic simulation. To address specific features of a particular simulation---discrete or continuous decisions, expensive or cheap simulations, single or multiple outputs, homogeneous or heterogeneous noise---various algorithms have been proposed in t…
▽ More
Simulation Optimization (SO) refers to the optimization of an objective function subject to constraints, both of which can be evaluated through a stochastic simulation. To address specific features of a particular simulation---discrete or continuous decisions, expensive or cheap simulations, single or multiple outputs, homogeneous or heterogeneous noise---various algorithms have been proposed in the literature. As one can imagine, there exist several competing algorithms for each of these classes of problems. This document emphasizes the difficulties in simulation optimization as compared to mathematical programming, makes reference to state-of-the-art algorithms in the field, examines and contrasts the different approaches used, reviews some of the diverse applications that have been tackled by these methods, and speculates on future directions in the field.
△ Less
Submitted 26 June, 2017;
originally announced June 2017.
-
The ALAMO approach to machine learning
Authors:
Zachary T. Wilson,
Nikolaos V. Sahinidis
Abstract:
ALAMO is a computational methodology for leaning algebraic functions from data. Given a data set, the approach begins by building a low-complexity, linear model composed of explicit non-linear transformations of the independent variables. Linear combinations of these non-linear transformations allow a linear model to better approximate complex behavior observed in real processes. The model is refi…
▽ More
ALAMO is a computational methodology for leaning algebraic functions from data. Given a data set, the approach begins by building a low-complexity, linear model composed of explicit non-linear transformations of the independent variables. Linear combinations of these non-linear transformations allow a linear model to better approximate complex behavior observed in real processes. The model is refined, as additional data are obtained in an adaptive fashion through error maximization sampling using derivative-free optimization. Models built using ALAMO can enforce constraints on the response variables to incorporate first-principles knowledge. The ability of ALAMO to generate simple and accurate models for a number of reaction problems is demonstrated. The error maximization sampling is compared with Latin hypercube designs to demonstrate its sampling efficiency. ALAMO's constrained regression methodology is used to further refine concentration models, resulting in models that perform better on validation data and satisfy upper and lower bounds placed on model outputs.
△ Less
Submitted 30 May, 2017;
originally announced May 2017.
-
Optimization of circuitry arrangements for heat exchangers using derivative-free optimization
Authors:
Nikolaos Ploskas,
Christopher Laughman,
Arvind U. Raghunathan,
Nikolaos V. Sahinidis
Abstract:
Optimization of the refrigerant circuitry can improve a heat exchanger's performance. Design engineers currently choose the refrigerant circuitry according to their experience and heat exchanger simulations. However, the design of an optimized refrigerant circuitry is difficult. The number of refrigerant circuitry candidates is enormous. Therefore, exhaustive search algorithms cannot be used and i…
▽ More
Optimization of the refrigerant circuitry can improve a heat exchanger's performance. Design engineers currently choose the refrigerant circuitry according to their experience and heat exchanger simulations. However, the design of an optimized refrigerant circuitry is difficult. The number of refrigerant circuitry candidates is enormous. Therefore, exhaustive search algorithms cannot be used and intelligent techniques must be developed to explore the solution space efficiently. In this paper, we formulate refrigerant circuitry design as a binary constrained optimization problem. We use CoilDesigner, a simulation and design tool of air to refrigerant heat exchangers, in order to simulate the performance of different refrigerant circuitry designs. We treat CoilDesigner as a black-box system since the exact relationship of the objective function with the decision variables is not explicit. Derivative-free optimization (DFO) algorithms are suitable for solving this black-box model since they do not require explicit functional representations of the objective function and the constraints. The aim of this paper is twofold. First, we compare four mixed-integer constrained DFO solvers and one box-bounded DFO solver and evaluate their ability to solve a difficult industrially relevant problem. Second, we demonstrate that the proposed formulation is suitable for optimizing the circuitry configuration of heat exchangers. We apply the DFO solvers to 17 heat exchanger design problems. Results show that TOMLAB/glcDirect and TOMLAB/glcSolve can find optimal or near-optimal refrigerant circuitry designs after a relatively small number of circuit simulations.
△ Less
Submitted 29 May, 2017;
originally announced May 2017.
-
Computer-aided molecular design: An introduction and review of tools, applications, and solution techniques
Authors:
Nick D. Austin,
Nikolaos V. Sahinidis,
Daniel W. Trahan
Abstract:
This article provides an introduction to and review of the field of computer-aided molecular design (CAMD). It is intended to be approachable for the absolute beginner as well as useful to the seasoned CAMD practitioner. We begin by discussing various quantitative structure-property relationships (QSPRs) which have been demonstrated to work well with CAMD problems. The methods discussed in this ar…
▽ More
This article provides an introduction to and review of the field of computer-aided molecular design (CAMD). It is intended to be approachable for the absolute beginner as well as useful to the seasoned CAMD practitioner. We begin by discussing various quantitative structure-property relationships (QSPRs) which have been demonstrated to work well with CAMD problems. The methods discussed in this article are (1) group contribution methods, (2) topological indices, and (3) signature descriptors. Next, we present general optimization formulations for various forms of the CAMD problem. Common design constraints are discussed and structural feasibility constraints are provided for the three types of QSPRs addressed. We then detail useful techniques for approaching CAMD optimization problems, including decomposition methods, heuristic approaches, and mathematical programming strategies. Finally, we discuss many applications that have been addressed using CAMD.
△ Less
Submitted 14 January, 2017;
originally announced January 2017.