-
Do Multimodal Foundation Models Understand Enterprise Workflows? A Benchmark for Business Process Management Tasks
Authors:
Michael Wornow,
Avanika Narayan,
Ben Viggiano,
Ishan S. Khare,
Tathagat Verma,
Tibor Thompson,
Miguel Angel Fuentes Hernandez,
Sudharsan Sundar,
Chloe Trujillo,
Krrish Chawla,
Rongfei Lu,
Justin Shen,
Divya Nagaraj,
Joshua Martinez,
Vardhan Agrawal,
Althea Hudson,
Nigam H. Shah,
Christopher Re
Abstract:
Existing ML benchmarks lack the depth and diversity of annotations needed for evaluating models on business process management (BPM) tasks. BPM is the practice of documenting, measuring, improving, and automating enterprise workflows. However, research has focused almost exclusively on one task - full end-to-end automation using agents based on multimodal foundation models (FMs) like GPT-4. This f…
▽ More
Existing ML benchmarks lack the depth and diversity of annotations needed for evaluating models on business process management (BPM) tasks. BPM is the practice of documenting, measuring, improving, and automating enterprise workflows. However, research has focused almost exclusively on one task - full end-to-end automation using agents based on multimodal foundation models (FMs) like GPT-4. This focus on automation ignores the reality of how most BPM tools are applied today - simply documenting the relevant workflow takes 60% of the time of the typical process optimization project. To address this gap we present WONDERBREAD, the first benchmark for evaluating multimodal FMs on BPM tasks beyond automation. Our contributions are: (1) a dataset containing 2928 documented workflow demonstrations; (2) 6 novel BPM tasks sourced from real-world applications ranging from workflow documentation to knowledge transfer to process improvement; and (3) an automated evaluation harness. Our benchmark shows that while state-of-the-art FMs can automatically generate documentation (e.g. recalling 88% of the steps taken in a video demonstration of a workflow), they struggle to re-apply that knowledge towards finer-grained validation of workflow completion (F1 < 0.3). We hope WONDERBREAD encourages the development of more "human-centered" AI tooling for enterprise applications and furthers the exploration of multimodal FMs for the broader universe of BPM tasks. We publish our dataset and experiments here: https://github.com/HazyResearch/wonderbread
△ Less
Submitted 19 June, 2024;
originally announced June 2024.
-
Complexity-theoretic foundations of BosonSampling with a linear number of modes
Authors:
Adam Bouland,
Daniel Brod,
Ishaun Datta,
Bill Fefferman,
Daniel Grier,
Felipe Hernandez,
Michal Oszmaniec
Abstract:
BosonSampling is the leading candidate for demonstrating quantum computational advantage in photonic systems. While we have recently seen many impressive experimental demonstrations, there is still a formidable distance between the complexity-theoretic hardness arguments and current experiments. One of the largest gaps involves the ratio of photons to modes: all current hardness evidence assumes a…
▽ More
BosonSampling is the leading candidate for demonstrating quantum computational advantage in photonic systems. While we have recently seen many impressive experimental demonstrations, there is still a formidable distance between the complexity-theoretic hardness arguments and current experiments. One of the largest gaps involves the ratio of photons to modes: all current hardness evidence assumes a "high-mode" regime in which the number of linear optical modes scales at least quadratically in the number of photons. By contrast, current experiments operate in a "low-mode" regime with a linear number of modes. In this paper we bridge this gap, bringing the hardness evidence for the low-mode experiments to the same level as had been previously established for the high-mode regime. This involves proving a new worst-to-average-case reduction for computing the Permanent that is robust to large numbers of row repetitions and also to distributions over matrices with correlated entries.
△ Less
Submitted 30 November, 2023;
originally announced December 2023.
-
Analyzing Populations of Neural Networks via Dynamical Model Embedding
Authors:
Jordan Cotler,
Kai Sheng Tai,
Felipe Hernández,
Blake Elias,
David Sussillo
Abstract:
A core challenge in the interpretation of deep neural networks is identifying commonalities between the underlying algorithms implemented by distinct networks trained for the same task. Motivated by this problem, we introduce DYNAMO, an algorithm that constructs low-dimensional manifolds where each point corresponds to a neural network model, and two points are nearby if the corresponding neural n…
▽ More
A core challenge in the interpretation of deep neural networks is identifying commonalities between the underlying algorithms implemented by distinct networks trained for the same task. Motivated by this problem, we introduce DYNAMO, an algorithm that constructs low-dimensional manifolds where each point corresponds to a neural network model, and two points are nearby if the corresponding neural networks enact similar high-level computational processes. DYNAMO takes as input a collection of pre-trained neural networks and outputs a meta-model that emulates the dynamics of the hidden states as well as the outputs of any model in the collection. The specific model to be emulated is determined by a model embedding vector that the meta-model takes as input; these model embedding vectors constitute a manifold corresponding to the given population of models. We apply DYNAMO to both RNNs and CNNs, and find that the resulting model embedding spaces enable novel applications: clustering of neural networks on the basis of their high-level computational processes in a manner that is less sensitive to reparameterization; model averaging of several neural networks trained on the same task to arrive at a new, operable neural network with similar task performance; and semi-supervised learning via optimization on the model embedding space. Using a fixed-point analysis of meta-models trained on populations of RNNs, we gain new insights into how similarities of the topology of RNN dynamics correspond to similarities of their high-level computational processes.
△ Less
Submitted 27 February, 2023;
originally announced February 2023.
-
A polynomial-time approximation to a minimum dominating set in a graph
Authors:
Frank Hernandez,
Ernesto Parra,
Jose Maria Sigarreta,
Nodari Vakhania
Abstract:
A {\em dominating set} of a graph $G=(V,E)$ is a subset of vertices $S\subseteq V$ such that every vertex $v\in V\setminus S$ has at least one neighbor in $S$. Finding a dominating set with the minimum cardinality in a connected graph $G=(V,E)$ is known to be NP-hard. A polynomial-time approximation algorithm for this problem, described here, works in two stages. At the first stage a dominant set…
▽ More
A {\em dominating set} of a graph $G=(V,E)$ is a subset of vertices $S\subseteq V$ such that every vertex $v\in V\setminus S$ has at least one neighbor in $S$. Finding a dominating set with the minimum cardinality in a connected graph $G=(V,E)$ is known to be NP-hard. A polynomial-time approximation algorithm for this problem, described here, works in two stages. At the first stage a dominant set is generated by a greedy algorithm, and at the second stage this dominating set is purified (reduced). The reduction is achieved by the analysis of the flowchart of the algorithm of the first stage and a special kind of clustering of the dominating set generated at the first stage. The clustering of the dominating set naturally leads to a special kind of a spanning forest of graph $G$, which serves as a basis for the second purification stage. We expose some types of graphs for which the algorithm of the first stage already delivers an optimal solution and derive sufficient conditions when the overall algorithm constructs an optimal solution. We give three alternative approximation ratios for the algorithm of the first stage, two of which are expressed in terms of solely invariant problem instance parameters, and we also give one additional approximation ratio for the overall two-stage algorithm. The greedy algorithm of the first stage turned out to be essentially the same as the earlier known state-of-the-art algorithms for the set cover and dominating set problem Chvátal \cite{chvatal} and Parekh \cite{parekh}. The second purification stage results in a significant reduction of the dominant set created at the first stage, in practice. The practical behavior of both stages was verified for randomly generated problem instances. The computational experiments emphasize the gap between a solution of Stage 1 and a solution of Stage 2.
△ Less
Submitted 10 July, 2022;
originally announced July 2022.
-
The complete weight enumerator of a subclass of optimal three-weight cyclic codes
Authors:
Gerardo Vega,
Félix Hernández
Abstract:
A class of optimal three-weight cyclic codes of dimension 3 over any finite field was presented by Vega [Finite Fields Appl., 42 (2016) 23-38]. Shortly thereafter, Heng and Yue [IEEE Trans. Inf. Theory, 62(8) (2016) 4501-4513] generalized this result by presenting several classes of cyclic codes with either optimal three weights or a few weights. On the other hand, a class of optimal five-weight c…
▽ More
A class of optimal three-weight cyclic codes of dimension 3 over any finite field was presented by Vega [Finite Fields Appl., 42 (2016) 23-38]. Shortly thereafter, Heng and Yue [IEEE Trans. Inf. Theory, 62(8) (2016) 4501-4513] generalized this result by presenting several classes of cyclic codes with either optimal three weights or a few weights. On the other hand, a class of optimal five-weight cyclic codes of dimension 4 over a prime field was recently presented by Li, et al. [Adv. Math. Commun., 13(1) (2019) 137-156]. One of the purposes of this work is to present a more general description for these optimal five-weight cyclic codes, which gives place to an enlarged class of optimal five-weight cyclic codes of dimension 4 over any finite field. As an application of this enlarged class, we present the complete weight enumerator of a subclass of the optimal three-weight cyclic codes over any finite field that were studied by Vega [Finite Fields Appl., 42 (2016) 23-38]. In addition, we study the dual codes in this enlarged class of optimal five-weight cyclic codes, and show that they are cyclic codes of length $q^2-1$, dimension $q^2-5$, and minimum Hamming distance 4. In fact, through several examples, we see that those parameters are the best known parameters for linear codes.
△ Less
Submitted 11 November, 2021;
originally announced November 2021.
-
Optimal three-weight cyclic codes whose duals are also optimal
Authors:
Gerardo Vega,
Félix Hernández
Abstract:
A class of optimal three-weight cyclic codes of dimension 3 over any finite field was presented by Vega [Finite Fields Appl., 42 (2016) 23-38]. Shortly thereafter, Heng and Yue [IEEE Trans. Inf. Theory, 62(8) (2016) 4501-4513] generalized this result by presenting several classes of cyclic codes with either optimal three weights or a few weights. Here we present a new class of optimal three-weight…
▽ More
A class of optimal three-weight cyclic codes of dimension 3 over any finite field was presented by Vega [Finite Fields Appl., 42 (2016) 23-38]. Shortly thereafter, Heng and Yue [IEEE Trans. Inf. Theory, 62(8) (2016) 4501-4513] generalized this result by presenting several classes of cyclic codes with either optimal three weights or a few weights. Here we present a new class of optimal three-weight cyclic codes of length $q+1$ and dimension 3 over any finite field $F_q$, and show that the nonzero weights are $q-1$, $q$, and $q+1$. We then study the dual codes in this new class, and show that they are also optimal cyclic codes of length $q+1$, dimension $q-2$, and minimum Hamming distance $4$. Lastly, as an application of the Krawtchouck polynomials, we obtain the weight distribution of the dual codes.
△ Less
Submitted 9 July, 2021;
originally announced July 2021.
-
Time-Series Imputation with Wasserstein Interpolation for Optimal Look-Ahead-Bias and Variance Tradeoff
Authors:
Jose Blanchet,
Fernando Hernandez,
Viet Anh Nguyen,
Markus Pelger,
Xuhui Zhang
Abstract:
Missing time-series data is a prevalent practical problem. Imputation methods in time-series data often are applied to the full panel data with the purpose of training a model for a downstream out-of-sample task. For example, in finance, imputation of missing returns may be applied prior to training a portfolio optimization model. Unfortunately, this practice may result in a look-ahead-bias in the…
▽ More
Missing time-series data is a prevalent practical problem. Imputation methods in time-series data often are applied to the full panel data with the purpose of training a model for a downstream out-of-sample task. For example, in finance, imputation of missing returns may be applied prior to training a portfolio optimization model. Unfortunately, this practice may result in a look-ahead-bias in the future performance on the downstream task. There is an inherent trade-off between the look-ahead-bias of using the full data set for imputation and the larger variance in the imputation from using only the training data. By connecting layers of information revealed in time, we propose a Bayesian posterior consensus distribution which optimally controls the variance and look-ahead-bias trade-off in the imputation. We demonstrate the benefit of our methodology both in synthetic and real financial data.
△ Less
Submitted 11 April, 2023; v1 submitted 25 February, 2021;
originally announced February 2021.
-
The Ubiqus English-Inuktitut System for WMT20
Authors:
François Hernandez,
Vincent Nguyen
Abstract:
This paper describes Ubiqus' submission to the WMT20 English-Inuktitut shared news translation task. Our main system, and only submission, is based on a multilingual approach, jointly training a Transformer model on several agglutinative languages. The English-Inuktitut translation task is challenging at every step, from data selection, preparation and tokenization to quality evaluation down the l…
▽ More
This paper describes Ubiqus' submission to the WMT20 English-Inuktitut shared news translation task. Our main system, and only submission, is based on a multilingual approach, jointly training a Transformer model on several agglutinative languages. The English-Inuktitut translation task is challenging at every step, from data selection, preparation and tokenization to quality evaluation down the line. Difficulties emerge both because of the peculiarities of the Inuktitut language as well as the low-resource context.
△ Less
Submitted 18 November, 2020;
originally announced November 2020.
-
Leverage Unlabeled Data for Abstractive Speech Summarization with Self-Supervised Learning and Back-Summarization
Authors:
Paul Tardy,
Louis de Seynes,
François Hernandez,
Vincent Nguyen,
David Janiszek,
Yannick Estève
Abstract:
Supervised approaches for Neural Abstractive Summarization require large annotated corpora that are costly to build. We present a French meeting summarization task where reports are predicted based on the automatic transcription of the meeting audio recordings. In order to build a corpus for this task, it is necessary to obtain the (automatic or manual) transcription of each meeting, and then to s…
▽ More
Supervised approaches for Neural Abstractive Summarization require large annotated corpora that are costly to build. We present a French meeting summarization task where reports are predicted based on the automatic transcription of the meeting audio recordings. In order to build a corpus for this task, it is necessary to obtain the (automatic or manual) transcription of each meeting, and then to segment and align it with the corresponding manual report to produce training examples suitable for training. On the other hand, we have access to a very large amount of unaligned data, in particular reports without corresponding transcription. Reports are professionally written and well formatted making pre-processing straightforward. In this context, we study how to take advantage of this massive amount of unaligned data using two approaches (i) self-supervised pre-training using a target-side denoising encoder-decoder model; (ii) back-summarization i.e. reversing the summarization process by learning to predict the transcription given the report, in order to align single reports with generated transcription, and use this synthetic dataset for further training. We report large improvements compared to the previous baseline (trained on aligned data only) for both approaches on two evaluation sets. Moreover, combining the two gives even better results, outperforming the baseline by a large margin of +6 ROUGE-1 and ROUGE-L and +5 ROUGE-2 on two evaluation sets
△ Less
Submitted 17 September, 2020; v1 submitted 30 July, 2020;
originally announced July 2020.
-
WLCG Networks: Update on Monitoring and Analytics
Authors:
Marian Babik,
Shawn McKee,
Pedro Andrade,
Brian Paul Bockelman,
Robert Gardner,
Edgar Mauricio Fajardo Hernandez,
Edoardo Martelli,
Ilija Vukotic,
Derek Weitzel,
Marian Zvada
Abstract:
WLCG relies on the network as a critical part of its infrastructure and therefore needs to guarantee effective network usage and prompt detection and resolution of any network issues including connection failures, congestion and traffic routing. The OSG Networking Area, in partnership with WLCG, is focused on being the primary source of networking information for its partners and constituents. It…
▽ More
WLCG relies on the network as a critical part of its infrastructure and therefore needs to guarantee effective network usage and prompt detection and resolution of any network issues including connection failures, congestion and traffic routing. The OSG Networking Area, in partnership with WLCG, is focused on being the primary source of networking information for its partners and constituents. It was established to ensure sites and experiments can better understand and fix networking issues, while providing an analytics platform that aggregates network monitoring data with higher level workload and data trans-fer services. This has been facilitated by the global network of the perfSONAR instances that have been commissioned and are operated in collaboration with WLCG Network Throughput Working Group. An additional important updateis the inclusion of the newly funded NSF project SAND (Service Analytics and Network Diagnosis) which is focusing on network analytics. This paper describes the current state of the network measurement and analytics platform and summarizes the activities taken by the working group and our collaborators. This includes the progress being made in providing higher level analytics,alerting and alarming from the rich set of network metrics we are gathering.
△ Less
Submitted 1 July, 2020;
originally announced July 2020.
-
StashCache: A Distributed Caching Federation for the Open Science Grid
Authors:
Derek Weitzel,
Marian Zvada,
Ilija Vukotic,
Rob Gardner,
Brian Bockelman,
Mats Rynge,
Edgar Fajardo Hernandez,
Brian Lin,
Matyas Selmeci
Abstract:
Data distribution for opportunistic users is challenging as they neither own the computing resources they are using or any nearby storage. Users are motivated to use opportunistic computing to expand their data processing capacity, but they require storage and fast networking to distribute data to that processing. Since it requires significant management overhead, it is rare for resource providers…
▽ More
Data distribution for opportunistic users is challenging as they neither own the computing resources they are using or any nearby storage. Users are motivated to use opportunistic computing to expand their data processing capacity, but they require storage and fast networking to distribute data to that processing. Since it requires significant management overhead, it is rare for resource providers to allow opportunistic access to storage. Additionally, in order to use opportunistic storage at several distributed sites, users assume the responsibility to maintain their data. In this paper we present StashCache, a distributed caching federation that enables opportunistic users to utilize nearby opportunistic storage. StashCache is comprised of four components: data origins, redirectors, caches, and clients. StashCache has been deployed in the Open Science Grid for several years and has been used by many projects. Caches are deployed in geographically distributed locations across the U.S. and Europe. We will present the architecture of StashCache, as well as utilization information of the infrastructure. We will also present performance analysis comparing distributed HTTP Proxies vs StashCache.
△ Less
Submitted 16 May, 2019;
originally announced May 2019.
-
Shakedown: compiler-based moving target protection for Return Oriented Programing attacks on an industrial IoT device
Authors:
Fady Copty,
Francisco Hernandez,
Dov Murik,
Olmo Rayón
Abstract:
Cybercriminals use Return Oriented Programming techniques to attack systems and IoT devices. While defenses have been developed, not all of them are applicable to constrained devices. We present Shakedown, which is a compile-time randomizing build tool which creates several versions of the binary, each with a distinct memory layout. An attack developed against one device will not work on another d…
▽ More
Cybercriminals use Return Oriented Programming techniques to attack systems and IoT devices. While defenses have been developed, not all of them are applicable to constrained devices. We present Shakedown, which is a compile-time randomizing build tool which creates several versions of the binary, each with a distinct memory layout. An attack developed against one device will not work on another device which has a different memory layout. We tested Shakedown on an industrial IoT device and shown that its normal functionality remained intact while an exploit was blocked.
△ Less
Submitted 11 October, 2018; v1 submitted 4 October, 2018;
originally announced October 2018.
-
TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation
Authors:
François Hernandez,
Vincent Nguyen,
Sahar Ghannay,
Natalia Tomashenko,
Yannick Estève
Abstract:
In this paper, we present TED-LIUM release 3 corpus dedicated to speech recognition in English, that multiplies by more than two the available data to train acoustic models in comparison with TED-LIUM 2. We present the recent development on Automatic Speech Recognition (ASR) systems in comparison with the two previous releases of the TED-LIUM Corpus from 2012 and 2014. We demonstrate that, passing…
▽ More
In this paper, we present TED-LIUM release 3 corpus dedicated to speech recognition in English, that multiplies by more than two the available data to train acoustic models in comparison with TED-LIUM 2. We present the recent development on Automatic Speech Recognition (ASR) systems in comparison with the two previous releases of the TED-LIUM Corpus from 2012 and 2014. We demonstrate that, passing from 207 to 452 hours of transcribed speech training data is really more useful for end-to-end ASR systems than for HMM-based state-of-the-art ones, even if the HMM-based ASR system still outperforms end-to-end ASR system when the size of audio training data is 452 hours, with respectively a Word Error Rate (WER) of 6.6% and 13.7%. Last, we propose two repartitions of the TED-LIUM release 3 corpus: the legacy one that is the same as the one existing in release 2, and a new one, calibrated and designed to make experiments on speaker adaptation. Like the two first releases, TED-LIUM 3 corpus will be freely available for the research community.
△ Less
Submitted 13 June, 2019; v1 submitted 12 May, 2018;
originally announced May 2018.
-
Data Access for LIGO on the OSG
Authors:
Derek Weitzel,
Brian Bockelman,
Duncan A. Brown,
Peter Couvares,
Frank Würthwein,
Edgar Fajardo Hernandez
Abstract:
During 2015 and 2016, the Laser Interferometer Gravitational-Wave Observatory (LIGO) conducted a three-month observing campaign. These observations delivered the first direct detection of gravitational waves from binary black hole mergers. To search for these signals, the LIGO Scientific Collaboration uses the PyCBC search pipeline. To deliver science results in a timely manner, LIGO collaborated…
▽ More
During 2015 and 2016, the Laser Interferometer Gravitational-Wave Observatory (LIGO) conducted a three-month observing campaign. These observations delivered the first direct detection of gravitational waves from binary black hole mergers. To search for these signals, the LIGO Scientific Collaboration uses the PyCBC search pipeline. To deliver science results in a timely manner, LIGO collaborated with the Open Science Grid (OSG) to distribute the required computation across a series of dedicated, opportunistic, and allocated resources. To deliver the petabytes necessary for such a large-scale computation, our team deployed a distributed data access infrastructure based on the XRootD server suite and the CernVM File System (CVMFS). This data access strategy grew from simply accessing remote storage to a POSIX-based interface underpinned by distributed, secure caches across the OSG.
△ Less
Submitted 17 May, 2017;
originally announced May 2017.
-
Cryptanalysis of a Classical chaos-based cryptosystem with some quantum cryptography features
Authors:
David Arroyo,
Fernando Hernandez,
Amalia B. Orúe
Abstract:
The application of synchronization theory to build up new cryptosystems has been a hot topic during the last two decades. In this paper we analyze a recent proposal in this field. We pinpoint the main limitations of the software implementation of chaos-based systems designed on the grounds of synchronization theory. In addition, we show that the cryptosystem under evaluation possesses serious secu…
▽ More
The application of synchronization theory to build up new cryptosystems has been a hot topic during the last two decades. In this paper we analyze a recent proposal in this field. We pinpoint the main limitations of the software implementation of chaos-based systems designed on the grounds of synchronization theory. In addition, we show that the cryptosystem under evaluation possesses serious security problems that imply a clear reduction of the key space.
△ Less
Submitted 26 October, 2016;
originally announced October 2016.
-
Status Report of the DPHEP Study Group: Towards a Global Effort for Sustainable Data Preservation in High Energy Physics
Authors:
Z. Akopov,
Silvia Amerio,
David Asner,
Eduard Avetisyan,
Olof Barring,
James Beacham,
Matthew Bellis,
Gregorio Bernardi,
Siegfried Bethke,
Amber Boehnlein,
Travis Brooks,
Thomas Browder,
Rene Brun,
Concetta Cartaro,
Marco Cattaneo,
Gang Chen,
David Corney,
Kyle Cranmer,
Ray Culbertson,
Sunje Dallmeier-Tiessen,
Dmitri Denisov,
Cristinel Diaconu,
Vitaliy Dodonov,
Tony Doyle,
Gregory Dubois-Felsmann
, et al. (65 additional authors not shown)
Abstract:
Data from high-energy physics (HEP) experiments are collected with significant financial and human effort and are mostly unique. An inter-experimental study group on HEP data preservation and long-term analysis was convened as a panel of the International Committee for Future Accelerators (ICFA). The group was formed by large collider-based experiments and investigated the technical and organisati…
▽ More
Data from high-energy physics (HEP) experiments are collected with significant financial and human effort and are mostly unique. An inter-experimental study group on HEP data preservation and long-term analysis was convened as a panel of the International Committee for Future Accelerators (ICFA). The group was formed by large collider-based experiments and investigated the technical and organisational aspects of HEP data preservation. An intermediate report was released in November 2009 addressing the general issues of data preservation in HEP. This paper includes and extends the intermediate report. It provides an analysis of the research case for data preservation and a detailed description of the various projects at experiment, laboratory and international levels. In addition, the paper provides a concrete proposal for an international organisation in charge of the data management and policies in high-energy physics.
△ Less
Submitted 21 May, 2012;
originally announced May 2012.
