-
Implementing a hybrid approach in a knowledge engineering process to manage technical advice relating to feedback from the operation of complex sensitive equipment
Authors:
Alain Claude Hervé Berger,
Sébastien Boblet,
Thierry Cartié,
Jean-Pierre Cotton,
François Vexler
Abstract:
How can technical advice on operating experience feedback be managed efficiently in an organization that has never used knowledge engineering techniques and methods? This article explains how an industrial company in the nuclear and defense sectors adopted such an approach, adapted to its "TA KM" organizational context and falls within the ISO30401 framework, to build a complete system with a "SAR…
▽ More
How can technical advice on operating experience feedback be managed efficiently in an organization that has never used knowledge engineering techniques and methods? This article explains how an industrial company in the nuclear and defense sectors adopted such an approach, adapted to its "TA KM" organizational context and falls within the ISO30401 framework, to build a complete system with a "SARBACANES" application to support its business processes and perpetuate its know-how and expertise in a knowledge base. Over and above the classic transfer of knowledge between experts and business specialists, SARBACANES also reveals the ability of this type of engineering to deliver multi-functional operation. Modeling was accelerated by the use of a tool adapted to this type of operation: the Ardans Knowledge Maker platform.
△ Less
Submitted 8 July, 2024;
originally announced July 2024.
-
Topologically faithful multi-class segmentation in medical images
Authors:
Alexander H. Berger,
Nico Stucki,
Laurin Lux,
Vincent Buergin,
Suprosanna Shit,
Anna Banaszak,
Daniel Rueckert,
Ulrich Bauer,
Johannes C. Paetzold
Abstract:
Topological accuracy in medical image segmentation is a highly important property for downstream applications such as network analysis and flow modeling in vessels or cell counting. Recently, significant methodological advancements have brought well-founded concepts from algebraic topology to binary segmentation. However, these approaches have been underexplored in multi-class segmentation scenari…
▽ More
Topological accuracy in medical image segmentation is a highly important property for downstream applications such as network analysis and flow modeling in vessels or cell counting. Recently, significant methodological advancements have brought well-founded concepts from algebraic topology to binary segmentation. However, these approaches have been underexplored in multi-class segmentation scenarios, where topological errors are common. We propose a general loss function for topologically faithful multi-class segmentation extending the recent Betti matching concept, which is based on induced matchings of persistence barcodes. We project the N-class segmentation problem to N single-class segmentation tasks, which allows us to use 1-parameter persistent homology making training of neural networks computationally feasible. We validate our method on a comprehensive set of four medical datasets with highly variant topological characteristics. Our loss formulation significantly enhances topological correctness in cardiac, cell, artery-vein, and Circle of Willis segmentation.
△ Less
Submitted 16 March, 2024;
originally announced March 2024.
-
Cross-domain and Cross-dimension Learning for Image-to-Graph Transformers
Authors:
Alexander H. Berger,
Laurin Lux,
Suprosanna Shit,
Ivan Ezhov,
Georgios Kaissis,
Martin J. Menten,
Daniel Rueckert,
Johannes C. Paetzold
Abstract:
Direct image-to-graph transformation is a challenging task that solves object detection and relationship prediction in a single model. Due to the complexity of this task, large training datasets are rare in many domains, which makes the training of large networks challenging. This data sparsity necessitates the establishment of pre-training strategies akin to the state-of-the-art in computer visio…
▽ More
Direct image-to-graph transformation is a challenging task that solves object detection and relationship prediction in a single model. Due to the complexity of this task, large training datasets are rare in many domains, which makes the training of large networks challenging. This data sparsity necessitates the establishment of pre-training strategies akin to the state-of-the-art in computer vision. In this work, we introduce a set of methods enabling cross-domain and cross-dimension transfer learning for image-to-graph transformers. We propose (1) a regularized edge sampling loss for sampling the optimal number of object relationships (edges) across domains, (2) a domain adaptation framework for image-to-graph transformers that aligns features from different domains, and (3) a simple projection function that allows us to pretrain 3D transformers on 2D input data. We demonstrate our method's utility in cross-domain and cross-dimension experiments, where we pretrain our models on 2D satellite images before applying them to vastly different target domains in 2D and 3D. Our method consistently outperforms a series of baselines on challenging benchmarks, such as retinal or whole-brain vessel graph extraction.
△ Less
Submitted 11 March, 2024;
originally announced March 2024.
-
An improved bound for the price of anarchy for related machine scheduling
Authors:
Andre Berger,
Arman Rouhani,
Marc Schröder
Abstract:
In this paper, we introduce an improved upper bound for the efficiency of Nash equilibria in utilitarian scheduling games on related machines. The machines have varying speeds and adhere to the Shortest Processing Time (SPT) policy as the global order for job processing. The goal of each job is to minimize its completion time, while the social objective is to minimize the sum of completion times.…
▽ More
In this paper, we introduce an improved upper bound for the efficiency of Nash equilibria in utilitarian scheduling games on related machines. The machines have varying speeds and adhere to the Shortest Processing Time (SPT) policy as the global order for job processing. The goal of each job is to minimize its completion time, while the social objective is to minimize the sum of completion times. Our main result provides an upper bound of $2-\frac{1}{2\cdot(2m-1)}$ on the price of anarchy for the general case of $m$ machines. We improve this bound to 3/2 for the case of two machines, and to $2-\frac{1}{2\cdot m}$ for the general case of $m$ machines when the machines have divisible speeds.
△ Less
Submitted 11 January, 2024;
originally announced January 2024.
-
Towards Autonomous Crop Monitoring: Inserting Sensors in Cluttered Environments
Authors:
Moonyoung Lee,
Aaron Berger,
Dominic Guri,
Kevin Zhang,
Lisa Coffee,
George Kantor,
Oliver Kroemer
Abstract:
We present a contact-based phenotyping robot platform that can autonomously insert nitrate sensors into cornstalks to proactively monitor macronutrient levels in crops. This task is challenging because inserting such sensors requires sub-centimeter precision in an environment which contains high levels of clutter, lighting variation, and occlusion. To address these challenges, we develop a robust…
▽ More
We present a contact-based phenotyping robot platform that can autonomously insert nitrate sensors into cornstalks to proactively monitor macronutrient levels in crops. This task is challenging because inserting such sensors requires sub-centimeter precision in an environment which contains high levels of clutter, lighting variation, and occlusion. To address these challenges, we develop a robust perception-action pipeline to detect and grasp stalks, and create a custom robot gripper which mechanically aligns the sensor before inserting it into the stalk. Through experimental validation on 48 unique stalks in a cornfield in Iowa, we demonstrate our platform's capability of detecting a stalk with 94% success, grasping a stalk with 90% success, and inserting a sensor with 60% success. In addition to developing an autonomous phenotyping research platform, we share key challenges and insights obtained from deployment in the field. Our research platform is open-sourced, with additional information available at https://kantor-lab.github.io/cornbot.
△ Less
Submitted 6 November, 2023;
originally announced November 2023.
-
Approximating Min-Diameter: Standard and Bichromatic
Authors:
Aaron Berger,
Jenny Kaufmann,
Virginia Vassilevska Williams
Abstract:
The min-diameter of a directed graph $G$ is a measure of the largest distance between nodes. It is equal to the maximum min-distance $d_{min}(u,v)$ across all pairs $u,v \in V(G)$, where $d_{min}(u,v) = \min(d(u,v), d(v,u))$. Our work provides a $O(m^{1.426}n^{0.288})$-time $3/2$-approximation algorithm for min-diameter in DAGs, and a faster $O(m^{0.713}n)$-time almost-$3/2$-approximation variant.…
▽ More
The min-diameter of a directed graph $G$ is a measure of the largest distance between nodes. It is equal to the maximum min-distance $d_{min}(u,v)$ across all pairs $u,v \in V(G)$, where $d_{min}(u,v) = \min(d(u,v), d(v,u))$. Our work provides a $O(m^{1.426}n^{0.288})$-time $3/2$-approximation algorithm for min-diameter in DAGs, and a faster $O(m^{0.713}n)$-time almost-$3/2$-approximation variant. (An almost-$α$-approximation algorithm determines the min-diameter to within a multiplicative factor of $α$ plus constant additive error.) By a conditional lower bound result of [Abboud et al, SODA 2016], a better than $3/2$-approximation can't be achieved in truly subquadratic time under the Strong Exponential Time Hypothesis (SETH), so our result is conditionally tight. We additionally obtain a new conditional lower bound for min-diameter approximation in general directed graphs, showing that under SETH, one cannot achieve an approximation factor below 2 in truly subquadratic time. We also present the first study of approximating bichromatic min-diameter, which is the maximum min-distance between oppositely colored vertices in a 2-colored graph.
△ Less
Submitted 16 August, 2023;
originally announced August 2023.
-
Improving Zero-Shot Text Matching for Financial Auditing with Large Language Models
Authors:
Lars Hillebrand,
Armin Berger,
Tobias Deußer,
Tim Dilmaghani,
Mohamed Khaled,
Bernd Kliem,
Rüdiger Loitz,
Maren Pielka,
David Leonhard,
Christian Bauckhage,
Rafet Sifa
Abstract:
Auditing financial documents is a very tedious and time-consuming process. As of today, it can already be simplified by employing AI-based solutions to recommend relevant text passages from a report for each legal requirement of rigorous accounting standards. However, these methods need to be fine-tuned regularly, and they require abundant annotated data, which is often lacking in industrial envir…
▽ More
Auditing financial documents is a very tedious and time-consuming process. As of today, it can already be simplified by employing AI-based solutions to recommend relevant text passages from a report for each legal requirement of rigorous accounting standards. However, these methods need to be fine-tuned regularly, and they require abundant annotated data, which is often lacking in industrial environments. Hence, we present ZeroShotALI, a novel recommender system that leverages a state-of-the-art large language model (LLM) in conjunction with a domain-specifically optimized transformer-based text-matching solution. We find that a two-step approach of first retrieving a number of best matching document sections per legal requirement with a custom BERT-based model and second filtering these selections using an LLM yields significant performance improvements over existing approaches.
△ Less
Submitted 14 August, 2023; v1 submitted 11 August, 2023;
originally announced August 2023.
-
The Entoptic Field Camera as Metaphor-Driven Research-through-Design with AI Technologies
Authors:
Jesse Josua Benjamin,
Heidi Biggs,
Arne Berger,
Julija Rukanskaitė,
Michael Heidt,
Nick Merrill,
James Pierce,
Joseph Lindley
Abstract:
Artificial intelligence (AI) technologies are widely deployed in smartphone photography; and prompt-based image synthesis models have rapidly become commonplace. In this paper, we describe a Research-through-Design (RtD) project which explores this shift in the means and modes of image production via the creation and use of the Entoptic Field Camera. Entoptic phenomena usually refer to perceptions…
▽ More
Artificial intelligence (AI) technologies are widely deployed in smartphone photography; and prompt-based image synthesis models have rapidly become commonplace. In this paper, we describe a Research-through-Design (RtD) project which explores this shift in the means and modes of image production via the creation and use of the Entoptic Field Camera. Entoptic phenomena usually refer to perceptions of floaters or bright blue dots stemming from the physiological interplay of the eye and brain. We use the term entoptic as a metaphor to investigate how the material interplay of data and models in AI technologies shapes human experiences of reality. Through our case study using first-person design and a field study, we offer implications for critical, reflective, more-than-human and ludic design to engage AI technologies; the conceptualisation of an RtD research space which contributes to AI literacy discourses; and outline a research trajectory concerning materiality and design affordances of AI technologies.
△ Less
Submitted 23 January, 2023;
originally announced January 2023.
-
Illuminating Large-Scale IPv6 Scanning in the Internet
Authors:
Philipp Richter,
Oliver Gasser,
Arthur Berger
Abstract:
While scans of the IPv4 space are ubiquitous, today little is known about scanning activity in the IPv6 Internet. In this work, we present a longitudinal and detailed empirical study on large-scale IPv6 scanning behavior in the Internet, based on firewall logs captured at some 230,000 hosts of a major Content Distribution Network (CDN). We develop methods to identify IPv6 scans, assess current and…
▽ More
While scans of the IPv4 space are ubiquitous, today little is known about scanning activity in the IPv6 Internet. In this work, we present a longitudinal and detailed empirical study on large-scale IPv6 scanning behavior in the Internet, based on firewall logs captured at some 230,000 hosts of a major Content Distribution Network (CDN). We develop methods to identify IPv6 scans, assess current and past levels of IPv6 scanning activity, and study dominant characteristics of scans, including scanner origins, targeted services, and insights on how scanners find target IPv6 addresses. Where possible, we compare our findings to what can be assessed from publicly available traces. Our work identifies and highlights new challenges to detect scanning activity in the IPv6 Internet, and uncovers that today's scans of the IPv6 space show widely different characteristics when compared to the more well-known IPv4 scans.
△ Less
Submitted 19 October, 2022;
originally announced October 2022.
-
Playing The Ethics Card: Ethical Aspects In Design Tools For Inspiration And Education
Authors:
Albrecht Kurze,
Arne Berger
Abstract:
This paper relates findings of own research in the domain of co-design tools in terms of ethical aspects and their opportunities for inspiration and in HCI education. We overview a number of selected general-purpose HCI/design tools as well as domain specific tools for the Internet of Things. These tools are often card-based, not only suitable for workshops with co-designers but also for internal…
▽ More
This paper relates findings of own research in the domain of co-design tools in terms of ethical aspects and their opportunities for inspiration and in HCI education. We overview a number of selected general-purpose HCI/design tools as well as domain specific tools for the Internet of Things. These tools are often card-based, not only suitable for workshops with co-designers but also for internal workshops with students to include these aspects in the built-up of their expertise, sometimes even in a playful way.
△ Less
Submitted 25 January, 2022;
originally announced January 2022.
-
Machine Learning Uncertainty as a Design Material: A Post-Phenomenological Inquiry
Authors:
Jesse Josua Benjamin,
Arne Berger,
Nick Merrill,
James Pierce
Abstract:
Design research is important for understanding and interrogating how emerging technologies shape human experience. However, design research with Machine Learning (ML) is relatively underdeveloped. Crucially, designers have not found a grasp on ML uncertainty as a design opportunity rather than an obstacle. The technical literature points to data and model uncertainties as two main properties of ML…
▽ More
Design research is important for understanding and interrogating how emerging technologies shape human experience. However, design research with Machine Learning (ML) is relatively underdeveloped. Crucially, designers have not found a grasp on ML uncertainty as a design opportunity rather than an obstacle. The technical literature points to data and model uncertainties as two main properties of ML. Through post-phenomenology, we position uncertainty as one defining material attribute of ML processes which mediate human experience. To understand ML uncertainty as a design material, we investigate four design research case studies involving ML. We derive three provocative concepts: thingly uncertainty: ML-driven artefacts have uncertain, variable relations to their environments; pattern leakage: ML uncertainty can lead to patterns shaping the world they are meant to represent; and futures creep: ML technologies texture human relations to time with uncertainty. Finally, we outline design research trajectories and sketch a post-phenomenological approach to human-ML relations.
△ Less
Submitted 11 January, 2021;
originally announced January 2021.
-
Towards Reconstructing Multi-Step Cyber Attacks in Modern Cloud Environments with Tripwires
Authors:
Mario Kahlhofer,
Michael Hölzl,
Andreas Berger
Abstract:
Rapidly-changing cloud environments that consist of heavily interconnected components are difficult to secure. Existing solutions often try to correlate many weak indicators to identify and reconstruct multi-step cyber attacks. The lack of a true, causal link between most of these indicators still leaves administrators with a lot of false-positives to browse through. We argue that cyber deception…
▽ More
Rapidly-changing cloud environments that consist of heavily interconnected components are difficult to secure. Existing solutions often try to correlate many weak indicators to identify and reconstruct multi-step cyber attacks. The lack of a true, causal link between most of these indicators still leaves administrators with a lot of false-positives to browse through. We argue that cyber deception can improve the precision of attack detection systems, if used in a structured, and automatic way, i.e., in the form of so-called tripwires that ultimately span an attack graph, which assists attack reconstruction algorithms. This paper proposes an idea for a framework that combines cyber deception, automatic tripwire injection and attack graphs, which eventually enables us to reconstruct multi-step cyber attacks in modern cloud environments.
△ Less
Submitted 25 September, 2020;
originally announced September 2020.
-
Memoryless Worker-Task Assignment with Polylogarithmic Switching Cost
Authors:
Aaron Berger,
William Kuszmaul,
Adam Polak,
Jonathan Tidor,
Nicole Wein
Abstract:
We study the basic problem of assigning memoryless workers to tasks with dynamically changing demands. Given a set of $w$ workers and a multiset $T \subseteq[t]$ of $|T|=w$ tasks, a memoryless worker-task assignment function is any function $φ$ that assigns the workers $[w]$ to the tasks $T$ based only on the current value of $T$. The assignment function $φ$ is said to have switching cost at most…
▽ More
We study the basic problem of assigning memoryless workers to tasks with dynamically changing demands. Given a set of $w$ workers and a multiset $T \subseteq[t]$ of $|T|=w$ tasks, a memoryless worker-task assignment function is any function $φ$ that assigns the workers $[w]$ to the tasks $T$ based only on the current value of $T$. The assignment function $φ$ is said to have switching cost at most $k$ if, for every task multiset $T$, changing the contents of $T$ by one task changes $φ(T)$ by at most $k$ worker assignments. The goal of memoryless worker task assignment is to construct an assignment function with the smallest possible switching cost.
In past work, the problem of determining the optimal switching cost has been posed as an open question. There are no known sub-linear upper bounds, and after considerable effort, the best known lower bound remains 4 (ICALP 2020).
We show that it is possible to achieve polylogarithmic switching cost. We give a construction via the probabilistic method that achieves switching cost $O(\log w \log (wt))$ and an explicit construction that achieves switching cost $\operatorname{polylog} (wt)$. We also prove a super-constant lower bound on switching cost: we show that for any value of $w$, there exists a value of $t$ for which the optimal switching cost is $w$. Thus it is not possible to achieve a switching cost that is sublinear strictly as a function of $w$.
Finally, we present an application of the worker-task assignment problem to a metric embeddings problem. In particular, we use our results to give the first low-distortion embedding from sparse binary vectors into low-dimensional Hamming space.
△ Less
Submitted 28 April, 2022; v1 submitted 24 August, 2020;
originally announced August 2020.
-
QURATOR: Innovative Technologies for Content and Data Curation
Authors:
Georg Rehm,
Peter Bourgonje,
Stefanie Hegele,
Florian Kintzel,
Julián Moreno Schneider,
Malte Ostendorff,
Karolina Zaczynska,
Armin Berger,
Stefan Grill,
Sören Räuchle,
Jens Rauenbusch,
Lisa Rutenburg,
André Schmidt,
Mikka Wild,
Henry Hoffmann,
Julian Fink,
Sarah Schulz,
Jurica Seva,
Joachim Quantz,
Joachim Böttger,
Josefine Matthey,
Rolf Fricke,
Jan Thomsen,
Adrian Paschke,
Jamal Al Qundus
, et al. (15 additional authors not shown)
Abstract:
In all domains and sectors, the demand for intelligent systems to support the processing and generation of digital content is rapidly increasing. The availability of vast amounts of content and the pressure to publish new content quickly and in rapid succession requires faster, more efficient and smarter processing and generation methods. With a consortium of ten partners from research and industr…
▽ More
In all domains and sectors, the demand for intelligent systems to support the processing and generation of digital content is rapidly increasing. The availability of vast amounts of content and the pressure to publish new content quickly and in rapid succession requires faster, more efficient and smarter processing and generation methods. With a consortium of ten partners from research and industry and a broad range of expertise in AI, Machine Learning and Language Technologies, the QURATOR project, funded by the German Federal Ministry of Education and Research, develops a sustainable and innovative technology platform that provides services to support knowledge workers in various industries to address the challenges they face when curating digital content. The project's vision and ambition is to establish an ecosystem for content curation technologies that significantly pushes the current state of the art and transforms its region, the metropolitan area Berlin-Brandenburg, into a global centre of excellence for curation technologies.
△ Less
Submitted 25 April, 2020;
originally announced April 2020.
-
From Ideation to Implications: Directions for the Internet of Things in the Home
Authors:
Albrecht Kurze,
Arne Berger,
Teresa Denefleh
Abstract:
In this paper we give a brief overview of our approaches and ongoing work for future directions of the Internet of Things (IoT) with a focus on the IoT in the home. We highlight some of our activities including tools and methods for an ideation-driven approach as well as for an implications-driven approach. We point to some findings of workshops and empirical field-studies. We show examples for ne…
▽ More
In this paper we give a brief overview of our approaches and ongoing work for future directions of the Internet of Things (IoT) with a focus on the IoT in the home. We highlight some of our activities including tools and methods for an ideation-driven approach as well as for an implications-driven approach. We point to some findings of workshops and empirical field-studies. We show examples for new classes of idiosyncratic IoT devices, how implications emerge by (mis)using sensor data and how users interacted with IoT systems in shared spaces.
△ Less
Submitted 31 December, 2019;
originally announced December 2019.
-
Degree-Bounded Generalized Polymatroids and Approximating the Metric Many-Visits TSP
Authors:
Kristóf Bérczi,
André Berger,
Matthias Mnich,
Roland Vincze
Abstract:
In the Bounded Degree Matroid Basis Problem, we are given a matroid and a hypergraph on the same ground set, together with costs for the elements of that set as well as lower and upper bounds $f(\varepsilon)$ and $g(\varepsilon)$ for each hyperedge $\varepsilon$. The objective is to find a minimum-cost basis $B$ such that $f(\varepsilon) \leq |B \cap \varepsilon| \leq g(\varepsilon)$ for each hype…
▽ More
In the Bounded Degree Matroid Basis Problem, we are given a matroid and a hypergraph on the same ground set, together with costs for the elements of that set as well as lower and upper bounds $f(\varepsilon)$ and $g(\varepsilon)$ for each hyperedge $\varepsilon$. The objective is to find a minimum-cost basis $B$ such that $f(\varepsilon) \leq |B \cap \varepsilon| \leq g(\varepsilon)$ for each hyperedge $\varepsilon$. Király et al. (Combinatorica, 2012) provided an algorithm that finds a basis of cost at most the optimum value which violates the lower and upper bounds by at most $2 Δ-1$, where $Δ$ is the maximum degree of the hypergraph. When only lower or only upper bounds are present for each hyperedge, this additive error is decreased to $Δ-1$.
We consider an extension of the matroid basis problem to generalized polymatroids, or g-polymatroids, and additionally allow element multiplicities. The Bounded Degree g-polymatroid Element Problem with Multiplicities takes as input a g-polymatroid $Q(p,b)$ instead of a matroid, and besides the lower and upper bounds, each hyperedge $\varepsilon$ has element multiplicities $m_\varepsilon$. Building on the approach of Király et al., we provide an algorithm for finding a solution of cost at most the optimum value, having the same additive approximation guarantee.
As an application, we develop a $1.5$-approximation for the metric Many-Visits TSP, where the goal is to find a minimum-cost tour that visits each city $v$ a positive $r(v)$ number of times. Our approach combines our algorithm for the Bounded Degree g-polymatroid Element Problem with Multiplicities with the principle of Christofides' algorithm from 1976 for the (single-visit) metric TSP, whose approximation guarantee it matches.
△ Less
Submitted 14 December, 2019; v1 submitted 22 November, 2019;
originally announced November 2019.
-
Integrated Bounds for Disintegrated Storage
Authors:
Alon Berger,
Idit Keidar,
Alexander Spiegelman
Abstract:
We point out a somewhat surprising similarity between non-authenticated Byzantine storage, coded storage, and certain emulations of shared registers from smaller ones. A common characteristic in all of these is the inability of reads to safely return a value obtained in a single atomic access to shared storage. We collectively refer to such systems as disintegrated storage, and show integrated spa…
▽ More
We point out a somewhat surprising similarity between non-authenticated Byzantine storage, coded storage, and certain emulations of shared registers from smaller ones. A common characteristic in all of these is the inability of reads to safely return a value obtained in a single atomic access to shared storage. We collectively refer to such systems as disintegrated storage, and show integrated space lower bounds for asynchronous regular wait-free emulations in all of them. In a nutshell, if readers are invisible, then the storage cost of such systems is inherently exponential in the size of written values; otherwise, it is at least linear in the number of readers. Our bounds are asymptotically tight to known algorithms, and thus justify their high costs.
△ Less
Submitted 6 August, 2018; v1 submitted 16 May, 2018;
originally announced May 2018.
-
A time- and space-optimal algorithm for the many-visits TSP
Authors:
André Berger,
László Kozma,
Matthias Mnich,
Roland Vincze
Abstract:
The many-visits traveling salesperson problem (MV-TSP) asks for an optimal tour of $n$ cities that visits each city $c$ a prescribed number $k_c$ of times. Travel costs may be asymmetric, and visiting a city twice in a row may incur a non-zero cost. The MV-TSP problem finds applications in scheduling, geometric approximation, and Hamiltonicity of certain graph families.
The fastest known algorit…
▽ More
The many-visits traveling salesperson problem (MV-TSP) asks for an optimal tour of $n$ cities that visits each city $c$ a prescribed number $k_c$ of times. Travel costs may be asymmetric, and visiting a city twice in a row may incur a non-zero cost. The MV-TSP problem finds applications in scheduling, geometric approximation, and Hamiltonicity of certain graph families.
The fastest known algorithm for MV-TSP is due to Cosmadakis and Papadimitriou (SICOMP, 1984). It runs in time $n^{O(n)} + O(n^3 \log \sum_c k_c )$ and requires $n^{Θ(n)}$ space. An interesting feature of the Cosmadakis-Papadimitriou algorithm is its \emph{logarithmic} dependence on the total length $\sum_c k_c$ of the tour, allowing the algorithm to handle instances with very long tours. The \emph{superexponential} dependence on the number of cities in both the time and space complexity, however, renders the algorithm impractical for all but the narrowest range of this parameter.
In this paper we improve upon the Cosmadakis-Papadimitriou algorithm, giving an MV-TSP algorithm that runs in time $2^{O(n)}$, i.e.\ \emph{single-exponential} in the number of cities, using \emph{polynomial} space. Our algorithm is deterministic, and arguably both simpler and easier to analyse than the original approach of Cosmadakis and Papadimitriou. It involves an optimization over directed spanning trees and a recursive, centroid-based decomposition of trees.
△ Less
Submitted 21 April, 2020; v1 submitted 17 April, 2018;
originally announced April 2018.
-
Algorithm guided outlining of 105 pancreatic cancer liver metastases in Ultrasound
Authors:
Alexander Hann,
Lucas Bettac,
Mark M. Haenle,
Tilmann Graeter,
Andreas W. Berger,
Jens Dreyhaupt,
Dieter Schmalstieg,
Wolfram G. Zoller,
Jan Egger
Abstract:
Manual segmentation of hepatic metastases in ultrasound images acquired from patients suffering from pancreatic cancer is common practice. Semiautomatic measurements promising assistance in this process are often assessed using a small number of lesions performed by examiners who already know the algorithm. In this work, we present the application of an algorithm for the segmentation of liver meta…
▽ More
Manual segmentation of hepatic metastases in ultrasound images acquired from patients suffering from pancreatic cancer is common practice. Semiautomatic measurements promising assistance in this process are often assessed using a small number of lesions performed by examiners who already know the algorithm. In this work, we present the application of an algorithm for the segmentation of liver metastases due to pancreatic cancer using a set of 105 different images of metastases. The algorithm and the two examiners had never assessed the images before. The examiners first performed a manual segmentation and, after five weeks, a semiautomatic segmentation using the algorithm. They were satisfied in up to 90% of the cases with the semiautomatic segmentation results. Using the algorithm was significantly faster and resulted in a median Dice similarity score of over 80%. Estimation of the inter-operator variability by using the intra class correlation coefficient was good with 0.8. In conclusion, the algorithm facilitates fast and accurate segmentation of liver metastases, comparable to the current gold standard of manual segmentation.
△ Less
Submitted 9 October, 2017;
originally announced October 2017.
-
kIP: a Measured Approach to IPv6 Address Anonymization
Authors:
David Plonka,
Arthur Berger
Abstract:
Privacy-minded Internet service operators anonymize IPv6 addresses by truncating them to a fixed length, perhaps due to long-standing use of this technique with IPv4 and a belief that it's "good enough." We claim that simple anonymization by truncation is suspect since it does not entail privacy guarantees nor does it take into account some common address assignment practices observed today. To in…
▽ More
Privacy-minded Internet service operators anonymize IPv6 addresses by truncating them to a fixed length, perhaps due to long-standing use of this technique with IPv4 and a belief that it's "good enough." We claim that simple anonymization by truncation is suspect since it does not entail privacy guarantees nor does it take into account some common address assignment practices observed today. To investigate, with standard activity logs as input, we develop a counting method to determine a lower bound on the number of active IPv6 addresses that are simultaneously assigned, such as those of clients that access World-Wide Web services. In many instances, we find that these empirical measurements offer no evidence that truncating IPv6 addresses to a fixed number of bits, e.g., 48 in common practice, protects individuals' privacy.
To remedy this problem, we propose kIP anonymization, an aggregation method that ensures a certain level of address privacy. Our method adaptively determines variable truncation lengths using parameter k, the desired number of active (rather than merely potential) addresses, e.g., 32 or 256, that can not be distinguished from each other once anonymized. We describe our implementation and present first results of its application to millions of real IPv6 client addresses active over a week's time, demonstrating both feasibility at large scale and ability to automatically adapt to each network's address assignment practice and synthesize a set of anonymous aggregates (prefixes), each of which is guaranteed to cover (contain) at least k of the active addresses. Each address is anonymized by truncating it to the length of its longest matching prefix in that set.
△ Less
Submitted 12 July, 2017;
originally announced July 2017.
-
A unifying framework for fast randomization of ecological networks with fixed (node) degrees
Authors:
Corrie Jacobien Carstens,
Annabell Berger,
Giovanni Strona
Abstract:
The switching model is a Markov chain approach to sample graphs with fixed degree sequence uniformly at random. The recently invented Curveball algorithm for bipartite graphs applies several switches simultaneously (`trades'). Here, we introduce Curveball algorithms for simple (un)directed graphs which use single or simultaneous trades. We show experimentally that these algorithms converge magnitu…
▽ More
The switching model is a Markov chain approach to sample graphs with fixed degree sequence uniformly at random. The recently invented Curveball algorithm for bipartite graphs applies several switches simultaneously (`trades'). Here, we introduce Curveball algorithms for simple (un)directed graphs which use single or simultaneous trades. We show experimentally that these algorithms converge magnitudes faster than the corresponding switching models.
△ Less
Submitted 26 July, 2018; v1 submitted 16 September, 2016;
originally announced September 2016.
-
Query Complexity of Mastermind Variants
Authors:
Aaron Berger,
Christopher Chute,
Matthew Stone
Abstract:
We study variants of Mastermind, a popular board game in which the objective is sequence reconstruction. In this two-player game, the so-called \textit{codemaker} constructs a hidden sequence $H = (h_1, h_2, \ldots, h_n)$ of colors selected from an alphabet $\mathcal{A} = \{1,2,\ldots, k\}$ (\textit{i.e.,} $h_i\in\mathcal{A}$ for all $i\in\{1,2,\ldots, n\}$). The game then proceeds in turns, each…
▽ More
We study variants of Mastermind, a popular board game in which the objective is sequence reconstruction. In this two-player game, the so-called \textit{codemaker} constructs a hidden sequence $H = (h_1, h_2, \ldots, h_n)$ of colors selected from an alphabet $\mathcal{A} = \{1,2,\ldots, k\}$ (\textit{i.e.,} $h_i\in\mathcal{A}$ for all $i\in\{1,2,\ldots, n\}$). The game then proceeds in turns, each of which consists of two parts: in turn $t$, the second player (the \textit{codebreaker}) first submits a query sequence $Q_t = (q_1, q_2, \ldots, q_n)$ with $q_i\in \mathcal{A}$ for all $i$, and second receives feedback $Δ(Q_t, H)$, where $Δ$ is some agreed-upon function of distance between two sequences with $n$ components. The game terminates when $Q_t = H$, and the codebreaker seeks to end the game in as few turns as possible. Throughout we let $f(n,k)$ denote the smallest integer such that the codebreaker can determine any $H$ in $f(n,k)$ turns. We prove three main results: First, when $H$ is known to be a permutation of $\{1,2,\ldots, n\}$, we prove that $f(n, n)\ge n - \log\log n$ for all sufficiently large $n$. Second, we show that Knuth's Minimax algorithm identifies any $H$ in at most $nk$ queries. Third, when feedback is not received until all queries have been submitted, we show that $f(n,k)=Ω(n\log k)$.
△ Less
Submitted 25 September, 2017; v1 submitted 15 July, 2016;
originally announced July 2016.
-
Entropy/IP: Uncovering Structure in IPv6 Addresses
Authors:
Pawel Foremski,
David Plonka,
Arthur Berger
Abstract:
In this paper, we introduce Entropy/IP: a system that discovers Internet address structure based on analyses of a subset of IPv6 addresses known to be active, i.e., training data, gleaned by readily available passive and active means. The system is completely automated and employs a combination of information-theoretic and machine learning techniques to probabilistically model IPv6 addresses. We p…
▽ More
In this paper, we introduce Entropy/IP: a system that discovers Internet address structure based on analyses of a subset of IPv6 addresses known to be active, i.e., training data, gleaned by readily available passive and active means. The system is completely automated and employs a combination of information-theoretic and machine learning techniques to probabilistically model IPv6 addresses. We present results showing that our system is effective in exposing structural characteristics of portions of the IPv6 Internet address space populated by active client, service, and router addresses.
In addition to visualizing the address structure for exploration, the system uses its models to generate candidate target addresses for scanning. For each of 15 evaluated datasets, we train on 1K addresses and generate 1M candidates for scanning. We achieve some success in 14 datasets, finding up to 40% of the generated addresses to be active. In 11 of these datasets, we find active network identifiers (e.g., /64 prefixes or `subnets') not seen in training. Thus, we provide the first evidence that it is practical to discover subnets and hosts by scanning probabilistically selected areas of the IPv6 address space not known to contain active hosts a priori.
△ Less
Submitted 21 November, 2016; v1 submitted 14 June, 2016;
originally announced June 2016.
-
Beyond Counting: New Perspectives on the Active IPv4 Address Space
Authors:
Philipp Richter,
Georgios Smaragdakis,
David Plonka,
Arthur Berger
Abstract:
In this study, we report on techniques and analyses that enable us to capture Internet-wide activity at individual IP address-level granularity by relying on server logs of a large commercial content delivery network (CDN) that serves close to 3 trillion HTTP requests on a daily basis. Across the whole of 2015, these logs recorded client activity involving 1.2 billion unique IPv4 addresses, the hi…
▽ More
In this study, we report on techniques and analyses that enable us to capture Internet-wide activity at individual IP address-level granularity by relying on server logs of a large commercial content delivery network (CDN) that serves close to 3 trillion HTTP requests on a daily basis. Across the whole of 2015, these logs recorded client activity involving 1.2 billion unique IPv4 addresses, the highest ever measured, in agreement with recent estimates. Monthly client IPv4 address counts showed constant growth for years prior, but since 2014, the IPv4 count has stagnated while IPv6 counts have grown. Thus, it seems we have entered an era marked by increased complexity, one in which the sole enumeration of active IPv4 addresses is of little use to characterize recent growth of the Internet as a whole.
With this observation in mind, we consider new points of view in the study of global IPv4 address activity. Our analysis shows significant churn in active IPv4 addresses: the set of active IPv4 addresses varies by as much as 25% over the course of a year. Second, by looking across the active addresses in a prefix, we are able to identify and attribute activity patterns to network restructurings, user behaviors, and, in particular, various address assignment practices. Third, by combining spatio-temporal measures of address utilization with measures of traffic volume, and sampling-based estimates of relative host counts, we present novel perspectives on worldwide IPv4 address activity, including empirical observation of under-utilization in some areas, and complete utilization, or exhaustion, in others.
△ Less
Submitted 9 September, 2016; v1 submitted 1 June, 2016;
originally announced June 2016.
-
Marathon: An open source software library for the analysis of Markov-Chain Monte Carlo algorithms
Authors:
Steffen Rechner,
Annabell Berger
Abstract:
In this paper, we consider the Markov-Chain Monte Carlo (MCMC) approach for random sampling of combinatorial objects. The running time of such an algorithm depends on the total mixing time of the underlying Markov chain and is unknown in general. For some Markov chains, upper bounds on this total mixing time exist but are too large to be applicable in practice. We try to answer the question, wheth…
▽ More
In this paper, we consider the Markov-Chain Monte Carlo (MCMC) approach for random sampling of combinatorial objects. The running time of such an algorithm depends on the total mixing time of the underlying Markov chain and is unknown in general. For some Markov chains, upper bounds on this total mixing time exist but are too large to be applicable in practice. We try to answer the question, whether the total mixing time is close to its upper bounds, or if there is a significant gap between them. In doing so, we present the software library marathon which is designed to support the analysis of MCMC based sampling algorithms. The main application of this library is to compute properties of so-called state graphs which represent the structure of Markov chains. We use marathon to investigate the quality of several bounding methods on four well-known Markov chains for sampling perfect matchings and bipartite graph realizations. In a set of experiments, we compute the total mixing time and several of its bounds for a large number of input instances. We find that the upper bound gained by the famous canonical path method is several magnitudes larger than the total mixing time and deteriorates with growing input size. In contrast, the spectral bound is found to be a precise approximation of the total mixing time.
△ Less
Submitted 14 September, 2016; v1 submitted 19 August, 2015;
originally announced August 2015.
-
Temporal and Spatial Classification of Active IPv6 Addresses
Authors:
David Plonka,
Arthur Berger
Abstract:
There is striking volume of World-Wide Web activity on IPv6 today. In early 2015, one large Content Distribution Network handles 50 billion IPv6 requests per day from hundreds of millions of IPv6 client addresses; billions of unique client addresses are observed per month. Address counts, however, obscure the number of hosts with IPv6 connectivity to the global Internet. There are numerous address…
▽ More
There is striking volume of World-Wide Web activity on IPv6 today. In early 2015, one large Content Distribution Network handles 50 billion IPv6 requests per day from hundreds of millions of IPv6 client addresses; billions of unique client addresses are observed per month. Address counts, however, obscure the number of hosts with IPv6 connectivity to the global Internet. There are numerous address assignment and subnetting options in use; privacy addresses and dynamic subnet pools significantly inflate the number of active IPv6 addresses. As the IPv6 address space is vast, it is infeasible to comprehensively probe every possible unicast IPv6 address. Thus, to survey the characteristics of IPv6 addressing, we perform a year-long passive measurement study, analyzing the IPv6 addresses gleaned from activity logs for all clients accessing a global CDN.
The goal of our work is to develop flexible classification and measurement methods for IPv6, motivated by the fact that its addresses are not merely more numerous; they are different in kind. We introduce the notion of classifying addresses and prefixes in two ways: (1) temporally, according to their instances of activity to discern which addresses can be considered stable; (2) spatially, according to the density or sparsity of aggregates in which active addresses reside. We present measurement and classification results numerically and visually that: provide details on IPv6 address use and structure in global operation across the past year; establish the efficacy of our classification methods; and demonstrate that such classification can clarify dimensions of the Internet that otherwise appear quite blurred by current IPv6 addressing practices.
△ Less
Submitted 17 July, 2015; v1 submitted 26 June, 2015;
originally announced June 2015.
-
Computational Cost Reduction in Learned Transform Classifications
Authors:
Emerson Lopes Machado,
Cristiano Jacques Miosso,
Ricardo von Borries,
Murilo Coutinho,
Pedro de Azevedo Berger,
Thiago Marques,
Ricardo Pezzuol Jacobi
Abstract:
We present a theoretical analysis and empirical evaluations of a novel set of techniques for computational cost reduction of classifiers that are based on learned transform and soft-threshold. By modifying optimization procedures for dictionary and classifier training, as well as the resulting dictionary entries, our techniques allow to reduce the bit precision and to replace each floating-point m…
▽ More
We present a theoretical analysis and empirical evaluations of a novel set of techniques for computational cost reduction of classifiers that are based on learned transform and soft-threshold. By modifying optimization procedures for dictionary and classifier training, as well as the resulting dictionary entries, our techniques allow to reduce the bit precision and to replace each floating-point multiplication by a single integer bit shift. We also show how the optimization algorithms in some dictionary training methods can be modified to penalize higher-energy dictionaries. We applied our techniques with the classifier Learning Algorithm for Soft-Thresholding, testing on the datasets used in its original paper. Our results indicate it is feasible to use solely sums and bit shifts of integers to classify at test time with a limited reduction of the classification accuracy. These low power operations are a valuable trade off in FPGA implementations as they increase the classification throughput while decrease both energy consumption and manufacturing cost.
△ Less
Submitted 30 April, 2016; v1 submitted 25 April, 2015;
originally announced April 2015.
-
Energy Efficient and Reliable Wireless Sensor Networks - An Extension to IEEE 802.15.4e
Authors:
Achim Berger,
Markus Pichler,
Werner Haslmayr,
Andreas Springer
Abstract:
Collecting sensor data in industrial environments from up to some tenth of battery powered sensor nodes with sampling rates up to 100Hz requires energy aware protocols, which avoid collisions and long listening phases. The IEEE 802.15.4 standard focuses on energy aware wireless sensor networks (WSNs) and the Task Group 4e has published an amendment to fulfill up to 100 sensor value transmissions p…
▽ More
Collecting sensor data in industrial environments from up to some tenth of battery powered sensor nodes with sampling rates up to 100Hz requires energy aware protocols, which avoid collisions and long listening phases. The IEEE 802.15.4 standard focuses on energy aware wireless sensor networks (WSNs) and the Task Group 4e has published an amendment to fulfill up to 100 sensor value transmissions per second per sensor node (Low Latency Deterministic Network (LLDN) mode) to satisfy demands of factory automation. To improve the reliability of the data collection in the star topology of the LLDN mode, we propose a relay strategy, which can be performed within the LLDN schedule. Furthermore we propose an extension of the star topology to collect data from two-hop sensor nodes. The proposed Retransmission Mode enables power savings in the sensor node of more than 33%, while reducing the packet loss by up to 50%. To reach this performance, an optimum spatial distribution is necessary, which is discussed in detail.
△ Less
Submitted 6 June, 2014;
originally announced June 2014.
-
Broder's Chain Is Not Rapidly Mixing
Authors:
Annabell Berger,
Steffen Rechner
Abstract:
We prove that Broder's Markov chain for approximate sampling near-perfect and perfect matchings is not rapidly mixing for Hamiltonian, regular, threshold and planar bipartite graphs, filling a gap in the literature. In the second part we experimentally compare Broder's chain with the Markov chain by Jerrum, Sinclair and Vigoda from 2004. For the first time, we provide a systematic experimental inv…
▽ More
We prove that Broder's Markov chain for approximate sampling near-perfect and perfect matchings is not rapidly mixing for Hamiltonian, regular, threshold and planar bipartite graphs, filling a gap in the literature. In the second part we experimentally compare Broder's chain with the Markov chain by Jerrum, Sinclair and Vigoda from 2004. For the first time, we provide a systematic experimental investigation of mixing time bounds for these Markov chains. We observe that the exact total mixing time is in many cases significantly lower than known upper bounds using canonical path or multicommodity flow methods, even if the structure of an underlying state graph is known. In contrast we observe comparatively tighter upper bounds using spectral gaps.
△ Less
Submitted 16 April, 2014;
originally announced April 2014.
-
The Connection between the Number of Realizations for Degree Sequences and Majorization
Authors:
Annabell Berger
Abstract:
The \emph{graph realization problem} is to find for given nonnegative integers $a_1,\dots,a_n$ a simple graph (no loops or multiple edges) such that each vertex $v_i$ has degree $a_i.$ Given pairs of nonnegative integers $(a_1,b_1),\dots,(a_n,b_n),$ (i) the \emph{bipartite realization problem} ask whether there is a bipartite graph (no loops or multiple edges) such that vectors $(a_1,...,a_n)$ and…
▽ More
The \emph{graph realization problem} is to find for given nonnegative integers $a_1,\dots,a_n$ a simple graph (no loops or multiple edges) such that each vertex $v_i$ has degree $a_i.$ Given pairs of nonnegative integers $(a_1,b_1),\dots,(a_n,b_n),$ (i) the \emph{bipartite realization problem} ask whether there is a bipartite graph (no loops or multiple edges) such that vectors $(a_1,...,a_n)$ and $(b_1,...,b_n)$ correspond to the lists of degrees in the two partite sets, (ii) the \emph{digraph realization problem} is to find a digraph (no loops or multiple arcs) such that each vertex $v_i$ has indegree $a_i$ and outdegree $b_i.$\\ The classic literature provides characterizations for the existence of such realizations that are strongly related to the concept of majorization. Aigner and Triesch (1994) extended this approach to a more general result for graphs, leading to an efficient realization algorithm and a short and simple proof for the Erdős-Gallai Theorem. We extend this approach to the bipartite realization problem and the digraph realization problem.\\ Our main result is the connection between majorization and the number of realizations for a degree list in all three problems. We show: if degree list $S'$ majorizes $S$ in a certain sense, then $S$ possesses more realizations than $S'.$ We prove that constant lists possess the largest number of realizations for fixed $n$ and a fixed number of arcs $m$ when $n$ divides $m.$ So-called \emph{minconvex lists} for graphs and bipartite graphs or \emph{opposed minconvex lists} for digraphs maximize the number of realizations when $n$ does not divide $m$.
△ Less
Submitted 1 July, 2014; v1 submitted 21 December, 2012;
originally announced December 2012.
-
How to Attack the NP-complete Dag Realization Problem in Practice
Authors:
Annabell Berger,
Matthias Müller-Hannemann
Abstract:
We study the following fundamental realization problem of directed acyclic graphs (dags). Given a sequence S:=(a_1,b_1),...,(a_n, b_n) with a_i, b_i in Z_0^+, does there exist a dag (no parallel arcs allowed) with labeled vertex set V:= {v_1,...,v_n} such that for all v_i in V indegree and outdegree of v_i match exactly the given numbers a_i and b_i, respectively? Recently this decision problem ha…
▽ More
We study the following fundamental realization problem of directed acyclic graphs (dags). Given a sequence S:=(a_1,b_1),...,(a_n, b_n) with a_i, b_i in Z_0^+, does there exist a dag (no parallel arcs allowed) with labeled vertex set V:= {v_1,...,v_n} such that for all v_i in V indegree and outdegree of v_i match exactly the given numbers a_i and b_i, respectively? Recently this decision problem has been shown to be NP-complete by Nichterlein (2011). However, we can show that several important classes of sequences are efficiently solvable.
In previous work (Berger and Mueller-Hannemann, FCT2011), we have proved that yes-instances always have a special kind of topological order which allows us to reduce the number of possible topological orderings in most cases drastically. This leads to an exact exponential-time algorithm which significantly improves upon a straightforward approach. Moreover, a combination of this exponential-time algorithm with a special strategy gives a linear-time algorithm. Interestingly, in systematic experiments we observed that we could solve a huge majority of all instances by the linear-time heuristic. This motivates us to develop characteristics like dag density and "distance to provably easy sequences" which can give us an indicator how easy or difficult a given sequence can be realized.
Furthermore, we propose a randomized algorithm which exploits our structural insight on topological sortings and uses a number of reduction rules. We observe that it clearly outperforms all other variants and behaves surprisingly well for almost all instances. Another striking observation is that our simple linear-time algorithm solves a set of real-world instances from different domains, namely ordered binary decision diagrams (OBDDs), train and flight schedules, as well as instances derived from food-web networks without any exception.
△ Less
Submitted 16 March, 2012;
originally announced March 2012.
-
Uniform sampling of undirected and directed graphs with a fixed degree sequence
Authors:
Annabell Berger,
Matthias Müller-Hannemann
Abstract:
Many applications in network analysis require algorithms to sample uniformly at random from the set of all graphs with a prescribed degree sequence. We present a Markov chain based approach which converges to the uniform distribution of all realizations for both the directed and undirected case. It remains an open challenge whether these Markov chains are rapidly mixing.
For the case of direct…
▽ More
Many applications in network analysis require algorithms to sample uniformly at random from the set of all graphs with a prescribed degree sequence. We present a Markov chain based approach which converges to the uniform distribution of all realizations for both the directed and undirected case. It remains an open challenge whether these Markov chains are rapidly mixing.
For the case of directed graphs, we also explain in this paper that a popular switching algorithm fails in general to sample uniformly at random because the state graph of the Markov chain decomposes into different isomorphic components. We call degree sequences for which the state graph is strongly connected arc swap sequences. To handle arbitrary degree sequences, we develop two different solutions. The first uses an additional operation (a reorientation of induced directed 3-cycles) which makes the state graph strongly connected, the second selects randomly one of the isomorphic components and samples inside it. Our main contribution is a precise characterization of arc swap sequences, leading to an efficient recognition algorithm. Finally, we point out some interesting consequences for network analysis.
△ Less
Submitted 5 March, 2010; v1 submitted 3 December, 2009;
originally announced December 2009.
-
A Model of Lexical Attraction and Repulsion
Authors:
Doug Beeferman,
Adam Berger,
John Lafferty
Abstract:
This paper introduces new methods based on exponential families for modeling the correlations between words in text and speech. While previous work assumed the effects of word co-occurrence statistics to be constant over a window of several hundred words, we show that their influence is nonstationary on a much smaller time scale. Empirical data drawn from English and Japanese text, as well as co…
▽ More
This paper introduces new methods based on exponential families for modeling the correlations between words in text and speech. While previous work assumed the effects of word co-occurrence statistics to be constant over a window of several hundred words, we show that their influence is nonstationary on a much smaller time scale. Empirical data drawn from English and Japanese text, as well as conversational speech, reveals that the ``attraction'' between words decays exponentially, while stylistic and syntactic contraints create a ``repulsion'' between words that discourages close co-occurrence. We show that these characteristics are well described by simple mixture models based on two-stage exponential distributions which can be trained using the EM algorithm. The resulting distance distributions can then be incorporated as penalizing features in an exponential language model.
△ Less
Submitted 16 June, 1997; v1 submitted 12 June, 1997;
originally announced June 1997.
-
Text Segmentation Using Exponential Models
Authors:
Doug Beeferman,
Adam Berger,
John Lafferty
Abstract:
This paper introduces a new statistical approach to partitioning text automatically into coherent segments. Our approach enlists both short-range and long-range language models to help it sniff out likely sites of topic changes in text. To aid its search, the system consults a set of simple lexical hints it has learned to associate with the presence of boundaries through inspection of a large co…
▽ More
This paper introduces a new statistical approach to partitioning text automatically into coherent segments. Our approach enlists both short-range and long-range language models to help it sniff out likely sites of topic changes in text. To aid its search, the system consults a set of simple lexical hints it has learned to associate with the presence of boundaries through inspection of a large corpus of annotated data. We also propose a new probabilistically motivated error metric for use by the natural language processing and information retrieval communities, intended to supersede precision and recall for appraising segmentation algorithms. Qualitative assessment of our algorithm as well as evaluation using this new metric demonstrate the effectiveness of our approach in two very different domains, Wall Street Journal articles and the TDT Corpus, a collection of newswire articles and broadcast news transcripts.
△ Less
Submitted 12 June, 1997; v1 submitted 11 June, 1997;
originally announced June 1997.
