-
NICE: CVPR 2023 Challenge on Zero-shot Image Captioning
Authors:
Taehoon Kim,
Pyunghwan Ahn,
Sangyun Kim,
Sihaeng Lee,
Mark Marsden,
Alessandra Sala,
Seung Hwan Kim,
Bohyung Han,
Kyoung Mu Lee,
Honglak Lee,
Kyounghoon Bae,
Xiangyu Wu,
Yi Gao,
Hailiang Zhang,
Yang Yang,
Weili Guo,
Jianfeng Lu,
Youngtaek Oh,
Jae Won Cho,
Dong-jin Kim,
In So Kweon,
Junmo Kim,
Wooyoung Kang,
Won Young Jhoo,
Byungseok Roh
, et al. (17 additional authors not shown)
Abstract:
In this report, we introduce NICE (New frontiers for zero-shot Image Captioning Evaluation) project and share the results and outcomes of 2023 challenge. This project is designed to challenge the computer vision community to develop robust image captioning models that advance the state-of-the-art both in terms of accuracy and fairness. Through the challenge, the image captioning models were tested…
▽ More
In this report, we introduce NICE (New frontiers for zero-shot Image Captioning Evaluation) project and share the results and outcomes of 2023 challenge. This project is designed to challenge the computer vision community to develop robust image captioning models that advance the state-of-the-art both in terms of accuracy and fairness. Through the challenge, the image captioning models were tested using a new evaluation dataset that includes a large variety of visual concepts from many domains. There was no specific training data provided for the challenge, and therefore the challenge entries were required to adapt to new types of image descriptions that had not been seen during training. This report includes information on the newly proposed NICE dataset, evaluation methods, challenge results, and technical details of top-ranking entries. We expect that the outcomes of the challenge will contribute to the improvement of AI models on various vision-language tasks.
△ Less
Submitted 10 September, 2023; v1 submitted 5 September, 2023;
originally announced September 2023.
-
Large-Scale Bidirectional Training for Zero-Shot Image Captioning
Authors:
Taehoon Kim,
Mark Marsden,
Pyunghwan Ahn,
Sangyun Kim,
Sihaeng Lee,
Alessandra Sala,
Seung Hwan Kim
Abstract:
When trained on large-scale datasets, image captioning models can understand the content of images from a general domain but often fail to generate accurate, detailed captions. To improve performance, pretraining-and-finetuning has been a key strategy for image captioning. However, we find that large-scale bidirectional training between image and text enables zero-shot image captioning. In this pa…
▽ More
When trained on large-scale datasets, image captioning models can understand the content of images from a general domain but often fail to generate accurate, detailed captions. To improve performance, pretraining-and-finetuning has been a key strategy for image captioning. However, we find that large-scale bidirectional training between image and text enables zero-shot image captioning. In this paper, we introduce Bidirectional Image Text Training in largER Scale, BITTERS, an efficient training and inference framework for zero-shot image captioning. We also propose a new evaluation benchmark which comprises of high quality datasets and an extensive set of metrics to properly evaluate zero-shot captioning accuracy and societal bias. We additionally provide an efficient finetuning approach for keyword extraction. We show that careful selection of large-scale training set and model architecture is the key to achieving zero-shot image captioning.
△ Less
Submitted 1 October, 2023; v1 submitted 12 November, 2022;
originally announced November 2022.
-
Can you recommend content to creatives instead of final consumers? A RecSys based on user's preferred visual styles
Authors:
Raul Gomez Bruballa,
Lauren Burnham-King,
Alessandra Sala
Abstract:
Providing meaningful recommendations in a content marketplace is challenging due to the fact that users are not the final content consumers. Instead, most users are creatives whose interests, linked to the projects they work on, change rapidly and abruptly. To address the challenging task of recommending images to content creators, we design a RecSys that learns visual styles preferences transvers…
▽ More
Providing meaningful recommendations in a content marketplace is challenging due to the fact that users are not the final content consumers. Instead, most users are creatives whose interests, linked to the projects they work on, change rapidly and abruptly. To address the challenging task of recommending images to content creators, we design a RecSys that learns visual styles preferences transversal to the semantics of the projects users work on. We analyze the challenges of the task compared to content-based recommendations driven by semantics, propose an evaluation setup, and explain its applications in a global image marketplace.
This technical report is an extension of the paper "Learning Users' Preferred Visual Styles in an Image Marketplace", presented at ACM RecSys '22.
△ Less
Submitted 23 August, 2022;
originally announced August 2022.
-
Towards Quantifying the Distance between Opinions
Authors:
Saket Gurukar,
Deepak Ajwani,
Sourav Dutta,
Juho Lauri,
Srinivasan Parthasarathy,
Alessandra Sala
Abstract:
Increasingly, critical decisions in public policy, governance, and business strategy rely on a deeper understanding of the needs and opinions of constituent members (e.g. citizens, shareholders). While it has become easier to collect a large number of opinions on a topic, there is a necessity for automated tools to help navigate the space of opinions. In such contexts understanding and quantifying…
▽ More
Increasingly, critical decisions in public policy, governance, and business strategy rely on a deeper understanding of the needs and opinions of constituent members (e.g. citizens, shareholders). While it has become easier to collect a large number of opinions on a topic, there is a necessity for automated tools to help navigate the space of opinions. In such contexts understanding and quantifying the similarity between opinions is key. We find that measures based solely on text similarity or on overall sentiment often fail to effectively capture the distance between opinions. Thus, we propose a new distance measure for capturing the similarity between opinions that leverages the nuanced observation -- similar opinions express similar sentiment polarity on specific relevant entities-of-interest. Specifically, in an unsupervised setting, our distance measure achieves significantly better Adjusted Rand Index scores (up to 56x) and Silhouette coefficients (up to 21x) compared to existing approaches. Similarly, in a supervised setting, our opinion distance measure achieves considerably better accuracy (up to 20% increase) compared to extant approaches that rely on text similarity, stance similarity, and sentiment similarity
△ Less
Submitted 27 January, 2020;
originally announced January 2020.
-
Any-k: Anytime Top-k Tree Pattern Retrieval in Labeled Graphs
Authors:
Xiaofeng Yang,
Deepak Ajwani,
Wolfgang Gatterbauer,
Patrick K. Nicholson,
Mirek Riedewald,
Alessandra Sala
Abstract:
Many problems in areas as diverse as recommendation systems, social network analysis, semantic search, and distributed root cause analysis can be modeled as pattern search on labeled graphs (also called "heterogeneous information networks" or HINs). Given a large graph and a query pattern with node and edge label constraints, a fundamental challenge is to nd the top-k matches ac- cording to a rank…
▽ More
Many problems in areas as diverse as recommendation systems, social network analysis, semantic search, and distributed root cause analysis can be modeled as pattern search on labeled graphs (also called "heterogeneous information networks" or HINs). Given a large graph and a query pattern with node and edge label constraints, a fundamental challenge is to nd the top-k matches ac- cording to a ranking function over edge and node weights. For users, it is di cult to select value k . We therefore propose the novel notion of an any-k ranking algorithm: for a given time budget, re- turn as many of the top-ranked results as possible. Then, given additional time, produce the next lower-ranked results quickly as well. It can be stopped anytime, but may have to continues until all results are returned. This paper focuses on acyclic patterns over arbitrary labeled graphs. We are interested in practical algorithms that effectively exploit (1) properties of heterogeneous networks, in particular selective constraints on labels, and (2) that the users often explore only a fraction of the top-ranked results. Our solution, KARPET, carefully integrates aggressive pruning that leverages the acyclic nature of the query, and incremental guided search. It enables us to prove strong non-trivial time and space guarantees, which is generally considered very hard for this type of graph search problem. Through experimental studies we show that KARPET achieves running times in the order of milliseconds for tree patterns on large networks with millions of nodes and edges.
△ Less
Submitted 10 April, 2018; v1 submitted 16 February, 2018;
originally announced February 2018.
-
Distributed Entity Disambiguation with Per-Mention Learning
Authors:
Tiep Mai,
Bichen Shi,
Patrick K. Nicholson,
Deepak Ajwani,
Alessandra Sala
Abstract:
Entity disambiguation, or mapping a phrase to its canonical representation in a knowledge base, is a fundamental step in many natural language processing applications. Existing techniques based on global ranking models fail to capture the individual peculiarities of the words and hence, either struggle to meet the accuracy requirements of many real-world applications or they are too complex to sat…
▽ More
Entity disambiguation, or mapping a phrase to its canonical representation in a knowledge base, is a fundamental step in many natural language processing applications. Existing techniques based on global ranking models fail to capture the individual peculiarities of the words and hence, either struggle to meet the accuracy requirements of many real-world applications or they are too complex to satisfy real-time constraints of applications.
In this paper, we propose a new disambiguation system that learns specialized features and models for disambiguating each ambiguous phrase in the English language. To train and validate the hundreds of thousands of learning models for this purpose, we use a Wikipedia hyperlink dataset with more than 170 million labelled annotations. We provide an extensive experimental evaluation to show that the accuracy of our approach compares favourably with respect to many state-of-the-art disambiguation systems. The training required for our approach can be easily distributed over a cluster. Furthermore, updating our system for new entities or calibrating it for special ones is a computationally fast process, that does not affect the disambiguation of the other entities.
△ Less
Submitted 20 April, 2016;
originally announced April 2016.
-
Design of false color palettes for grayscale reproduction
Authors:
Filip A. Sala
Abstract:
Design of false color palette is quite easy but some effort has to be done to achieve good dynamic range, contrast and overall appearance of the palette. Such palettes, for instance, are commonly used in scientific papers for presenting the data. However, to lower the cost of the paper most scientists decide to let the data to be printed in grayscale. The same applies to e-book readers based on e-…
▽ More
Design of false color palette is quite easy but some effort has to be done to achieve good dynamic range, contrast and overall appearance of the palette. Such palettes, for instance, are commonly used in scientific papers for presenting the data. However, to lower the cost of the paper most scientists decide to let the data to be printed in grayscale. The same applies to e-book readers based on e-ink where most of them are still grayscale. For majority of false color palettes reproducing them in grayscale results in ambiguous mapping of the colors and may be misleading for the reader. In this article design of false color palettes suitable for grayscale reproduction is described. Due to the monotonic change of luminance of these palettes grayscale representation is very similar to the data directly presented with a grayscale palette. Some suggestions and examples how to design such palettes are provided.
△ Less
Submitted 31 March, 2016; v1 submitted 6 February, 2016;
originally announced February 2016.
-
Profiling user activities with minimal traffic traces
Authors:
Tiep Mai,
Deepak Ajwani,
Alessandra Sala
Abstract:
Understanding user behavior is essential to personalize and enrich a user's online experience. While there are significant benefits to be accrued from the pursuit of personalized services based on a fine-grained behavioral analysis, care must be taken to address user privacy concerns. In this paper, we consider the use of web traces with truncated URLs - each URL is trimmed to only contain the web…
▽ More
Understanding user behavior is essential to personalize and enrich a user's online experience. While there are significant benefits to be accrued from the pursuit of personalized services based on a fine-grained behavioral analysis, care must be taken to address user privacy concerns. In this paper, we consider the use of web traces with truncated URLs - each URL is trimmed to only contain the web domain - for this purpose. While such truncation removes the fine-grained sensitive information, it also strips the data of many features that are crucial to the profiling of user activity. We show how to overcome the severe handicap of lack of crucial features for the purpose of filtering out the URLs representing a user activity from the noisy network traffic trace (including advertisement, spam, analytics, webscripts) with high accuracy. This activity profiling with truncated URLs enables the network operators to provide personalized services while mitigating privacy concerns by storing and sharing only truncated traffic traces.
In order to offset the accuracy loss due to truncation, our statistical methodology leverages specialized features extracted from a group of consecutive URLs that represent a micro user action like web click, chat reply, etc., which we call bursts. These bursts, in turn, are detected by a novel algorithm which is based on our observed characteristics of the inter-arrival time of HTTP records. We present an extensive experimental evaluation on a real dataset of mobile web traces, consisting of more than 130 million records, representing the browsing activities of 10,000 users over a period of 30 days. Our results show that the proposed methodology achieves around 90% accuracy in segregating URLs representing user activities from non-representative URLs.
△ Less
Submitted 7 April, 2015;
originally announced April 2015.
-
Phantom cascades: The effect of hidden nodes on information diffusion
Authors:
Václav Belák,
Afra Mashhadi,
Alessandra Sala,
Donn Morrison
Abstract:
Research on information diffusion generally assumes complete knowledge of the underlying network. However, in the presence of factors such as increasing privacy awareness, restrictions on application programming interfaces (APIs) and sampling strategies, this assumption rarely holds in the real world which in turn leads to an underestimation of the size of information cascades. In this work we stu…
▽ More
Research on information diffusion generally assumes complete knowledge of the underlying network. However, in the presence of factors such as increasing privacy awareness, restrictions on application programming interfaces (APIs) and sampling strategies, this assumption rarely holds in the real world which in turn leads to an underestimation of the size of information cascades. In this work we study the effect of hidden network structure on information diffusion processes. We characterise information cascades through activation paths traversing visible and hidden parts of the network. We quantify diffusion estimation error while varying the amount of hidden structure in five empirical and synthetic network datasets and demonstrate the effect of topological properties on this error. Finally, we suggest practical recommendations for practitioners and propose a model to predict the cascade size with minimal information regarding the underlying network.
△ Less
Submitted 21 May, 2015; v1 submitted 5 February, 2015;
originally announced February 2015.
-
A Geometric Distance Oracle for Large Real-World Graphs
Authors:
Deepak Ajwani,
W. Sean Kennedy,
Alessandra Sala,
Iraj Saniee
Abstract:
Many graph processing algorithms require determination of shortest-path distances between arbitrary numbers of node pairs. Since computation of exact distances between all node-pairs of a large graph, e.g., 10M nodes and up, is prohibitively expensive both in computational time and storage space, distance approximation is often used in place of exact computation. In this paper, we present a novel…
▽ More
Many graph processing algorithms require determination of shortest-path distances between arbitrary numbers of node pairs. Since computation of exact distances between all node-pairs of a large graph, e.g., 10M nodes and up, is prohibitively expensive both in computational time and storage space, distance approximation is often used in place of exact computation. In this paper, we present a novel and scalable distance oracle that leverages the hyperbolic core of real-world large graphs for fast and scalable distance approximation. We show empirically that the proposed oracle significantly outperforms prior oracles on a random set of test cases drawn from public domain graph libraries. There are two sets of prior work against which we benchmark our approach. The first set, which often outperforms other oracles, employs embedding of the graph into low dimensional Euclidean spaces with carefully constructed hyperbolic distances, but provides no guarantees on the distance estimation error. The second set leverages Gromov-type tree contraction of the graph with the additive error guaranteed not to exceed $2δ\log{n}$, where $δ$ is the hyperbolic constant of the graph. We show that our proposed oracle 1) is significantly faster than those oracles that use hyperbolic embedding (first set) with similar approximation error and, perhaps surprisingly, 2) exhibits substantially lower average estimation error compared to Gromov-like tree contractions (second set). We substantiate our claims through numerical computations on a collection of a dozen real world networks and synthetic test cases from multiple domains, ranging in size from 10s of thousand to 10s of millions of nodes.
△ Less
Submitted 19 April, 2014;
originally announced April 2014.
-
Multi-scale Dynamics in a Massive Online Social Network
Authors:
Xiaohan Zhao,
Alessandra Sala,
Christo Wilson,
Xiao Wang,
Sabrina Gaito,
Haitao Zheng,
Ben Y. Zhao
Abstract:
Data confidentiality policies at major social network providers have severely limited researchers' access to large-scale datasets. The biggest impact has been on the study of network dynamics, where researchers have studied citation graphs and content-sharing networks, but few have analyzed detailed dynamics in the massive social networks that dominate the web today. In this paper, we present resu…
▽ More
Data confidentiality policies at major social network providers have severely limited researchers' access to large-scale datasets. The biggest impact has been on the study of network dynamics, where researchers have studied citation graphs and content-sharing networks, but few have analyzed detailed dynamics in the massive social networks that dominate the web today. In this paper, we present results of analyzing detailed dynamics in the Renren social network, covering a period of 2 years when the network grew from 1 user to 19 million users and 199 million edges. Rather than validate a single model of network dynamics, we analyze dynamics at different granularities (user-, community- and network- wide) to determine how much, if any, users are influenced by dynamics processes at different scales. We observe in- dependent predictable processes at each level, and find that while the growth of communities has moderate and sustained impact on users, significant events such as network merge events have a strong but short-lived impact that is quickly dominated by the continuous arrival of new users.
△ Less
Submitted 17 May, 2012;
originally announced May 2012.
-
On the Bursty Evolution of Online Social Networks
Authors:
Sabrina Gaito,
Matteo Zignani,
Gian Paolo Rossi,
Alessandra Sala,
Xiao Wang,
Haitao Zheng,
Ben Y. Zhao
Abstract:
The high level of dynamics in today's online social networks (OSNs) creates new challenges for their infrastructures and providers. In particular, dynamics involving edge creation has direct implications on strategies for resource allocation, data partitioning and replication. Understanding network dynamics in the context of physical time is a critical first step towards a predictive approach towa…
▽ More
The high level of dynamics in today's online social networks (OSNs) creates new challenges for their infrastructures and providers. In particular, dynamics involving edge creation has direct implications on strategies for resource allocation, data partitioning and replication. Understanding network dynamics in the context of physical time is a critical first step towards a predictive approach towards infrastructure management in OSNs. Despite increasing efforts to study social network dynamics, current analyses mainly focus on change over time of static metrics computed on snapshots of social graphs. The limited prior work models network dynamics with respect to a logical clock. In this paper, we present results of analyzing a large timestamped dataset describing the initial growth and evolution of Renren, the leading social network in China. We analyze and model the burstiness of link creation process, using the second derivative, i.e. the acceleration of the degree. This allows us to detect bursts, and to characterize the social activity of a OSN user as one of four phases: acceleration at the beginning of an activity burst, where link creation rate is increasing; deceleration when burst is ending and link creation process is slowing; cruising, when node activity is in a steady state, and complete inactivity.
△ Less
Submitted 25 May, 2012; v1 submitted 30 March, 2012;
originally announced March 2012.
-
Revisiting Degree Distribution Models for Social Graph Analysis
Authors:
Alessandra Sala,
Sabrina Gaito,
Gian Paolo Rossi,
Haitao Zheng,
Ben Y. Zhao
Abstract:
Degree distribution models are incredibly important tools for analyzing and understanding the structure and formation of social networks, and can help guide the design of efficient graph algorithms. In particular, the Power-law degree distribution has long been used to model the structure of online social networks, and is the basis for algorithms and heuristics in graph applications such as influe…
▽ More
Degree distribution models are incredibly important tools for analyzing and understanding the structure and formation of social networks, and can help guide the design of efficient graph algorithms. In particular, the Power-law degree distribution has long been used to model the structure of online social networks, and is the basis for algorithms and heuristics in graph applications such as influence maximization and social search. Along with recent measurement results, our interest in this topic was sparked by our own experimental results on social graphs that deviated significantly from those predicted by a Power-law model. In this work, we seek a deeper understanding of these deviations, and propose an alternative model with significant implications on graph algorithms and applications. We start by quantifying this artifact using a variety of real social graphs, and show that their structures cannot be accurately modeled using elementary distributions including the Power-law. Instead, we propose the Pareto-Lognormal (PLN) model, verify its goodness-of-fit using graphical and statistical methods, and present an analytical study of its asymptotical differences with the Power-law. To demonstrate the quantitative benefits of the PLN model, we compare the results of three wide-ranging graph applications on real social graphs against those on synthetic graphs generated using the PLN and Power-law models. We show that synthetic graphs generated using PLN are much better predictors of degree distributions in real graphs, and produce experimental results with errors that are orders-of-magnitude smaller than those produced by the Power-law model.
△ Less
Submitted 29 July, 2011;
originally announced August 2011.
-
Fast and Scalable Analysis of Massive Social Graphs
Authors:
Xiaohan Zhao,
Alessandra Sala,
Haitao Zheng,
Ben Y. Zhao
Abstract:
Graph analysis is a critical component of applications such as online social networks, protein interactions in biological networks, and Internet traffic analysis. The arrival of massive graphs with hundreds of millions of nodes, e.g. social graphs, presents a unique challenge to graph analysis applications. Most of these applications rely on computing distances between node pairs, which for large…
▽ More
Graph analysis is a critical component of applications such as online social networks, protein interactions in biological networks, and Internet traffic analysis. The arrival of massive graphs with hundreds of millions of nodes, e.g. social graphs, presents a unique challenge to graph analysis applications. Most of these applications rely on computing distances between node pairs, which for large graphs can take minutes to compute using traditional algorithms such as breadth-first-search (BFS). In this paper, we study ways to enable scalable graph processing on today's massive graphs. We explore the design space of graph coordinate systems, a new approach that accurately approximates node distances in constant time by embedding graphs into coordinate spaces. We show that a hyperbolic embedding produces relatively low distortion error, and propose Rigel, a hyperbolic graph coordinate system that lends itself to efficient parallelization across a compute cluster. Rigel produces significantly more accurate results than prior systems, and is naturally parallelizable across compute clusters, allowing it to provide accurate results for graphs up to 43 million nodes. Finally, we show that Rigel's functionality can be easily extended to locate (near-) shortest paths between node pairs. After a one- time preprocessing cost, Rigel answers node-distance queries in 10's of microseconds, and also produces shortest path results up to 18 times faster than prior shortest-path systems with similar levels of accuracy.
△ Less
Submitted 29 July, 2011; v1 submitted 26 July, 2011;
originally announced July 2011.