-
PropSAM: A Propagation-Based Model for Segmenting Any 3D Objects in Multi-Modal Medical Images
Authors:
Zifan Chen,
Xinyu Nan,
Jiazheng Li,
Jie Zhao,
Haifeng Li,
Zilin Lin,
Haoshen Li,
Heyun Chen,
Yiting Liu,
Bin Dong,
Li Zhang,
Lei Tang
Abstract:
Volumetric segmentation is crucial for medical imaging but is often constrained by labor-intensive manual annotations and the need for scenario-specific model training. Furthermore, existing general segmentation models are inefficient due to their design and inferential approaches. Addressing this clinical demand, we introduce PropSAM, a propagation-based segmentation model that optimizes the use…
▽ More
Volumetric segmentation is crucial for medical imaging but is often constrained by labor-intensive manual annotations and the need for scenario-specific model training. Furthermore, existing general segmentation models are inefficient due to their design and inferential approaches. Addressing this clinical demand, we introduce PropSAM, a propagation-based segmentation model that optimizes the use of 3D medical structure information. PropSAM integrates a CNN-based UNet for intra-slice processing with a Transformer-based module for inter-slice propagation, focusing on structural and semantic continuities to enhance segmentation across various modalities. Distinctively, PropSAM operates on a one-view prompt, such as a 2D bounding box or sketch mask, unlike conventional models that require two-view prompts. It has demonstrated superior performance, significantly improving the Dice Similarity Coefficient (DSC) across 44 medical datasets and various imaging modalities, outperforming models like MedSAM and SegVol with an average DSC improvement of 18.1%. PropSAM also maintains stable predictions despite prompt deviations and varying propagation configurations, confirmed by one-way ANOVA tests with P>0.5985 and P>0.6131, respectively. Moreover, PropSAM's efficient architecture enables faster inference speeds (Wilcoxon rank-sum test, P<0.001) and reduces user interaction time by 37.8% compared to two-view prompt models. Its ability to handle irregular and complex objects with robust performance further demonstrates its potential in clinical settings, facilitating more automated and reliable medical imaging analyses with minimal retraining.
△ Less
Submitted 25 August, 2024;
originally announced August 2024.
-
TEAdapter: Supply abundant guidance for controllable text-to-music generation
Authors:
Jialing Zou,
Jiahao Mei,
Xudong Nan,
Jinghua Li,
Daoguo Dong,
Liang He
Abstract:
Although current text-guided music generation technology can cope with simple creative scenarios, achieving fine-grained control over individual text-modality conditions remains challenging as user demands become more intricate. Accordingly, we introduce the TEAcher Adapter (TEAdapter), a compact plugin designed to guide the generation process with diverse control information provided by users. In…
▽ More
Although current text-guided music generation technology can cope with simple creative scenarios, achieving fine-grained control over individual text-modality conditions remains challenging as user demands become more intricate. Accordingly, we introduce the TEAcher Adapter (TEAdapter), a compact plugin designed to guide the generation process with diverse control information provided by users. In addition, we explore the controllable generation of extended music by leveraging TEAdapter control groups trained on data of distinct structural functionalities. In general, we consider controls over global, elemental, and structural levels. Experimental results demonstrate that the proposed TEAdapter enables multiple precise controls and ensures high-quality music generation. Our module is also lightweight and transferable to any diffusion model architecture. Available code and demos will be found soon at https://github.com/Ashley1101/TEAdapter.
△ Less
Submitted 9 August, 2024;
originally announced August 2024.
-
Proximity QA: Unleashing the Power of Multi-Modal Large Language Models for Spatial Proximity Analysis
Authors:
Jianing Li,
Xi Nan,
Ming Lu,
Li Du,
Shanghang Zhang
Abstract:
Multi-modal large language models (MLLMs) have demonstrated remarkable vision-language capabilities, primarily due to the exceptional in-context understanding and multi-task learning strengths of large language models (LLMs). The advent of visual instruction tuning has further enhanced MLLMs' performance in vision-language understanding. However, while existing MLLMs adeptly recognize \textit{what…
▽ More
Multi-modal large language models (MLLMs) have demonstrated remarkable vision-language capabilities, primarily due to the exceptional in-context understanding and multi-task learning strengths of large language models (LLMs). The advent of visual instruction tuning has further enhanced MLLMs' performance in vision-language understanding. However, while existing MLLMs adeptly recognize \textit{what} objects are in an image, they still face challenges in effectively discerning \textit{where} these objects are, particularly along the distance (scene depth) axis. To overcome this limitation in MLLMs, we introduce Proximity Question Answering (Proximity QA), a novel framework designed to enable MLLMs to infer the proximity relationship between objects in images. The framework operates in two phases: the first phase focuses on guiding the models to understand the relative depth of objects, and the second phase further encourages the models to infer the proximity relationships between objects based on their depth perceptions. We also propose a VQA dataset called Proximity-110K, containing additional instructions that incorporate depth information and the proximity relationships of objects. We have conducted extensive experiments to validate Proximity QA's superior ability in depth perception and proximity analysis, outperforming other state-of-the-art MLLMs. Code and dataset will be released at \textcolor{magenta}{https://github.com/NorthSummer/ProximityQA.git}.
△ Less
Submitted 31 January, 2024;
originally announced January 2024.
-
"Generate" the Future of Work through AI: Empirical Evidence from Online Labor Markets
Authors:
Jin Liu,
Xingchen Xu,
Xi Nan,
Yongjun Li,
Yong Tan
Abstract:
Large Language Model (LLM) based generative AI, such as ChatGPT, is considered the first generation of Artificial General Intelligence (AGI), exhibiting zero-shot learning abilities for a wide variety of downstream tasks. Due to its general-purpose and emergent nature, its impact on labor dynamics becomes complex and difficult to anticipate. Leveraging an extensive dataset from a prominent online…
▽ More
Large Language Model (LLM) based generative AI, such as ChatGPT, is considered the first generation of Artificial General Intelligence (AGI), exhibiting zero-shot learning abilities for a wide variety of downstream tasks. Due to its general-purpose and emergent nature, its impact on labor dynamics becomes complex and difficult to anticipate. Leveraging an extensive dataset from a prominent online labor market, we uncover a post-ChatGPT decline in labor demand, supply, and transactions for submarkets pertaining to text-related and programming-related jobs, in comparison to those not directly exposed to ChatGPT's core functionalities. Meanwhile, these affected submarkets exhibit a discernible increase in the complexity of the remaining jobs and a heightened level of competition among freelancers. Intriguingly, our findings indicate that the diminution in the labor supply pertaining to programming is comparatively less pronounced, a phenomenon ascribed to the transition of freelancers previously engaged in text-related tasks now bidding for programming-related opportunities. Although the per-period job diversity freelancers apply for tends to be more limited, those who successfully navigate skill transitions from text to programming demonstrate greater resilience to ChatGPT's overall market contraction impact. As AI becomes increasingly versatile and potent, our paper offers crucial insights into AI's influence on labor markets and individuals' reactions, underscoring the necessity for proactive interventions to address the challenges and opportunities presented by this transformative technology.
△ Less
Submitted 6 June, 2024; v1 submitted 9 August, 2023;
originally announced August 2023.
-
FoodWise: Food Waste Reduction and Behavior Change on Campus with Data Visualization and Gamification
Authors:
Yue Yu,
Sophia Yi,
Xi Nan,
Leo Yu-Ho Lo,
Kento Shigyo,
Liwenhan Xie,
Jeffry Wicaksana,
Kwang-Ting Cheng,
Huamin Qu
Abstract:
Food waste presents a substantial challenge with significant environmental and economic ramifications, and its severity on campus environments is of particular concern. In response to this, we introduce FoodWise, a dual-component system tailored to inspire and incentivize campus communities to reduce food waste. The system consists of a data storytelling dashboard that graphically displays food wa…
▽ More
Food waste presents a substantial challenge with significant environmental and economic ramifications, and its severity on campus environments is of particular concern. In response to this, we introduce FoodWise, a dual-component system tailored to inspire and incentivize campus communities to reduce food waste. The system consists of a data storytelling dashboard that graphically displays food waste information from university canteens, coupled with a mobile web application that encourages users to log their food waste reduction actions and rewards active participants for their efforts.
Deployed during a two-week food-saving campaign at The Hong Kong University of Science and Technology (HKUST) in March 2023, FoodWise engaged over 200 participants from the university community, resulting in the logging of over 800 daily food-saving actions. Feedback collected post-campaign underscores the system's efficacy in elevating user consciousness about food waste and prompting behavioral shifts towards a more sustainable campus. This paper also provides insights for enhancing our system, contributing to a broader discourse on sustainable campus initiatives.
△ Less
Submitted 27 July, 2023; v1 submitted 24 July, 2023;
originally announced July 2023.
-
SGL: Structure Guidance Learning for Camera Localization
Authors:
Xudong Zhang,
Shuang Gao,
Xiaohu Nan,
Haikuan Ning,
Yuchen Yang,
Yishan Ping,
Jixiang Wan,
Shuzhou Dong,
Jijunnan Li,
Yandong Guo
Abstract:
Camera localization is a classical computer vision task that serves various Artificial Intelligence and Robotics applications. With the rapid developments of Deep Neural Networks (DNNs), end-to-end visual localization methods are prosperous in recent years. In this work, we focus on the scene coordinate prediction ones and propose a network architecture named as Structure Guidance Learning (SGL) w…
▽ More
Camera localization is a classical computer vision task that serves various Artificial Intelligence and Robotics applications. With the rapid developments of Deep Neural Networks (DNNs), end-to-end visual localization methods are prosperous in recent years. In this work, we focus on the scene coordinate prediction ones and propose a network architecture named as Structure Guidance Learning (SGL) which utilizes the receptive branch and the structure branch to extract both high-level and low-level features to estimate the 3D coordinates. We design a confidence strategy to refine and filter the predicted 3D observations, which enables us to estimate the camera poses by employing the Perspective-n-Point (PnP) with RANSAC. In the training part, we design the Bundle Adjustment trainer to help the network fit the scenes better. Comparisons with some state-of-the-art (SOTA) methods and sufficient ablation experiments confirm the validity of our proposed architecture.
△ Less
Submitted 11 April, 2023;
originally announced April 2023.
-
Detecting Transaction-based Tax Evasion Activities on Social Media Platforms Using Multi-modal Deep Neural Networks
Authors:
Lelin Zhang,
Xi Nan,
Eva Huang,
Sidong Liu
Abstract:
Social media platforms now serve billions of users by providing convenient means of communication, content sharing and even payment between different users. Due to such convenient and anarchic nature, they have also been used rampantly to promote and conduct business activities between unregistered market participants without paying taxes. Tax authorities worldwide face difficulties in regulating…
▽ More
Social media platforms now serve billions of users by providing convenient means of communication, content sharing and even payment between different users. Due to such convenient and anarchic nature, they have also been used rampantly to promote and conduct business activities between unregistered market participants without paying taxes. Tax authorities worldwide face difficulties in regulating these hidden economy activities by traditional regulatory means. This paper presents a machine learning based Regtech tool for international tax authorities to detect transaction-based tax evasion activities on social media platforms. To build such a tool, we collected a dataset of 58,660 Instagram posts and manually labelled 2,081 sampled posts with multiple properties related to transaction-based tax evasion activities. Based on the dataset, we developed a multi-modal deep neural network to automatically detect suspicious posts. The proposed model combines comments, hashtags and image modalities to produce the final output. As shown by our experiments, the combined model achieved an AUC of 0.808 and F1 score of 0.762, outperforming any single modality models. This tool could help tax authorities to identify audit targets in an efficient and effective manner, and combat social e-commerce tax evasion in scale.
△ Less
Submitted 27 July, 2020;
originally announced July 2020.
-
Towards the AlexNet Moment for Homomorphic Encryption: HCNN, theFirst Homomorphic CNN on Encrypted Data with GPUs
Authors:
Ahmad Al Badawi,
Jin Chao,
Jie Lin,
Chan Fook Mun,
Jun Jie Sim,
Benjamin Hong Meng Tan,
Xiao Nan,
Khin Mi Mi Aung,
Vijay Ramaseshan Chandrasekhar
Abstract:
Deep Learning as a Service (DLaaS) stands as a promising solution for cloud-based inference applications. In this setting, the cloud has a pre-learned model whereas the user has samples on which she wants to run the model. The biggest concern with DLaaS is user privacy if the input samples are sensitive data. We provide here an efficient privacy-preserving system by employing high-end technologies…
▽ More
Deep Learning as a Service (DLaaS) stands as a promising solution for cloud-based inference applications. In this setting, the cloud has a pre-learned model whereas the user has samples on which she wants to run the model. The biggest concern with DLaaS is user privacy if the input samples are sensitive data. We provide here an efficient privacy-preserving system by employing high-end technologies such as Fully Homomorphic Encryption (FHE), Convolutional Neural Networks (CNNs) and Graphics Processing Units (GPUs). FHE, with its widely-known feature of computing on encrypted data, empowers a wide range of privacy-concerned applications. This comes at high cost as it requires enormous computing power. In this paper, we show how to accelerate the performance of running CNNs on encrypted data with GPUs. We evaluated two CNNs to classify homomorphically the MNIST and CIFAR-10 datasets. Our solution achieved a sufficient security level (> 80 bit) and reasonable classification accuracy (99%) and (77.55%) for MNIST and CIFAR-10, respectively. In terms of latency, we could classify an image in 5.16 seconds and 304.43 seconds for MNIST and CIFAR-10, respectively. Our system can also classify a batch of images (> 8,000) without extra overhead.
△ Less
Submitted 18 August, 2020; v1 submitted 2 November, 2018;
originally announced November 2018.
