Showing 1–41 of 41 results for author: Halder, S

An Evaluation of Continual Learning for Advanced Node Semiconductor Defect Inspection

Authors: Amit Prasad, Bappaditya Dey, Victor Blanco, Sandip Halder

Abstract: Deep learning-based semiconductor defect inspection has gained traction in recent years, offering a powerful and versatile approach that provides high accuracy, adaptability, and efficiency in detecting and classifying nano-scale defects. However, semiconductor manufacturing processes are continually evolving, leading to the emergence of new types of defects over time. This presents a significant… ▽ More Deep learning-based semiconductor defect inspection has gained traction in recent years, offering a powerful and versatile approach that provides high accuracy, adaptability, and efficiency in detecting and classifying nano-scale defects. However, semiconductor manufacturing processes are continually evolving, leading to the emergence of new types of defects over time. This presents a significant challenge for conventional supervised defect detectors, as they may suffer from catastrophic forgetting when trained on new defect datasets, potentially compromising performance on previously learned tasks. An alternative approach involves the constant storage of previously trained datasets alongside pre-trained model versions, which can be utilized for (re-)training from scratch or fine-tuning whenever encountering a new defect dataset. However, adhering to such a storage template is impractical in terms of size, particularly when considering High-Volume Manufacturing (HVM). Additionally, semiconductor defect datasets, especially those encompassing stochastic defects, are often limited and expensive to obtain, thus lacking sufficient representation of the entire universal set of defectivity. This work introduces a task-agnostic, meta-learning approach aimed at addressing this challenge, which enables the incremental addition of new defect classes and scales to create a more robust and generalized model for semiconductor defect inspection. We have benchmarked our approach using real resist-wafer SEM (Scanning Electron Microscopy) datasets for two process steps, ADI and AEI, demonstrating its superior performance compared to conventional supervised training methods. △ Less

Submitted 17 July, 2024; originally announced July 2024.

Comments: Accepted for presentation at the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases 2024 Industry Track

Addressing Class Imbalance and Data Limitations in Advanced Node Semiconductor Defect Inspection: A Generative Approach for SEM Images

Authors: Bappaditya Dey, Vic De Ridder, Victor Blanco, Sandip Halder, Bartel Van Waeyenberge

Abstract: Precision in identifying nanometer-scale device-killer defects is crucial in both semiconductor research and development as well as in production processes. The effectiveness of existing ML-based approaches in this context is largely limited by the scarcity of data, as the production of real semiconductor wafer data for training these models involves high financial and time costs. Moreover, the ex… ▽ More Precision in identifying nanometer-scale device-killer defects is crucial in both semiconductor research and development as well as in production processes. The effectiveness of existing ML-based approaches in this context is largely limited by the scarcity of data, as the production of real semiconductor wafer data for training these models involves high financial and time costs. Moreover, the existing simulation methods fall short of replicating images with identical noise characteristics, surface roughness and stochastic variations at advanced nodes. We propose a method for generating synthetic semiconductor SEM images using a diffusion model within a limited data regime. In contrast to images generated through conventional simulation methods, SEM images generated through our proposed DL method closely resemble real SEM images, replicating their noise characteristics and surface roughness adaptively. Our main contributions, which are validated on three different real semiconductor datasets, are: i) proposing a patch-based generative framework utilizing DDPM to create SEM images with intended defect classes, addressing challenges related to class-imbalance and data insufficiency, ii) demonstrating generated synthetic images closely resemble real SEM images acquired from the tool, preserving all imaging conditions and metrology characteristics without any metadata supervision, iii) demonstrating a defect detector trained on generated defect dataset, either independently or combined with a limited real dataset, can achieve similar or improved performance on real wafer SEM images during validation/testing compared to exclusive training on a real defect dataset, iv) demonstrating the ability of the proposed approach to transfer defect types, critical dimensions, and imaging conditions from one specified CD/Pitch and metrology specifications to another, thereby highlighting its versatility. △ Less

Submitted 14 July, 2024; originally announced July 2024.

Comments: 8 pages, 11 figures, to be presented at 2024 International Symposium ELMAR, and published by IEEE in the conference proceedings

Towards Improved Semiconductor Defect Inspection for high-NA EUVL based on SEMI-SuperYOLO-NAS

Authors: Ying-Lin Chen, Jacob Deforce, Vic De Ridder, Bappaditya Dey, Victor Blanco, Sandip Halder, Philippe Leray

Abstract: Due to potential pitch reduction, the semiconductor industry is adopting High-NA EUVL technology. However, its low depth of focus presents challenges for High Volume Manufacturing. To address this, suppliers are exploring thinner photoresists and new underlayers/hardmasks. These may suffer from poor SNR, complicating defect detection. Vision-based ML algorithms offer a promising solution for semic… ▽ More Due to potential pitch reduction, the semiconductor industry is adopting High-NA EUVL technology. However, its low depth of focus presents challenges for High Volume Manufacturing. To address this, suppliers are exploring thinner photoresists and new underlayers/hardmasks. These may suffer from poor SNR, complicating defect detection. Vision-based ML algorithms offer a promising solution for semiconductor defect inspection. However, developing a robust ML model across various image resolutions without explicit training remains a challenge for nano-scale defect inspection. This research's goal is to propose a scale-invariant ADCD framework capable to upscale images, addressing this issue. We propose an improvised ADCD framework as SEMI-SuperYOLO-NAS, which builds upon the baseline YOLO-NAS architecture. This framework integrates a SR assisted branch to aid in learning HR features by the defect detection backbone, particularly for detecting nano-scale defect instances from LR images. Additionally, the SR-assisted branch can recursively generate upscaled images from their corresponding downscaled counterparts, enabling defect detection inference across various image resolutions without requiring explicit training. Moreover, we investigate improved data augmentation strategy aimed at generating diverse and realistic training datasets to enhance model performance. We have evaluated our proposed approach using two original FAB datasets obtained from two distinct processes and captured using two different imaging tools. Finally, we demonstrate zero-shot inference for our model on a new, originating from a process condition distinct from the training dataset and possessing different Pitch characteristics. Experimental validation demonstrates that our proposed ADCD framework aids in increasing the throughput of imaging tools for defect inspection by reducing the required image pixel resolutions. △ Less

Submitted 8 April, 2024; originally announced April 2024.

Dual-sided transparent display

Authors: Suman Halder, Yunho Shin, Yidan Peng, Long Wang, Liye Duan, Paul Schmalenberg, Guangkui Qin, Yuxi Gao, Ercan M. Dede, Deng-Ke Yang, Sean P. Rodrigues

Abstract: In the past decade, display technology has been reimagined to meet the needs of the virtual world. By mapping information onto a scene through a transparent display, users can simultaneously visualize both the real world and layers of virtual elements. However, advances in augmented reality (AR) technology have primarily focused on wearable gear or personal devices. Here we present a single displa… ▽ More In the past decade, display technology has been reimagined to meet the needs of the virtual world. By mapping information onto a scene through a transparent display, users can simultaneously visualize both the real world and layers of virtual elements. However, advances in augmented reality (AR) technology have primarily focused on wearable gear or personal devices. Here we present a single display capable of delivering visual information to observers positioned on either side of the transparent device. This dual-sided display system employs a polymer stabilized liquid crystal waveguide technology to achieve a transparency window of 65% while offering active-matrix control. An early-stage prototype exhibits full-color information via time-sequential processing of a red-green-blue (RGB) light-emitting diode (LED) strip. The dual-sided display provides a perspective on transparent mediums as display devices for human-centric and service-related experiences that can support both enhanced bi-directional user interactions and new media platforms. △ Less

Submitted 7 March, 2024; originally announced March 2024.

Malicious Package Detection using Metadata Information

Authors: S. Halder, M. Bewong, A. Mahboubi, Y. Jiang, R. Islam, Z. Islam, R. Ip, E. Ahmed, G. Ramachandran, A. Babar

Abstract: Protecting software supply chains from malicious packages is paramount in the evolving landscape of software development. Attacks on the software supply chain involve attackers injecting harmful software into commonly used packages or libraries in a software repository. For instance, JavaScript uses Node Package Manager (NPM), and Python uses Python Package Index (PyPi) as their respective package… ▽ More Protecting software supply chains from malicious packages is paramount in the evolving landscape of software development. Attacks on the software supply chain involve attackers injecting harmful software into commonly used packages or libraries in a software repository. For instance, JavaScript uses Node Package Manager (NPM), and Python uses Python Package Index (PyPi) as their respective package repositories. In the past, NPM has had vulnerabilities such as the event-stream incident, where a malicious package was introduced into a popular NPM package, potentially impacting a wide range of projects. As the integration of third-party packages becomes increasingly ubiquitous in modern software development, accelerating the creation and deployment of applications, the need for a robust detection mechanism has become critical. On the other hand, due to the sheer volume of new packages being released daily, the task of identifying malicious packages presents a significant challenge. To address this issue, in this paper, we introduce a metadata-based malicious package detection model, MeMPtec. This model extracts a set of features from package metadata information. These extracted features are classified as either easy-to-manipulate (ETM) or difficult-to-manipulate (DTM) features based on monotonicity and restricted control properties. By utilising these metadata features, not only do we improve the effectiveness of detecting malicious packages, but also we demonstrate its resistance to adversarial attacks in comparison with existing state-of-the-art. Our experiments indicate a significant reduction in both false positives (up to 97.56%) and false negatives (up to 91.86%). △ Less

Submitted 12 February, 2024; originally announced February 2024.

A Robotic Cyber-Physical System for Automated Reality Capture and Visualization in Construction Progress Monitoring

Authors: Srijeet Halder, Kereshmeh Afsari, Abiola Akanmu

Abstract: Effective progress monitoring is crucial for the successful delivery of the construction project within the stipulated time and budget. Construction projects are often monitored irregularly through time-consuming physical site visits by multiple project stakeholders. Remote monitoring using robotic cyber-physical systems (CPS) can make the process more efficient and safer. This article presents a… ▽ More Effective progress monitoring is crucial for the successful delivery of the construction project within the stipulated time and budget. Construction projects are often monitored irregularly through time-consuming physical site visits by multiple project stakeholders. Remote monitoring using robotic cyber-physical systems (CPS) can make the process more efficient and safer. This article presents a conceptual framework for robotic CPS for automated reality capture and visualization for remote progress monitoring in construction. The CPS integrates quadruped robot, Building Information Modelling (BIM), and 360° reality capturing to autonomously capture, and visualize up-to-date site information. Additionally, the study explores the factors affecting acceptance of the proposed robotic CPS through semi-structured interviews with seventeen progress monitoring experts. The findings will guide construction management teams in adopting CPS in construction and drive further research in the human-centered development of CPS for construction. △ Less

Submitted 10 February, 2024; originally announced February 2024.

Applying Machine Learning Models on Metrology Data for Predicting Device Electrical Performance

Authors: Bappaditya Dey, Anh Tuan Ngo, Sara Sacchi, Victor Blanco, Philippe Leray, Sandip Halder

Abstract: Moore Law states that transistor density will double every two years, which is sustained until today due to continuous multi-directional innovations, such as extreme ultraviolet lithography, novel patterning techniques etc., leading the semiconductor industry towards 3nm node and beyond. For any patterning scheme, the most important metric to evaluate the quality of printed patterns is EPE, with o… ▽ More Moore Law states that transistor density will double every two years, which is sustained until today due to continuous multi-directional innovations, such as extreme ultraviolet lithography, novel patterning techniques etc., leading the semiconductor industry towards 3nm node and beyond. For any patterning scheme, the most important metric to evaluate the quality of printed patterns is EPE, with overlay being its largest contribution. Overlay errors can lead to fatal failures of IC devices such as short circuits or broken connections in terms of P2P electrical contacts. Therefore, it is essential to develop effective overlay analysis and control techniques to ensure good functionality of fabricated semiconductor devices. In this work we have used an imec N14 BEOL process flow using LELE patterning technique to print metal layers with minimum pitch of 48nm with 193i lithography. FF structures are decomposed into two mask layers (M1A and M1B) and then the LELE flow is carried out to make the final patterns. Since a single M1 layer is decomposed into two masks, control of overlay between the two masks is critical. The goal of this work is of two-fold as, (a) to quantify the impact of overlay on capacitance and (b) to see if we can predict the final capacitance measurements with selected machine learning models at an early stage. To do so, scatterometry spectra are collected on these electrical test structures at (a)post litho, (b)post TiN hardmask etch, and (c)post Cu plating and CMP. Critical Dimension and overlay measurements for line-space pattern are done with SEM post litho, post etch and post Cu CMP. Various machine learning models are applied to do the capacitance prediction with multiple metrology inputs at different steps of wafer processing. Finally, we demonstrate that by using appropriate machine learning models we are able to do better prediction of electrical results. △ Less

Submitted 20 November, 2023; originally announced December 2023.

Natural Interaction Modalities for Human-CPS Interaction in Construction Progress Monitoring

Authors: Srijeet Halder, Kereshmeh Afsari, Alireza Shojaei

Abstract: This article explores natural interaction modalities for human-cyber-physical systems (CPS) interaction in construction. CPS has been applied in construction for many purposes with the promise of improving the safety and productivity of construction operations. However, there is little research on human-CPS interaction in construction. This study proposes two methodologies for human-CPS interactio… ▽ More This article explores natural interaction modalities for human-cyber-physical systems (CPS) interaction in construction. CPS has been applied in construction for many purposes with the promise of improving the safety and productivity of construction operations. However, there is little research on human-CPS interaction in construction. This study proposes two methodologies for human-CPS interactions for construction progress monitoring - a) hand gesture interaction using transfer learning, and b) voice command interaction using natural language processing. User studies with thirty-two users validated the generalizability of the proposed methodologies. The proposed hand gesture recognition method achieved higher accuracy (99.69% vs 87.72%) and speed (36.05ms vs 578.91ms) than the proposed voice command recognition method, though users performed the progress monitoring task more correctly with voice commands than hand gestures (88% vs 66.1%). The main contribution of the study is the development of an ML pipeline and computational framework to recognize hand gestures and voice commands without the need for a large training dataset for human-CPS interaction. △ Less

Submitted 10 December, 2023; originally announced December 2023.

Comments: 15 pages

Improved Defect Detection and Classification Method for Advanced IC Nodes by Using Slicing Aided Hyper Inference with Refinement Strategy

Authors: Vic De Ridder, Bappaditya Dey, Victor Blanco, Sandip Halder, Bartel Van Waeyenberge

Abstract: In semiconductor manufacturing, lithography has often been the manufacturing step defining the smallest possible pattern dimensions. In recent years, progress has been made towards high-NA (Numerical Aperture) EUVL (Extreme-Ultraviolet-Lithography) paradigm, which promises to advance pattern shrinking (2 nm node and beyond). However, a significant increase in stochastic defects and the complexity… ▽ More In semiconductor manufacturing, lithography has often been the manufacturing step defining the smallest possible pattern dimensions. In recent years, progress has been made towards high-NA (Numerical Aperture) EUVL (Extreme-Ultraviolet-Lithography) paradigm, which promises to advance pattern shrinking (2 nm node and beyond). However, a significant increase in stochastic defects and the complexity of defect detection becomes more pronounced with high-NA. Present defect inspection techniques (both non-machine learning and machine learning based), fail to achieve satisfactory performance at high-NA dimensions. In this work, we investigate the use of the Slicing Aided Hyper Inference (SAHI) framework for improving upon current techniques. Using SAHI, inference is performed on size-increased slices of the SEM images. This leads to the object detector's receptive field being more effective in capturing small defect instances. First, the performance on previously investigated semiconductor datasets is benchmarked across various configurations, and the SAHI approach is demonstrated to substantially enhance the detection of small defects, by approx. 2x. Afterwards, we also demonstrated application of SAHI leads to flawless detection rates on a new test dataset, with scenarios not encountered during training, whereas previous trained models failed. Finally, we formulate an extension of SAHI that does not significantly reduce true-positive predictions while eliminating false-positive predictions. △ Less

Submitted 21 November, 2023; v1 submitted 19 November, 2023; originally announced November 2023.

Comments: 12 pages, 9 figures, to be presented at International Conference on Machine Intelligence with Applications (ICMIA), and to be published in conference proceedings by AIP

Benchmarking Feature Extractors for Reinforcement Learning-Based Semiconductor Defect Localization

Authors: Enrique Dehaerne, Bappaditya Dey, Sandip Halder, Stefan De Gendt

Abstract: As semiconductor patterning dimensions shrink, more advanced Scanning Electron Microscopy (SEM) image-based defect inspection techniques are needed. Recently, many Machine Learning (ML)-based approaches have been proposed for defect localization and have shown impressive results. These methods often rely on feature extraction from a full SEM image and possibly a number of regions of interest. In t… ▽ More As semiconductor patterning dimensions shrink, more advanced Scanning Electron Microscopy (SEM) image-based defect inspection techniques are needed. Recently, many Machine Learning (ML)-based approaches have been proposed for defect localization and have shown impressive results. These methods often rely on feature extraction from a full SEM image and possibly a number of regions of interest. In this study, we propose a deep Reinforcement Learning (RL)-based approach to defect localization which iteratively extracts features from increasingly smaller regions of the input image. We compare the results of 18 agents trained with different feature extractors. We discuss the advantages and disadvantages of different feature extractors as well as the RL-based framework in general for semiconductor defect localization. △ Less

Submitted 18 November, 2023; originally announced November 2023.

Comments: 5 pages, 5 figures, 3 tables

ACM Class: I.4.9

Journal ref: 2023 International Symposium ELMAR, Zadar, Croatia, 2023, pp. 49-53

Deep learning denoiser assisted roughness measurements extraction from thin resists with low Signal-to-Noise Ratio(SNR) SEM images: analysis with SMILE

Authors: Sara Sacchi, Bappaditya Dey, Iacopo Mochi, Sandip Halder, Philippe Leray

Abstract: The technological advance of High Numerical Aperture Extreme Ultraviolet Lithography (High NA EUVL) has opened the gates to extensive researches on thinner photoresists (below 30nm), necessary for the industrial implementation of High NA EUVL. Consequently, images from Scanning Electron Microscopy (SEM) suffer from reduced imaging contrast and low Signal-to-Noise Ratio (SNR), impacting the measure… ▽ More The technological advance of High Numerical Aperture Extreme Ultraviolet Lithography (High NA EUVL) has opened the gates to extensive researches on thinner photoresists (below 30nm), necessary for the industrial implementation of High NA EUVL. Consequently, images from Scanning Electron Microscopy (SEM) suffer from reduced imaging contrast and low Signal-to-Noise Ratio (SNR), impacting the measurement of unbiased Line Edge Roughness (uLER) and Line Width Roughness (uLWR). Thus, the aim of this work is to enhance the SNR of SEM images by using a Deep Learning denoiser and enable robust roughness extraction of the thin resist. For this study, we acquired SEM images of Line-Space (L/S) patterns with a Chemically Amplified Resist (CAR) with different thicknesses (15nm, 20nm, 25nm, 30nm), underlayers (Spin-On-Glass-SOG, Organic Underlayer-OUL) and frames of averaging (4, 8, 16, 32, and 64 Fr). After denoising, a systematic analysis has been carried out on both noisy and denoised images using an open-source metrology software, SMILE 2.3.2, for investigating mean CD, SNR improvement factor, biased and unbiased LWR/LER Power Spectral Density (PSD). Denoised images with lower number of frames present unaltered Critical Dimensions (CDs), enhanced SNR (especially for low number of integration frames), and accurate measurements of uLER and uLWR, with the same accuracy as for noisy images with a consistent higher number of frames. Therefore, images with a small number of integration frames and with SNR < 2 can be successfully denoised, and advantageously used in improving metrology throughput while maintaining reliable roughness measurements for the thin resist. △ Less

Submitted 23 October, 2023; originally announced October 2023.

Automated Semiconductor Defect Inspection in Scanning Electron Microscope Images: a Systematic Review

Authors: Thibault Lechien, Enrique Dehaerne, Bappaditya Dey, Victor Blanco, Sandip Halder, Stefan De Gendt, Wannes Meert

Abstract: A growing need exists for efficient and accurate methods for detecting defects in semiconductor materials and devices. These defects can have a detrimental impact on the efficiency of the manufacturing process, because they cause critical failures and wafer-yield limitations. As nodes and patterns get smaller, even high-resolution imaging techniques such as Scanning Electron Microscopy (SEM) produ… ▽ More A growing need exists for efficient and accurate methods for detecting defects in semiconductor materials and devices. These defects can have a detrimental impact on the efficiency of the manufacturing process, because they cause critical failures and wafer-yield limitations. As nodes and patterns get smaller, even high-resolution imaging techniques such as Scanning Electron Microscopy (SEM) produce noisy images due to operating close to sensitivity levels and due to varying physical properties of different underlayers or resist materials. This inherent noise is one of the main challenges for defect inspection. One promising approach is the use of machine learning algorithms, which can be trained to accurately classify and locate defects in semiconductor samples. Recently, convolutional neural networks have proved to be particularly useful in this regard. This systematic review provides a comprehensive overview of the state of automated semiconductor defect inspection on SEM images, including the most recent innovations and developments. 38 publications were selected on this topic, indexed in IEEE Xplore and SPIE databases. For each of these, the application, methodology, dataset, results, limitations and future work were summarized. A comprehensive overview and analysis of their methods is provided. Finally, promising avenues for future work in the field of SEM-based defect inspection are suggested. △ Less

Submitted 18 August, 2023; v1 submitted 16 August, 2023; originally announced August 2023.

Comments: 16 pages, 12 figures, 3 tables

SEMI-CenterNet: A Machine Learning Facilitated Approach for Semiconductor Defect Inspection

Authors: Vic De Ridder, Bappaditya Dey, Enrique Dehaerne, Sandip Halder, Stefan De Gendt, Bartel Van Waeyenberge

Abstract: Continual shrinking of pattern dimensions in the semiconductor domain is making it increasingly difficult to inspect defects due to factors such as the presence of stochastic noise and the dynamic behavior of defect patterns and types. Conventional rule-based methods and non-parametric supervised machine learning algorithms like KNN mostly fail at the requirements of semiconductor defect inspectio… ▽ More Continual shrinking of pattern dimensions in the semiconductor domain is making it increasingly difficult to inspect defects due to factors such as the presence of stochastic noise and the dynamic behavior of defect patterns and types. Conventional rule-based methods and non-parametric supervised machine learning algorithms like KNN mostly fail at the requirements of semiconductor defect inspection at these advanced nodes. Deep Learning (DL)-based methods have gained popularity in the semiconductor defect inspection domain because they have been proven robust towards these challenging scenarios. In this research work, we have presented an automated DL-based approach for efficient localization and classification of defects in SEM images. We have proposed SEMI-CenterNet (SEMI-CN), a customized CN architecture trained on SEM images of semiconductor wafer defects. The use of the proposed CN approach allows improved computational efficiency compared to previously studied DL models. SEMI-CN gets trained to output the center, class, size, and offset of a defect instance. This is different from the approach of most object detection models that use anchors for bounding box prediction. Previous methods predict redundant bounding boxes, most of which are discarded in postprocessing. CN mitigates this by only predicting boxes for likely defect center points. We train SEMI-CN on two datasets and benchmark two ResNet backbones for the framework. Initially, ResNet models pretrained on the COCO dataset undergo training using two datasets separately. Primarily, SEMI-CN shows significant improvement in inference time against previous research works. Finally, transfer learning (using weights of custom SEM dataset) is applied from ADI dataset to AEI dataset and vice-versa, which reduces the required training time for both backbones to reach the best mAP against conventional training method. △ Less

Submitted 15 August, 2023; v1 submitted 14 August, 2023; originally announced August 2023.

YOLOv8 for Defect Inspection of Hexagonal Directed Self-Assembly Patterns: A Data-Centric Approach

Authors: Enrique Dehaerne, Bappaditya Dey, Hossein Esfandiar, Lander Verstraete, Hyo Seon Suh, Sandip Halder, Stefan De Gendt

Abstract: Shrinking pattern dimensions leads to an increased variety of defect types in semiconductor devices. This has spurred innovation in patterning approaches such as Directed self-assembly (DSA) for which no traditional, automatic defect inspection software exists. Machine Learning-based SEM image analysis has become an increasingly popular research topic for defect inspection with supervised ML model… ▽ More Shrinking pattern dimensions leads to an increased variety of defect types in semiconductor devices. This has spurred innovation in patterning approaches such as Directed self-assembly (DSA) for which no traditional, automatic defect inspection software exists. Machine Learning-based SEM image analysis has become an increasingly popular research topic for defect inspection with supervised ML models often showing the best performance. However, little research has been done on obtaining a dataset with high-quality labels for these supervised models. In this work, we propose a method for obtaining coherent and complete labels for a dataset of hexagonal contact hole DSA patterns while requiring minimal quality control effort from a DSA expert. We show that YOLOv8, a state-of-the-art neural network, achieves defect detection precisions of more than 0.9 mAP on our final dataset which best reflects DSA expert defect labeling expectations. We discuss the strengths and limitations of our proposed labeling approach and suggest directions for future work in data-centric ML-based defect inspection. △ Less

Submitted 28 July, 2023; originally announced July 2023.

Comments: 8 pages, 10 figures, accepted for the 38th EMLC Conference 2023

ACM Class: I.4.9

Journal ref: Proceedings Volume 12802, 38th European Mask and Lithography Conference (EMLC 2023); 128020S (2023)

SEMI-DiffusionInst: A Diffusion Model Based Approach for Semiconductor Defect Classification and Segmentation

Authors: Vic De Ridder, Bappaditya Dey, Sandip Halder, Bartel Van Waeyenberge

Abstract: With continuous progression of Moore's Law, integrated circuit (IC) device complexity is also increasing. Scanning Electron Microscope (SEM) image based extensive defect inspection and accurate metrology extraction are two main challenges in advanced node (2 nm and beyond) technology. Deep learning (DL) algorithm based computer vision approaches gained popularity in semiconductor defect inspection… ▽ More With continuous progression of Moore's Law, integrated circuit (IC) device complexity is also increasing. Scanning Electron Microscope (SEM) image based extensive defect inspection and accurate metrology extraction are two main challenges in advanced node (2 nm and beyond) technology. Deep learning (DL) algorithm based computer vision approaches gained popularity in semiconductor defect inspection over last few years. In this research work, a new semiconductor defect inspection framework "SEMI-DiffusionInst" is investigated and compared to previous frameworks. To the best of the authors' knowledge, this work is the first demonstration to accurately detect and precisely segment semiconductor defect patterns by using a diffusion model. Different feature extractor networks as backbones and data sampling strategies are investigated towards achieving a balanced trade-off between precision and computing efficiency. Our proposed approach outperforms previous work on overall mAP and performs comparatively better or as per for almost all defect classes (per class APs). The bounding box and segmentation mAPs achieved by the proposed SEMI-DiffusionInst model are improved by 3.83% and 2.10%, respectively. Among individual defect types, precision on line collapse and thin bridge defects are improved approximately 15\% on detection task for both defect types. It has also been shown that by tuning inference hyperparameters, inference time can be improved significantly without compromising model precision. Finally, certain limitations and future work strategy to overcome them are discussed. △ Less

Submitted 16 August, 2023; v1 submitted 17 July, 2023; originally announced July 2023.

Comments: 6 pages, 5 figures, To be published by IEEE in the proceedings of the 2023 ELMAR conference

A Deep Learning Framework for Verilog Autocompletion Towards Design and Verification Automation

Authors: Enrique Dehaerne, Bappaditya Dey, Sandip Halder, Stefan De Gendt

Abstract: Innovative Electronic Design Automation (EDA) solutions are important to meet the design requirements for increasingly complex electronic devices. Verilog, a hardware description language, is widely used for the design and verification of digital circuits and is synthesized using specific EDA tools. However, writing code is a repetitive and time-intensive task. This paper proposes, primarily, a no… ▽ More Innovative Electronic Design Automation (EDA) solutions are important to meet the design requirements for increasingly complex electronic devices. Verilog, a hardware description language, is widely used for the design and verification of digital circuits and is synthesized using specific EDA tools. However, writing code is a repetitive and time-intensive task. This paper proposes, primarily, a novel deep learning framework for training a Verilog autocompletion model and, secondarily, a Verilog dataset of files and snippets obtained from open-source repositories. The framework involves integrating models pretrained on general programming language data and finetuning them on a dataset curated to be similar to a target downstream task. This is validated by comparing different pretrained models trained on different subsets of the proposed Verilog dataset using multiple evaluation metrics. These experiments demonstrate that the proposed framework achieves better BLEU, ROUGE-L, and chrF scores by 9.5%, 6.7%, and 6.9%, respectively, compared to a model trained from scratch. Code and data are made available at: https://github.com/99EnriqueD/verilog_autocompletion . △ Less

Submitted 7 June, 2023; v1 submitted 26 April, 2023; originally announced April 2023.

Comments: Updated text to correct language errors and added a link to supplementary code and data (https://github.com/99EnriqueD/verilog_autocompletion). 6 pages, 3 figures, 4 tables. To be presented as a WIP poster at DAC 2023

ACM Class: I.2.2

SEMI-PointRend: Improved Semiconductor Wafer Defect Classification and Segmentation as Rendering

Authors: MinJin Hwang, Bappaditya Dey, Enrique Dehaerne, Sandip Halder, Young-han Shin

Abstract: In this study, we applied the PointRend (Point-based Rendering) method to semiconductor defect segmentation. PointRend is an iterative segmentation algorithm inspired by image rendering in computer graphics, a new image segmentation method that can generate high-resolution segmentation masks. It can also be flexibly integrated into common instance segmentation meta-architecture such as Mask-RCNN a… ▽ More In this study, we applied the PointRend (Point-based Rendering) method to semiconductor defect segmentation. PointRend is an iterative segmentation algorithm inspired by image rendering in computer graphics, a new image segmentation method that can generate high-resolution segmentation masks. It can also be flexibly integrated into common instance segmentation meta-architecture such as Mask-RCNN and semantic meta-architecture such as FCN. We implemented a model, termed as SEMI-PointRend, to generate precise segmentation masks by applying the PointRend neural network module. In this paper, we focus on comparing the defect segmentation predictions of SEMI-PointRend and Mask-RCNN for various defect types (line-collapse, single bridge, thin bridge, multi bridge non-horizontal). We show that SEMI-PointRend can outperforms Mask R-CNN by up to 18.8% in terms of segmentation mean average precision. △ Less

Submitted 19 February, 2023; originally announced February 2023.

Comments: 7 pages, 6 figures, 5 tables. To be published by SPIE in the proceedings of Metrology, Inspection, and Process Control XXXVII

ACM Class: I.4.9

Journal ref: Proc. SPIE 12496, Metrology, Inspection, and Process Control XXXVII, 1249608 (27 April 2023)

Optimizing YOLOv7 for Semiconductor Defect Detection

Authors: Enrique Dehaerne, Bappaditya Dey, Sandip Halder, Stefan De Gendt

Abstract: The field of object detection using Deep Learning (DL) is constantly evolving with many new techniques and models being proposed. YOLOv7 is a state-of-the-art object detector based on the YOLO family of models which have become popular for industrial applications. One such possible application domain can be semiconductor defect inspection. The performance of any machine learning model depends on i… ▽ More The field of object detection using Deep Learning (DL) is constantly evolving with many new techniques and models being proposed. YOLOv7 is a state-of-the-art object detector based on the YOLO family of models which have become popular for industrial applications. One such possible application domain can be semiconductor defect inspection. The performance of any machine learning model depends on its hyperparameters. Furthermore, combining predictions of one or more models in different ways can also affect performance. In this research, we experiment with YOLOv7, a recently proposed, state-of-the-art object detector, by training and evaluating models with different hyperparameters to investigate which ones improve performance in terms of detection precision for semiconductor line space pattern defects. The base YOLOv7 model with default hyperparameters and Non Maximum Suppression (NMS) prediction combining outperforms all RetinaNet models from previous work in terms of mean Average Precision (mAP). We find that vertically flipping images randomly during training yields a 3% improvement in the mean AP of all defect classes. Other hyperparameter values improved AP only for certain classes compared to the default model. Combining models that achieve the best AP for different defect classes was found to be an effective ensembling strategy. Combining predictions from ensembles using Weighted Box Fusion (WBF) prediction gave the best performance. The best ensemble with WBF improved on the mAP of the default model by 10%. △ Less

Submitted 19 February, 2023; originally announced February 2023.

Comments: 8 pages, 4 figures, 5 tables. To be published by SPIE in the proceedings of Metrology, Inspection, and Process Control XXXVII

ACM Class: I.4.9

Journal ref: Proc. SPIE 12496, Metrology, Inspection, and Process Control XXXVII, 124962D (27 April 2023)

An Overview of Artificial Intelligence-based Soft Upper Limb Exoskeleton for Rehabilitation: A Descriptive Review

Authors: Sanjukta Halder, Amit Kumar

Abstract: The upper limb robotic exoskeleton is an electromechanical device which use to recover a patients motor dysfunction in the rehabilitation field. It can provide repetitive, comprehensive, focused, positive, and precise training to regain the joints and muscles capability. It has been shown that existing robotic exoskeletons are generally used rigid motors and mechanical structures. Soft robotic dev… ▽ More The upper limb robotic exoskeleton is an electromechanical device which use to recover a patients motor dysfunction in the rehabilitation field. It can provide repetitive, comprehensive, focused, positive, and precise training to regain the joints and muscles capability. It has been shown that existing robotic exoskeletons are generally used rigid motors and mechanical structures. Soft robotic devices can be a correct substitute for rigid ones. Soft exosuits are flexible, portable, comfortable, user-friendly, low-cost, and travel-friendly. Somehow, they need expertise or therapist to assist those devices. Also, they cannot be adaptable to different patients with non-identical physical parameters and various rehabilitation needs. For that reason, nowadays we need intelligent exoskeletons during rehabilitation which have to learn from patients previous data and act according to it with patients intention. There also has a big gap between theoretical and practical applications for using those exoskeletons. Most of the intelligent exoskeletons are prototype in manner. To solve this problem, the robotic exoskeleton should be made both criteria as ergonomic and portable. The exoskeletons have to the power of decision-making to avoid the presence of expertise. In this growing field, the present trend is to make the exoskeleton intelligent and make it more reliable to use in clinical practice. △ Less

Submitted 28 January, 2023; v1 submitted 11 January, 2023; originally announced January 2023.

Overview of The MediaEval 2022 Predicting Video Memorability Task

Authors: Lorin Sweeney, Mihai Gabriel Constantin, Claire-Hélène Demarty, Camilo Fosco, Alba G. Seco de Herrera, Sebastian Halder, Graham Healy, Bogdan Ionescu, Ana Matran-Fernandez, Alan F. Smeaton, Mushfika Sultana

Abstract: This paper describes the 5th edition of the Predicting Video Memorability Task as part of MediaEval2022. This year we have reorganised and simplified the task in order to lubricate a greater depth of inquiry. Similar to last year, two datasets are provided in order to facilitate generalisation, however, this year we have replaced the TRECVid2019 Video-to-Text dataset with the VideoMem dataset in o… ▽ More This paper describes the 5th edition of the Predicting Video Memorability Task as part of MediaEval2022. This year we have reorganised and simplified the task in order to lubricate a greater depth of inquiry. Similar to last year, two datasets are provided in order to facilitate generalisation, however, this year we have replaced the TRECVid2019 Video-to-Text dataset with the VideoMem dataset in order to remedy underlying data quality issues, and to prioritise short-term memorability prediction by elevating the Memento10k dataset as the primary dataset. Additionally, a fully fledged electroencephalography (EEG)-based prediction sub-task is introduced. In this paper, we outline the core facets of the task and its constituent sub-tasks; describing the datasets, evaluation metrics, and requirements for participant submissions. △ Less

Submitted 13 December, 2022; originally announced December 2022.

Comments: 6 pages. In: MediaEval Multimedia Benchmark Workshop Working Notes, 2022

Experiences from the MediaEval Predicting Media Memorability Task

Authors: Alba García Deco de Herrera, Mihai Gabriel Constantin, Chaire-Hélène Demarty, Camilo Fosco, Sebastian Halder, Graham Healy, Bogdan Ionescu, Ana Matran-Fernandez, Alan F. Smeaton, Mushfika Sultana, Lorin Sweeney

Abstract: The Predicting Media Memorability task in the MediaEval evaluation campaign has been running annually since 2018 and several different tasks and data sets have been used in this time. This has allowed us to compare the performance of many memorability prediction techniques on the same data and in a reproducible way and to refine and improve on those techniques. The resources created to compute med… ▽ More The Predicting Media Memorability task in the MediaEval evaluation campaign has been running annually since 2018 and several different tasks and data sets have been used in this time. This has allowed us to compare the performance of many memorability prediction techniques on the same data and in a reproducible way and to refine and improve on those techniques. The resources created to compute media memorability are now being used by researchers well beyond the actual evaluation campaign. In this paper we present a summary of the task, including the collective lessons we have learned for the research community. △ Less

Submitted 7 December, 2022; originally announced December 2022.

Comments: 7 pages, 2 figures, 1 table. Presented at the NeurIPS 2022 Workshop on Memory in Artificial and Real Intelligence (MemARI), 2 December 2022, New Orleans, USA

FinBERT-LSTM: Deep Learning based stock price prediction using News Sentiment Analysis

Authors: Shayan Halder

Abstract: Economy is severely dependent on the stock market. An uptrend usually corresponds to prosperity while a downtrend correlates to recession. Predicting the stock market has thus been a centre of research and experiment for a long time. Being able to predict short term movements in the market enables investors to reap greater returns on their investments. Stock prices are extremely volatile and sensi… ▽ More Economy is severely dependent on the stock market. An uptrend usually corresponds to prosperity while a downtrend correlates to recession. Predicting the stock market has thus been a centre of research and experiment for a long time. Being able to predict short term movements in the market enables investors to reap greater returns on their investments. Stock prices are extremely volatile and sensitive to financial market. In this paper we use Deep Learning networks to predict stock prices, assimilating financial, business and technology news articles which present information about the market. First, we create a simple Multilayer Perceptron (MLP) network and then expand into more complex Recurrent Neural Network (RNN) like Long Short Term Memory (LSTM), and finally propose FinBERT-LSTM model, which integrates news article sentiments to predict stock price with greater accuracy by analysing short-term market information. We then train the model on NASDAQ-100 index stock data and New York Times news articles to evaluate the performance of MLP, LSTM, FinBERT-LSTM models using mean absolute error (MAE), mean absolute percentage error (MAPE) and accuracy metrics. △ Less

Submitted 11 November, 2022; originally announced November 2022.

Deep Learning based Defect classification and detection in SEM images: A Mask R-CNN approach

Authors: Bappaditya Dey, Enrique Dehaerne, Kasem Khalil, Sandip Halder, Philippe Leray, Magdy A. Bayoumi

Abstract: In this research work, we have demonstrated the application of Mask-RCNN (Regional Convolutional Neural Network), a deep-learning algorithm for computer vision and specifically object detection, to semiconductor defect inspection domain. Stochastic defect detection and classification during semiconductor manufacturing has grown to be a challenging task as we continuously shrink circuit pattern dim… ▽ More In this research work, we have demonstrated the application of Mask-RCNN (Regional Convolutional Neural Network), a deep-learning algorithm for computer vision and specifically object detection, to semiconductor defect inspection domain. Stochastic defect detection and classification during semiconductor manufacturing has grown to be a challenging task as we continuously shrink circuit pattern dimensions (e.g., for pitches less than 32 nm). Defect inspection and analysis by state-of-the-art optical and e-beam inspection tools is generally driven by some rule-based techniques, which in turn often causes to misclassification and thereby necessitating human expert intervention. In this work, we have revisited and extended our previous deep learning-based defect classification and detection method towards improved defect instance segmentation in SEM images with precise extent of defect as well as generating a mask for each defect category/instance. This also enables to extract and calibrate each segmented mask and quantify the pixels that make up each mask, which in turn enables us to count each categorical defect instances as well as to calculate the surface area in terms of pixels. We are aiming at detecting and segmenting different types of inter-class stochastic defect patterns such as bridge, break, and line collapse as well as to differentiate accurately between intra-class multi-categorical defect bridge scenarios (as thin/single/multi-line/horizontal/non-horizontal) for aggressive pitches as well as thin resists (High NA applications). Our proposed approach demonstrates its effectiveness both quantitatively and qualitatively. △ Less

Submitted 3 November, 2022; originally announced November 2022.

Comments: arXiv admin note: text overlap with arXiv:2206.13505

MRI-MECH: Mechanics-informed MRI to estimate esophageal health

Authors: Sourav Halder, Ethan M. Johnson, Jun Yamasaki, Peter J. Kahrilas, Michael Markl, John E. Pandolfino, Neelesh A. Patankar

Abstract: Dynamic magnetic resonance imaging (MRI) is a popular medical imaging technique to generate image sequences of the flow of a contrast material inside tissues and organs. However, its application to imaging bolus movement through the esophagus has only been demonstrated in few feasibility studies and is relatively unexplored. In this work, we present a computational framework called mechanics-infor… ▽ More Dynamic magnetic resonance imaging (MRI) is a popular medical imaging technique to generate image sequences of the flow of a contrast material inside tissues and organs. However, its application to imaging bolus movement through the esophagus has only been demonstrated in few feasibility studies and is relatively unexplored. In this work, we present a computational framework called mechanics-informed MRI (MRI-MECH) that enhances that capability thereby increasing the applicability of dynamic MRI for diagnosing esophageal disorders. Pineapple juice was used as the swallowed contrast material for the dynamic MRI and the MRI image sequence was used as input to the MRI-MECH. The MRI-MECH modeled the esophagus as a flexible one-dimensional tube and the elastic tube walls followed a linear tube law. Flow through the esophagus was then governed by one-dimensional mass and momentum conservation equations. These equations were solved using a physics-informed neural network (PINN). The PINN minimized the difference between the measurements from the MRI and model predictions ensuring that the physics of the fluid flow problem was always followed. MRI-MECH calculated the fluid velocity and pressure during esophageal transit and estimated the mechanical health of the esophagus by calculating wall stiffness and active relaxation. Additionally, MRI-MECH predicted missing information about the lower esophageal sphincter during the emptying process, demonstrating its applicability to scenarios with missing data or poor image resolution. In addition to potentially improving clinical decisions based on quantitative estimates of the mechanical health of the esophagus, MRI-MECH can also be enhanced for application to other medical imaging modalities to enhance their functionality as well. △ Less

Submitted 15 September, 2022; originally announced September 2022.

Comments: 21 pages, 15 figures

Journal ref: Frontiers in Physiology. 14 (2023)

Where's the Learning in Representation Learning for Compositional Semantics and the Case of Thematic Fit

Authors: Mughilan Muthupari, Samrat Halder, Asad Sayeed, Yuval Marton

Abstract: Observing that for certain NLP tasks, such as semantic role prediction or thematic fit estimation, random embeddings perform as well as pretrained embeddings, we explore what settings allow for this and examine where most of the learning is encoded: the word embeddings, the semantic role embeddings, or ``the network''. We find nuanced answers, depending on the task and its relation to the training… ▽ More Observing that for certain NLP tasks, such as semantic role prediction or thematic fit estimation, random embeddings perform as well as pretrained embeddings, we explore what settings allow for this and examine where most of the learning is encoded: the word embeddings, the semantic role embeddings, or ``the network''. We find nuanced answers, depending on the task and its relation to the training objective. We examine these representation learning aspects in multi-task learning, where role prediction and role-filling are supervised tasks, while several thematic fit tasks are outside the models' direct supervision. We observe a non-monotonous relation between some tasks' quality score and the training data size. In order to better understand this observation, we analyze these results using easier, per-verb versions of these tasks. △ Less

Submitted 23 October, 2022; v1 submitted 9 August, 2022; originally announced August 2022.

Comments: Published in Blackbox NLP workshop, EMNLP 2022. 12 pages including Appendices, 1 figure (with 6 sub-figures)

Overview of the EEG Pilot Subtask at MediaEval 2021: Predicting Media Memorability

Authors: Lorin Sweeney, Ana Matran-Fernandez, Sebastian Halder, Alba G. Seco de Herrera, Alan Smeaton, Graham Healy

Abstract: The aim of the Memorability-EEG pilot subtask at MediaEval'2021 is to promote interest in the use of neural signals -- either alone or in combination with other data sources -- in the context of predicting video memorability by highlighting the utility of EEG data. The dataset created consists of pre-extracted features from EEG recordings of subjects while watching a subset of videos from Predicti… ▽ More The aim of the Memorability-EEG pilot subtask at MediaEval'2021 is to promote interest in the use of neural signals -- either alone or in combination with other data sources -- in the context of predicting video memorability by highlighting the utility of EEG data. The dataset created consists of pre-extracted features from EEG recordings of subjects while watching a subset of videos from Predicting Media Memorability subtask 1. This demonstration pilot gives interested researchers a sense of how neural signals can be used without any prior domain knowledge, and enables them to do so in a future memorability task. The dataset can be used to support the exploration of novel machine learning and processing strategies for predicting video memorability, while potentially increasing interdisciplinary interest in the subject of memorability, and opening the door to new combined EEG-computer vision approaches. △ Less

Submitted 15 December, 2021; originally announced January 2022.

Comments: 3 pages

Overview of The MediaEval 2021 Predicting Media Memorability Task

Authors: Rukiye Savran Kiziltepe, Mihai Gabriel Constantin, Claire-Helene Demarty, Graham Healy, Camilo Fosco, Alba Garcia Seco de Herrera, Sebastian Halder, Bogdan Ionescu, Ana Matran-Fernandez, Alan F. Smeaton, Lorin Sweeney

Abstract: This paper describes the MediaEval 2021 Predicting Media Memorability}task, which is in its 4th edition this year, as the prediction of short-term and long-term video memorability remains a challenging task. In 2021, two datasets of videos are used: first, a subset of the TRECVid 2019 Video-to-Text dataset; second, the Memento10K dataset in order to provide opportunities to explore cross-dataset g… ▽ More This paper describes the MediaEval 2021 Predicting Media Memorability}task, which is in its 4th edition this year, as the prediction of short-term and long-term video memorability remains a challenging task. In 2021, two datasets of videos are used: first, a subset of the TRECVid 2019 Video-to-Text dataset; second, the Memento10K dataset in order to provide opportunities to explore cross-dataset generalisation. In addition, an Electroencephalography (EEG)-based prediction pilot subtask is introduced. In this paper, we outline the main aspects of the task and describe the datasets, evaluation metrics, and requirements for participants' submissions. △ Less

Submitted 11 December, 2021; originally announced December 2021.

Comments: 3 pages, to appear in Proceedings of MediaEval 2021, December 13-15 2021, Online

Esophageal virtual disease landscape using mechanics-informed machine learning

Authors: Sourav Halder, Jun Yamasaki, Shashank Acharya, Wenjun Kou, Guy Elisha, Dustin A. Carlson, Peter J. Kahrilas, John E. Pandolfino, Neelesh A. Patankar

Abstract: The pathogenesis of esophageal disorders is related to the esophageal wall mechanics. Therefore, to understand the underlying fundamental mechanisms behind various esophageal disorders, it is crucial to map the esophageal wall mechanics-based parameters onto physiological and pathophysiological conditions corresponding to altered bolus transit and supraphysiologic IBP. In this work, we present a h… ▽ More The pathogenesis of esophageal disorders is related to the esophageal wall mechanics. Therefore, to understand the underlying fundamental mechanisms behind various esophageal disorders, it is crucial to map the esophageal wall mechanics-based parameters onto physiological and pathophysiological conditions corresponding to altered bolus transit and supraphysiologic IBP. In this work, we present a hybrid framework that combines fluid mechanics and machine learning to identify the underlying physics of the various esophageal disorders and maps them onto a parameter space which we call the virtual disease landscape (VDL). A one-dimensional inverse model processes the output from an esophageal diagnostic device called endoscopic functional lumen imaging probe (EndoFLIP) to estimate the mechanical "health" of the esophagus by predicting a set of mechanics-based parameters such as esophageal wall stiffness, muscle contraction pattern and active relaxation of esophageal walls. The mechanics-based parameters were then used to train a neural network that consists of a variational autoencoder (VAE) that generates a latent space and a side network that predicts mechanical work metrics for estimating esophagogastric junction motility. The latent vectors along with a set of discrete mechanics-based parameters define the VDL and form clusters corresponding to the various esophageal disorders. The VDL not only distinguishes different disorders but can also be used to predict disease progression in time. Finally, we also demonstrate the clinical applicability of this framework for estimating the effectiveness of a treatment and track patient condition after a treatment. △ Less

Submitted 18 November, 2021; originally announced November 2021.

Comments: 26 pages, 17 figures

Journal ref: Artificial Intelligence in Medicine. 134 (2022) 102435

Rain rendering for evaluating and improving robustness to bad weather

Authors: Maxime Tremblay, Shirsendu Sukanta Halder, Raoul de Charette, Jean-François Lalonde

Abstract: Rain fills the atmosphere with water particles, which breaks the common assumption that light travels unaltered from the scene to the camera. While it is well-known that rain affects computer vision algorithms, quantifying its impact is difficult. In this context, we present a rain rendering pipeline that enables the systematic evaluation of common computer vision algorithms to controlled amounts… ▽ More Rain fills the atmosphere with water particles, which breaks the common assumption that light travels unaltered from the scene to the camera. While it is well-known that rain affects computer vision algorithms, quantifying its impact is difficult. In this context, we present a rain rendering pipeline that enables the systematic evaluation of common computer vision algorithms to controlled amounts of rain. We present three different ways to add synthetic rain to existing images datasets: completely physic-based; completely data-driven; and a combination of both. The physic-based rain augmentation combines a physical particle simulator and accurate rain photometric modeling. We validate our rendering methods with a user study, demonstrating our rain is judged as much as 73% more realistic than the state-of-theart. Using our generated rain-augmented KITTI, Cityscapes, and nuScenes datasets, we conduct a thorough evaluation of object detection, semantic segmentation, and depth estimation algorithms and show that their performance decreases in degraded weather, on the order of 15% for object detection, 60% for semantic segmentation, and 6-fold increase in depth estimation error. Finetuning on our augmented synthetic data results in improvements of 21% on object detection, 37% on semantic segmentation, and 8% on depth estimation. △ Less

Submitted 6 September, 2020; originally announced September 2020.

Comments: 19 pages, 19 figures, IJCV 2020 preprint. arXiv admin note: text overlap with arXiv:1908.10335

Enhancing Perceptual Loss with Adversarial Feature Matching for Super-Resolution

Authors: Akella Ravi Tej, Shirsendu Sukanta Halder, Arunav Pratap Shandeelya, Vinod Pankajakshan

Abstract: Single image super-resolution (SISR) is an ill-posed problem with an indeterminate number of valid solutions. Solving this problem with neural networks would require access to extensive experience, either presented as a large training set over natural images or a condensed representation from another pre-trained network. Perceptual loss functions, which belong to the latter category, have achieved… ▽ More Single image super-resolution (SISR) is an ill-posed problem with an indeterminate number of valid solutions. Solving this problem with neural networks would require access to extensive experience, either presented as a large training set over natural images or a condensed representation from another pre-trained network. Perceptual loss functions, which belong to the latter category, have achieved breakthrough success in SISR and several other computer vision tasks. While perceptual loss plays a central role in the generation of photo-realistic images, it also produces undesired pattern artifacts in the super-resolved outputs. In this paper, we show that the root cause of these pattern artifacts can be traced back to a mismatch between the pre-training objective of perceptual loss and the super-resolution objective. To address this issue, we propose to augment the existing perceptual loss formulation with a novel content loss function that uses the latent features of a discriminator network to filter the unwanted artifacts across several levels of adversarial similarity. Further, our modification has a stabilizing effect on non-convex optimization in adversarial training. The proposed approach offers notable gains in perceptual quality based on an extensive human evaluation study and a competent reconstruction fidelity when tested on objective evaluation metrics. △ Less

Submitted 15 May, 2020; originally announced May 2020.

Comments: Accepted for publication in the International Joint Conference on Neural Networks (IJCNN) 2020

MA 3 : Model Agnostic Adversarial Augmentation for Few Shot learning

Authors: Rohit Jena, Shirsendu Sukanta Halder, Katia Sycara

Abstract: Despite the recent developments in vision-related problems using deep neural networks, there still remains a wide scope in the improvement of generalizing these models to unseen examples. In this paper, we explore the domain of few-shot learning with a novel augmentation technique. In contrast to other generative augmentation techniques, where the distribution over input images are learnt, we prop… ▽ More Despite the recent developments in vision-related problems using deep neural networks, there still remains a wide scope in the improvement of generalizing these models to unseen examples. In this paper, we explore the domain of few-shot learning with a novel augmentation technique. In contrast to other generative augmentation techniques, where the distribution over input images are learnt, we propose to learn the probability distribution over the image transformation parameters which are easier and quicker to learn. Our technique is fully differentiable which enables its extension to versatile data-sets and base models. We evaluate our proposed method on multiple base-networks and 2 data-sets to establish the robustness and efficiency of this method. We obtain an improvement of nearly 4% by adding our augmentation module without making any change in network architectures. We also make the code readily available for usage by the community. △ Less

Submitted 10 April, 2020; originally announced April 2020.

Comments: Accepted at CVPR Workshop on Visual Learning with Limited Labels 2020

Physics-Based Rendering for Improving Robustness to Rain

Authors: Shirsendu Sukanta Halder, Jean-François Lalonde, Raoul de Charette

Abstract: To improve the robustness to rain, we present a physically-based rain rendering pipeline for realistically inserting rain into clear weather images. Our rendering relies on a physical particle simulator, an estimation of the scene lighting and an accurate rain photometric modeling to augment images with arbitrary amount of realistic rain or fog. We validate our rendering with a user study, proving… ▽ More To improve the robustness to rain, we present a physically-based rain rendering pipeline for realistically inserting rain into clear weather images. Our rendering relies on a physical particle simulator, an estimation of the scene lighting and an accurate rain photometric modeling to augment images with arbitrary amount of realistic rain or fog. We validate our rendering with a user study, proving our rain is judged 40% more realistic that state-of-the-art. Using our generated weather augmented Kitti and Cityscapes dataset, we conduct a thorough evaluation of deep object detection and semantic segmentation algorithms and show that their performance decreases in degraded weather, on the order of 15% for object detection and 60% for semantic segmentation. Furthermore, we show refining existing networks with our augmented images improves the robustness of both object detection and semantic segmentation algorithms. We experiment on nuScenes and measure an improvement of 15% for object detection and 35% for semantic segmentation compared to original rainy performance. Augmented databases and code are available on the project page. △ Less

Submitted 27 August, 2019; originally announced August 2019.

Comments: ICCV 2019. Supplementary pdf / videos available on project page

Secure OTA Software Updates in Connected Vehicles: A survey

Authors: Subir Halder, Amrita Ghosal, Mauro Conti

Abstract: This survey highlights and discusses remote OTA software updates in the automotive sector, mainly from the security perspective. In particular, the major objective of this survey is to provide a comprehensive and structured outline of various research directions and approaches in OTA update technologies in vehicles. At first, we discuss the connected car technology and then integrate the relations… ▽ More This survey highlights and discusses remote OTA software updates in the automotive sector, mainly from the security perspective. In particular, the major objective of this survey is to provide a comprehensive and structured outline of various research directions and approaches in OTA update technologies in vehicles. At first, we discuss the connected car technology and then integrate the relationship of remote OTA update features with the connected car. We also present the benefits of remote OTA updates for cars along with relevant statistics. Then, we emphasize on the security challenges and requirements of remote OTA updates along with use cases and standard road safety regulations followed in different countries. We also provide for a classification of the existing works in literature that deal with implementing different secured techniques for remote OTA updates in vehicles. We further provide an analytical discussion on the present scenario of remote OTA updates with respect to care manufacturers. Finally, we identify possible future research directions of remote OTA updates for automobiles, particularly in the area of security. △ Less

Submitted 1 April, 2019; originally announced April 2019.

Comments: 18 pages

Movie Recommendation System using Sentiment Analysis from Microblogging Data

Authors: Sudhanshu Kumar, Shirsendu Sukanta Halder, Kanjar De, Partha Pratim Roy

Abstract: Recommendation systems are important intelligent systems that play a vital role in providing selective information to users. Traditional approaches in recommendation systems include collaborative filtering and content-based filtering. However, these approaches have certain limitations like the necessity of prior user history and habits for performing the task of recommendation. In order to reduce… ▽ More Recommendation systems are important intelligent systems that play a vital role in providing selective information to users. Traditional approaches in recommendation systems include collaborative filtering and content-based filtering. However, these approaches have certain limitations like the necessity of prior user history and habits for performing the task of recommendation. In order to reduce the effect of such dependencies, this paper proposes a hybrid recommendation system which combines the collaborative filtering, content-based filtering with sentiment analysis of movie tweets. The movie tweets have been collected from microblogging websites to understand the current trends and user response of the movie. Experiments conducted on public database produce promising results. △ Less

Submitted 26 November, 2018; originally announced November 2018.

Comments: 19 pages, 7 tables, 5 figures

Perceptual Conditional Generative Adversarial Networks for End-to-End Image Colourization

Authors: Shirsendu Sukanta Halder, Kanjar De, Partha Pratim Roy

Abstract: Colours are everywhere. They embody a significant part of human visual perception. In this paper, we explore the paradigm of hallucinating colours from a given gray-scale image. The problem of colourization has been dealt in previous literature but mostly in a supervised manner involving user-interference. With the emergence of Deep Learning methods numerous tasks related to computer vision and pa… ▽ More Colours are everywhere. They embody a significant part of human visual perception. In this paper, we explore the paradigm of hallucinating colours from a given gray-scale image. The problem of colourization has been dealt in previous literature but mostly in a supervised manner involving user-interference. With the emergence of Deep Learning methods numerous tasks related to computer vision and pattern recognition have been automatized and carried in an end-to-end fashion due to the availability of large data-sets and high-power computing systems. We investigate and build upon the recent success of Conditional Generative Adversarial Networks (cGANs) for Image-to-Image translations. In addition to using the training scheme in the basic cGAN, we propose an encoder-decoder generator network which utilizes the class-specific cross-entropy loss as well as the perceptual loss in addition to the original objective function of cGAN. We train our model on a large-scale dataset and present illustrative qualitative and quantitative analysis of our results. Our results vividly display the versatility and proficiency of our methods through life-like colourization outcomes. △ Less

Submitted 26 November, 2018; originally announced November 2018.

Comments: 16 pages, 8 figures, 3 tables

Reconstruction Loss Minimized FCN for Single Image Dehazing

Authors: Shirsendu Sukanta Halder, Sanchayan Santra, Bhabatosh Chanda

Abstract: Haze and fog reduce the visibility of outdoor scenes as a veil like semi-transparent layer appears over the objects. As a result, images captured under such conditions lack contrast. Image dehazing methods try to alleviate this problem by recovering a clear version of the image. In this paper, we propose a Fully Convolutional Neural Network based model to recover the clear scene radiance by estima… ▽ More Haze and fog reduce the visibility of outdoor scenes as a veil like semi-transparent layer appears over the objects. As a result, images captured under such conditions lack contrast. Image dehazing methods try to alleviate this problem by recovering a clear version of the image. In this paper, we propose a Fully Convolutional Neural Network based model to recover the clear scene radiance by estimating the environmental illumination and the scene transmittance jointly from a hazy image. The method uses a relaxed haze imaging model to allow for the situations with non-uniform illumination. We have trained the network by minimizing a custom-defined loss that measures the error of reconstructing the hazy image in three different ways. Additionally, we use a multilevel approach to determine the scene transmittance and the environmental illumination in order to reduce the dependence of the estimate on image scale. Evaluations show that our model performs well compared to the existing state-of-the-art methods. It also verifies the potential of our model in diverse situations and various lighting conditions. △ Less

Submitted 26 November, 2018; originally announced November 2018.

Comments: 12 pages, 9 figures, 3 tables

Face Synthesis (FASY) System for Determining the Characteristics of a Face Image

Authors: Santanu Halder, Debotosh Bhattacharjee, Mita Nasipuri, Dipak Kumar Basu, Mahantapas Kundu

Abstract: This paper aims at determining the characteristics of a face image by extracting its components. The FASY (FAce SYnthesis) System is a Face Database Retrieval and new Face generation System that is under development. One of its main features is the generation of the requested face when it is not found in the existing database, which allows a continuous growing of the database also. To generate the… ▽ More This paper aims at determining the characteristics of a face image by extracting its components. The FASY (FAce SYnthesis) System is a Face Database Retrieval and new Face generation System that is under development. One of its main features is the generation of the requested face when it is not found in the existing database, which allows a continuous growing of the database also. To generate the new face image, we need to store the face components in the database. So we have designed a new technique to extract the face components by a sophisticated method. After extraction of the facial feature points we have analyzed the components to determine their characteristics. After extraction and analysis we have stored the components along with their characteristics into the face database for later use during the face construction. △ Less

Submitted 5 July, 2010; originally announced July 2010.

Journal ref: RAIT 2009

Fuzzy Classification of Facial Component Parameters

Authors: S. Halder, Debotosh Bhattacharjee, M. Nasipuri, D. K. Basu, M. Kundu

Abstract: This paper presents a novel type-2 Fuzzy logic System to define the Shape of a facial component with the crisp output. This work is the part of our main research effort to design a system (called FASY) which offers a novel face construction approach based on the textual description and also extracts and analyzes the facial components from a face image by an efficient technique. The Fuzzy model, de… ▽ More This paper presents a novel type-2 Fuzzy logic System to define the Shape of a facial component with the crisp output. This work is the part of our main research effort to design a system (called FASY) which offers a novel face construction approach based on the textual description and also extracts and analyzes the facial components from a face image by an efficient technique. The Fuzzy model, designed in this paper, takes crisp value of width and height of a facial component and produces the crisp value of Shape for different facial components. This method is designed using Matlab 6.5 and Visual Basic 6.0 and tested with the facial components extracted from 200 male and female face images of different ages from different face databases. △ Less

Submitted 5 July, 2010; v1 submitted 30 June, 2010; originally announced June 2010.

Journal ref: IJRTE 2(2): 66-70(2009)

FPGA Based Assembling of Facial Components for Human Face Construction

Authors: Santanu Halder, Debotosh Bhattacharjee, Mita Nasipuri, Dipak Kumar Basu, Mahantapas Kundu

Abstract: This paper aims at VLSI realization for generation of a new face from textual description. The FASY (FAce SYnthesis) System is a Face Database Retrieval and new Face generation System that is under development. One of its main features is the generation of the requested face when it is not found in the existing database. The new face generation system works in three steps - searching phase, assemb… ▽ More This paper aims at VLSI realization for generation of a new face from textual description. The FASY (FAce SYnthesis) System is a Face Database Retrieval and new Face generation System that is under development. One of its main features is the generation of the requested face when it is not found in the existing database. The new face generation system works in three steps - searching phase, assembling phase and tuning phase. In this paper the tuning phase using hardware description language and its implementation in a Field Programmable Gate Array (FPGA) device is presented. △ Less

Submitted 30 June, 2010; originally announced June 2010.

Journal ref: IJRTE 1(1):541-545(2009)

Face Synthesis (FASY) System for Generation of a Face Image from Human Description

Authors: Santanu Halder, Debotosh Bhattacharjee, Mita Nasipuri, Dipak Kumar Basu, Mahantapas Kundu

Abstract: This paper aims at generating a new face based on the human like description using a new concept. The FASY (FAce SYnthesis) System is a Face Database Retrieval and new Face generation System that is under development. One of its main features is the generation of the requested face when it is not found in the existing database, which allows a continuous growing of the database also. This paper aims at generating a new face based on the human like description using a new concept. The FASY (FAce SYnthesis) System is a Face Database Retrieval and new Face generation System that is under development. One of its main features is the generation of the requested face when it is not found in the existing database, which allows a continuous growing of the database also. △ Less

Submitted 21 May, 2010; originally announced May 2010.

Journal ref: ICIIS 2008

A Multi-agent Framework for Performance Tuning in Distributed Environment

Authors: Sarbani Roy, Saikat Halder, Nandini Mukherjee

Abstract: This paper presents the overall design of a multi-agent framework for tuning the performance of an application executing in a distributed environment. The multi-agent framework provides services like resource brokering, analyzing performance monitoring data, local tuning and also rescheduling in case of any performance problem on a specific resource provider. The paper also briefly describes the i… ▽ More This paper presents the overall design of a multi-agent framework for tuning the performance of an application executing in a distributed environment. The multi-agent framework provides services like resource brokering, analyzing performance monitoring data, local tuning and also rescheduling in case of any performance problem on a specific resource provider. The paper also briefly describes the implementation of some part of the framework. In particular, job migration on the basis of performance monitoring data is particularly highlighted in this paper. △ Less

Submitted 12 May, 2010; originally announced May 2010.

Comments: International Conference On High Performance Computing HiPC 2005, Posters Web Proceedings, Goa, India, December 18-21, 2005