-
Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency
Authors:
Sakib Shahriar,
Brady Lund,
Nishith Reddy Mannuru,
Muhammad Arbab Arshad,
Kadhim Hayawi,
Ravi Varma Kumar Bevara,
Aashrith Mannuru,
Laiba Batool
Abstract:
As large language models (LLMs) continue to advance, evaluating their comprehensive capabilities becomes significant for their application in various fields. This research study comprehensively evaluates the language, vision, speech, and multimodal capabilities of GPT-4o. The study employs standardized exam questions, reasoning tasks, and translation assessments to assess the model's language capa…
▽ More
As large language models (LLMs) continue to advance, evaluating their comprehensive capabilities becomes significant for their application in various fields. This research study comprehensively evaluates the language, vision, speech, and multimodal capabilities of GPT-4o. The study employs standardized exam questions, reasoning tasks, and translation assessments to assess the model's language capability. Additionally, GPT-4o's vision and speech capabilities are tested through image classification and object recognition tasks, as well as accent classification. The multimodal evaluation assesses the model's performance in integrating visual and linguistic data. Our findings reveal that GPT-4o demonstrates high accuracy and efficiency across multiple domains in language and reasoning capabilities, excelling in tasks that require few-shot learning. GPT-4o also provides notable improvements in multimodal tasks compared to its predecessors. However, the model shows variability and faces limitations in handling complex and ambiguous inputs, particularly in audio and vision capabilities. This paper highlights the need for more comprehensive benchmarks and robust evaluation frameworks, encompassing qualitative assessments involving human judgment as well as error analysis. Future work should focus on expanding datasets, investigating prompt-based assessment, and enhancing few-shot learning techniques to test the model's practical applicability and performance in real-world scenarios.
△ Less
Submitted 19 June, 2024;
originally announced July 2024.
-
Inevitable-Metaverse: A Novel Twitter Dataset for Public Sentiments on Metaverse
Authors:
Kadhim Hayawi,
Sakib Shahriar,
Mohamed Adel Serhani,
Eiman Alothali
Abstract:
Metaverse has emerged as a novel technology with the objective to merge the physical world into the virtual world. This technology has seen a lot of interest and investment in recent times from prominent organizations including Facebook which has changed its company name to Meta with the goal of being the leader in developing this technology. Although people in general are excited about the prospe…
▽ More
Metaverse has emerged as a novel technology with the objective to merge the physical world into the virtual world. This technology has seen a lot of interest and investment in recent times from prominent organizations including Facebook which has changed its company name to Meta with the goal of being the leader in developing this technology. Although people in general are excited about the prospects of metaverse due to potential use cases such as virtual meetings and virtual learning environments, there are also concerns due to potential negative consequences. For instance, people are concerned about their data privacy as well as spending a lot of their time on the metaverse leading to negative impacts in real life. Therefore, this research aims to further investigate the public sentiments regarding metaverse on social media. A total of 86565 metaverse-related tweets were used to perform lexicon-based sentiment analysis. Furthermore, various machine and deep learning models with various text features were utilized to predict the sentiment class. The BERT transformer model was demonstrated to be the best at predicting the sentiment categories with 92.6% accuracy and 0.91 F-measure on the test dataset. Finally, the implications and future research directions were also discussed.
△ Less
Submitted 2 March, 2024;
originally announced March 2024.
-
The Imitation Game: Detecting Human and AI-Generated Texts in the Era of ChatGPT and BARD
Authors:
Kadhim Hayawi,
Sakib Shahriar,
Sujith Samuel Mathew
Abstract:
The potential of artificial intelligence (AI)-based large language models (LLMs) holds considerable promise in revolutionizing education, research, and practice. However, distinguishing between human-written and AI-generated text has become a significant task. This paper presents a comparative study, introducing a novel dataset of human-written and LLM-generated texts in different genres: essays,…
▽ More
The potential of artificial intelligence (AI)-based large language models (LLMs) holds considerable promise in revolutionizing education, research, and practice. However, distinguishing between human-written and AI-generated text has become a significant task. This paper presents a comparative study, introducing a novel dataset of human-written and LLM-generated texts in different genres: essays, stories, poetry, and Python code. We employ several machine learning models to classify the texts. Results demonstrate the efficacy of these models in discerning between human and AI-generated text, despite the dataset's limited sample size. However, the task becomes more challenging when classifying GPT-generated text, particularly in story writing. The results indicate that the models exhibit superior performance in binary classification tasks, such as distinguishing human-generated text from a specific LLM, compared to the more complex multiclass tasks that involve discerning among human-generated and multiple LLMs. Our findings provide insightful implications for AI text detection while our dataset paves the way for future research in this evolving area.
△ Less
Submitted 11 November, 2023; v1 submitted 22 July, 2023;
originally announced July 2023.
-
Let's have a chat! A Conversation with ChatGPT: Technology, Applications, and Limitations
Authors:
Sakib Shahriar,
Kadhim Hayawi
Abstract:
The emergence of an AI-powered chatbot that can generate human-like sentences and write coherent essays has caught the world's attention. This paper discusses the historical overview of chatbots and the technology behind Chat Generative Pre-trained Transformer, better known as ChatGPT. Moreover, potential applications of ChatGPT in various domains, including healthcare, education, and research, ar…
▽ More
The emergence of an AI-powered chatbot that can generate human-like sentences and write coherent essays has caught the world's attention. This paper discusses the historical overview of chatbots and the technology behind Chat Generative Pre-trained Transformer, better known as ChatGPT. Moreover, potential applications of ChatGPT in various domains, including healthcare, education, and research, are highlighted. Despite promising results, there are several privacy and ethical concerns surrounding ChatGPT. In addition, we highlight some of the important limitations of the current version of ChatGPT. We also ask ChatGPT to provide its point of view and present its responses to several questions we attempt to answer.
△ Less
Submitted 9 August, 2023; v1 submitted 27 February, 2023;
originally announced February 2023.
-
NFTGAN: Non-Fungible Token Art Generation Using Generative Adversarial Networks
Authors:
Sakib Shahriar,
Kadhim Hayawi
Abstract:
Digital arts have gained an unprecedented level of popularity with the emergence of non-fungible tokens (NFTs). NFTs are cryptographic assets that are stored on blockchain networks and represent a digital certificate of ownership that cannot be forged. NFTs can be incorporated into a smart contract which allows the owner to benefit from a future sale percentage. While digital art producers can ben…
▽ More
Digital arts have gained an unprecedented level of popularity with the emergence of non-fungible tokens (NFTs). NFTs are cryptographic assets that are stored on blockchain networks and represent a digital certificate of ownership that cannot be forged. NFTs can be incorporated into a smart contract which allows the owner to benefit from a future sale percentage. While digital art producers can benefit immensely with NFTs, their production is time consuming. Therefore, this paper explores the possibility of using generative adversarial networks (GANs) for automatic generation of digital arts. GANs are deep learning architectures that are widely and effectively used for synthesis of audio, images, and video contents. However, their application to NFT arts have been limited. In this paper, a GAN-based architecture is implemented and evaluated for novel NFT-style digital arts generation. Results from the qualitative case study indicate that the generated artworks are comparable to the real samples in terms of being interesting and inspiring and they were judged to be more innovative than real samples.
△ Less
Submitted 27 December, 2021; v1 submitted 17 December, 2021;
originally announced December 2021.
