-
MSAGPT: Neural Prompting Protein Structure Prediction via MSA Generative Pre-Training
Authors:
Bo Chen,
Zhilei Bei,
Xingyi Cheng,
Pan Li,
Jie Tang,
Le Song
Abstract:
Multiple Sequence Alignment (MSA) plays a pivotal role in unveiling the evolutionary trajectories of protein families. The accuracy of protein structure predictions is often compromised for protein sequences that lack sufficient homologous information to construct high quality MSA. Although various methods have been proposed to generate virtual MSA under these conditions, they fall short in compre…
▽ More
Multiple Sequence Alignment (MSA) plays a pivotal role in unveiling the evolutionary trajectories of protein families. The accuracy of protein structure predictions is often compromised for protein sequences that lack sufficient homologous information to construct high quality MSA. Although various methods have been proposed to generate virtual MSA under these conditions, they fall short in comprehensively capturing the intricate coevolutionary patterns within MSA or require guidance from external oracle models. Here we introduce MSAGPT, a novel approach to prompt protein structure predictions via MSA generative pretraining in the low MSA regime. MSAGPT employs a simple yet effective 2D evolutionary positional encoding scheme to model complex evolutionary patterns. Endowed by this, its flexible 1D MSA decoding framework facilitates zero or few shot learning. Moreover, we demonstrate that leveraging the feedback from AlphaFold2 can further enhance the model capacity via Rejective Fine tuning (RFT) and Reinforcement Learning from AF2 Feedback (RLAF). Extensive experiments confirm the efficacy of MSAGPT in generating faithful virtual MSA to enhance the structure prediction accuracy. The transfer learning capabilities also highlight its great potential for facilitating other protein tasks.
△ Less
Submitted 10 June, 2024; v1 submitted 8 June, 2024;
originally announced June 2024.
-
xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein
Authors:
Bo Chen,
Xingyi Cheng,
Pan Li,
Yangli-ao Geng,
Jing Gong,
Shen Li,
Zhilei Bei,
Xu Tan,
Boyan Wang,
Xin Zeng,
Chiming Liu,
Aohan Zeng,
Yuxiao Dong,
Jie Tang,
Le Song
Abstract:
Protein language models have shown remarkable success in learning biological information from protein sequences. However, most existing models are limited by either autoencoding or autoregressive pre-training objectives, which makes them struggle to handle protein understanding and generation tasks concurrently. We propose a unified protein language model, xTrimoPGLM, to address these two types of…
▽ More
Protein language models have shown remarkable success in learning biological information from protein sequences. However, most existing models are limited by either autoencoding or autoregressive pre-training objectives, which makes them struggle to handle protein understanding and generation tasks concurrently. We propose a unified protein language model, xTrimoPGLM, to address these two types of tasks simultaneously through an innovative pre-training framework. Our key technical contribution is an exploration of the compatibility and the potential for joint optimization of the two types of objectives, which has led to a strategy for training xTrimoPGLM at an unprecedented scale of 100 billion parameters and 1 trillion training tokens. Our extensive experiments reveal that 1) xTrimoPGLM significantly outperforms other advanced baselines in 18 protein understanding benchmarks across four categories. The model also facilitates an atomic-resolution view of protein structures, leading to an advanced 3D structural prediction model that surpasses existing language model-based tools. 2) xTrimoPGLM not only can generate de novo protein sequences following the principles of natural ones, but also can perform programmable generation after supervised fine-tuning (SFT) on curated sequences. These results highlight the substantial capability and versatility of xTrimoPGLM in understanding and generating protein sequences, contributing to the evolving landscape of foundation models in protein science.
△ Less
Submitted 11 January, 2024;
originally announced January 2024.
-
Approximate Discovery of Service Nodes by Duplicate Detection in Flows
Authors:
Zhou Changling,
Xiao Jianguo,
Cui Jian,
Zhang Bei,
Li Feng
Abstract:
Knowledge about which nodes provide services is of critical importance for network administrators. Discovery of service nodes can be done by making full use of duplicate element detection in flows. Because the amount of traffic across network is massive, especially in large ISPs or campus networks, we propose an approximate algorithm with Round-robin Buddy Bloom Filters(RBBF) for service detection…
▽ More
Knowledge about which nodes provide services is of critical importance for network administrators. Discovery of service nodes can be done by making full use of duplicate element detection in flows. Because the amount of traffic across network is massive, especially in large ISPs or campus networks, we propose an approximate algorithm with Round-robin Buddy Bloom Filters(RBBF) for service detection using NetFlow data solely. The properties and analysis of RBBF data structure are also given. Our method has better time/space efficiency than conventional algorithm with a small false positive rate.%portion of false positive. We also demonstrate the contributions through a prototype system by real world case studies.
△ Less
Submitted 14 March, 2015;
originally announced March 2015.
