Accelerate SARS-CoV-2 research with transfer learning using pre-trained language modeling model

Many researchers are trying right now to decipher the molecular mechanisms of SARS-CoV-2, in order to find a vaccine or a more differentiated, personalized treatment. For these steps to be successful as well as time-efficient, it is crucial to know the 3-dimensional shape of the virus’ proteins, e.g. to perform docking simulations with small molecules. While there are experimentally determined structures available for roughly half of the proteins in SARS-CoV-2, computational (prediction) methods are needed to gain insights about the other half. However, the quality of all current protein structure predictors (PSP) relies on finding an sufficient amount of evolutionary related proteins in today’s databases, i.e. gathering evolutionary information (EV). For SARS-CoV-2, the open reading frame 3a (ORF3a) and the non structural protein 2 (nsp2) provide only little evolutionary information, causing current structure predictions for those proteins to be too coarse-grained for functional analysis.\n \n Here, we propose a novel approach that allows us to predict protein structures based only on single protein sequences, overcoming the dependency on evolutionary information. Our approach builds up on and extends preliminary results which were obtained from training Language Models (LMs) as part of a a Director’s Discretion project (DD) on one of ORNL’s supercomputers, i.e. Summit. These initial results showed that these LM methods can provide very close accuracy to evolutionary information methods, while it is 100x times faster during inference phase using only 1 GPU with 8 GB memory. LMs use a specific neural network architecture to learn general information from large unlabelled data (here: protein sequence databases), in an automated, data-driven way. After this (pre-) training, the neural network can be used to extract features as numerical vectors (=embeddings) from single protein sequences which can then be used as an input for any machine learning device that makes predictions about aspects of proteins. In this work, we focus on the specific use-case of predicting protein structures based on a) our new embeddings as well as b) a combination of existing input (evolutionary information) combined with embeddings. First experiments using this novel workflow were already conducted as part of a structure prediction effort focusing on predicting SARS-CoV-2 protein structures organized by the Critical Assessment of Structure Predictions (CASP). Results obtained from this competition already showed promising results, especially for cases with little evolutionary information available, i.e. ORF3a and nsp2. \n \n Building up on the experience which we gained from the DD grant, a pool of LMs was carefully selected in order to learn more informative embeddings, i.e. Bert, Albert, XLNet, Electra, DistilBERT and T5. While the first (pre-) training phase of LMs is computationally demanding, inference is computationally light-weight which makes it easy to distribute the pre-trained models to other researchers who can apply it to their tasks using only consumer hardware. Additionally, the LMs are general feature extractors for proteins, making them also useful after a vaccine or treatment for SARS-CoV-2 was found. In a second phase, which happens in parallel rather sequentially because we can build up on our existing LMs, we will train a neural network to predict protein structures based on embeddings.