"# Part 22, ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction (Part 2 of ChemBERTa series)\n",
"# Part 23, ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction (Part 2 of ChemBERTa series)\n",
"\n",
"## Using the Smiles-Tokenizer class in DeepChem with ChemBERTa for attention visualization and fine-tuning.\n",
"\n",
@@ -4947,13 +4947,9 @@
"\n",
"Training RoBERTa over 10 epochs, the model achieves a pretty good loss of 0.198, and may likely continue to converge if trained for a larger number of epochs. The model can predict masked/corrupted tokens within a SMILES sequence/molecule, allowing for variants of a molecule within discoverable chemical space to be predicted.\n",
"\n",
"\n",
"BPE is a hybrid between character and word-level representations, which allows for the handling of large vocabularies in natural language corpora. Motivated by the intuition that rare and unknown words can often be decomposed into multiple known subwords, BPE finds the best word segmentation by iteratively and greedily merging frequent pairs of characters. In our work, we compared this\n",
"tokenization algorithm with a **custom SmilesTokenizer** based on a regex pattern, which we have released as part of DeepChem. To compare tokenizers, we pretrained an identical model tokenized using this novel tokenizer, on the PubChem-1M set. The pretrained model was evaluated on the Tox21 SR-p53 task in the paper. We found that the SmilesTokenizer narrowly outperformed the BPE algorithm by ∆PRC-AUC = +0.015. Though this result suggests that a more semantically relevant tokenization may provide performance benefits, further benchmarking on additional datasets is needed to validate this finding. **In this tutorial, we aim to do so, by testing this alternate model on the ClinTox dataset, attention visualization tasks, and maskesd token inference.**\n",
"\n",
"By applying the representations of functional groups and atoms learned by the model, we can try to tackle problems of toxicity, solubility, drug-likeness, and synthesis accessibility on smaller datasets using the learned representations as features for graph convolution and attention models on the graph structure of molecules, as well as fine-tuning of BERT. Finally, we propose the use of attention visualization as a helpful tool for chemistry practitioners and students to quickly identify important substructures in various chemical properties.\n",
"\n",
"Additionally, visualization of the attention mechanism have been seen through previous research as incredibly valuable towards chemical reaction classification. The applications of open-sourcing large-scale transformer models such as RoBERTa with HuggingFace may allow for the acceleration of these individual research directions.\n",
"Visualization of the attention mechanism have been seen through previous research as incredibly valuable towards chemical reaction classification. The applications of open-sourcing large-scale transformer models such as RoBERTa with HuggingFace may allow for the acceleration of these individual research directions.\n",
"\n",
"A link to a repository which includes the training, uploading and evaluation notebook (with sample predictions on compounds such as Remdesivir) can be found [here](https://github.com/seyonechithrananda/bert-loves-chemistry). All of the notebooks can be copied into a new Colab runtime for easy execution. This repository will be updated with new features, such as attention visualization, easier benchmarking infrastructure, and more. The work behind this tutorial has been published on [Arxiv](https://arxiv.org/abs/2010.09885), and has been accepted for a highlight presentation at NeurIPS 2020's ML for Molecules Workshop, where we will highlight updates to this work.\n",
"\n",
@@ -5053,7 +5049,10 @@
"id": "GOAEt4gsTZ5u"
},
"source": [
"We want to install NVIDIA's Apex tool, for the training pipeline used by `simple-transformers` and Weights and Biases. In order to support SmilesTokenizer, we have to install a fork of the `simple-transformers` library"
"We want to install NVIDIA's Apex tool, for the training pipeline used by `simple-transformers` and Weights and Biases. This package enables us to use 16-bit training, mixed precision, and distributed training without any changes to our code. Generally GPUs are good at doing 32-bit(single precision) math, not at 16-bit(half) nor 64-bit(double precision). Therefore traditionally deep learning model trainings are done in 32-bit. By switching to 16-bit, we’ll be using half the memory and theoretically less computation at the expense of the available number range and precision. However, pure 16-bit training creates a lot of problems for us (imprecise weight updates, gradient underflow and overflow). **Mixed precision training, with Apex, alleviates these problems**.\n",
"\n",
"\n",
"In order to support SmilesTokenizer, we have to install a fork of the `simple-transformers` library"
]
},
{
@@ -5153,15 +5152,6 @@
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uSuLMmOSW531"
},
"source": [
"Now, to ensure our the SmilesTokenizer-ChemBERTa model demonstrates an understanding of chemical syntax and molecular structure, we'll be testing it on predicting a masked token/character within the SMILES molecule for benzene."
]
},
{
"cell_type": "code",
"metadata": {
@@ -5174,6 +5164,30 @@
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "IEOw9Dzm8mfU"
},
"source": [
"## Why use Smiles-Tokenizer\n",
"\n",
"A tokenizer is in charge of preparing the inputs for a natural language processing model. For many scientific applications, it is possible to treat inputs as “words”/”sentences” and use NLP methods to make meaningful predictions. For example, SMILES strings or DNA sequences have grammatical structure and can be usefully modeled with NLP techniques. DeepChem provides some scientifically relevant tokenizers for use in different applications. These tokenizers are based on those from the Huggingface transformers library (which DeepChem tokenizers inherit from).\n",
"\n",
"The base classes PreTrainedTokenizer and PreTrainedTokenizerFast in HuggingFace implements the common methods for encoding string inputs in model inputs and instantiating/saving python tokenizers either from a local file or directory or from a pretrained tokenizer provided by the library (downloaded from HuggingFace’s AWS S3 repository).\n",
"\n",
"\n",
"PreTrainedTokenizer [(transformers.PreTrainedTokenizer)](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer)) thus implements the main methods for using all the tokenizers:\n",
"* Tokenizing (spliting strings in sub-word token strings), converting tokens strings to ids and back, and encoding/decoding (i.e. tokenizing + convert to integers),\n",
"\n",
"* Adding new tokens to the vocabulary in a way that is independant of the underlying structure (BPE, SentencePiece…),\n",
"\n",
"* Managing special tokens like mask, beginning-of-sentence, etc tokens (adding them, assigning them to attributes in the tokenizer for easy access and making sure they are not split during tokenization)\n",
"\n",
"\n",
"The default tokenizer used by ChemBERTa, is a Byte-Pair-Encoder (BPE). It is a hybrid between character and word-level representations, which allows for the handling of large vocabularies in natural language corpora. Motivated by the intuition that rare and unknown words can often be decomposed into multiple known subwords, BPE finds the best word segmentation by iteratively and greedily merging frequent pairs of characters. In this tutorial, we compared this tokenization algorithm with a **custom SmilesTokenizer** based on a regex pattern, which we have released as part of DeepChem. To compare tokenizers, we pretrained an identical model tokenized using this novel tokenizer, on the PubChem-1M set. The pretrained model was evaluated on the BBBP and Tox21 in the paper. We found that the SmilesTokenizer narrowly outperformed the BPE algorithm by ∆PRC-AUC = $+0.021$. Though this result suggests that a more semantically relevant tokenization may provide performance benefits, further benchmarking on additional datasets is needed to validate this finding. **In this tutorial, we aim to do so, by testing this alternate model on the ClinTox dataset, attention visualization tasks, and maskesd token inference.**\n"
]
},
{
"cell_type": "markdown",
"metadata": {
@@ -5239,6 +5253,15 @@
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uSuLMmOSW531"
},
"source": [
"Now, to ensure our the SmilesTokenizer-ChemBERTa model demonstrates an understanding of chemical syntax and molecular structure, we'll be testing it on predicting a masked token/character within the SMILES molecule for benzene."
]
},
{
"cell_type": "code",
"metadata": {
@@ -5310,7 +5333,7 @@
"id": "0XVpUyijW676"
},
"source": [
"Here, we get some interesting results. The final branch, `C1=CC=CC=C1`, is a benzene ring. Since its a pretty common molecule, the model is easily able to predict the final double carbon bond with a score of 0.98. Let's get a list of the top 5 predictions (including the target, Remdesivir), and visualize them (with a highlighted focus on the beginning of the final benzene-like pattern).\n"
"Here, we get some interesting results. The final branch, `C1=CC=CC=C1`, is a benzene ring. Since its a pretty common molecule, the model is easily able to predict the final double carbon bond with a score of 0.98. Let's get a list of the top 5 predictions (including the target, Remdesivir), and visualize them (with a highlighted focus on the beginning of the final benzene-like pattern). To visualize them, we'll be using the RDKit cheminoformatics package we installed earlier, specifically the `rdkit.chem.Draw` module.\n"
]
},
{
@@ -6210,7 +6233,7 @@
"id": "U3MMEtKrRXaO"
},
"source": [
"Let's start by importing the MolNet dataloder from `bert-loves-chemistry`, before importing apex and transformers, the tool which will allow us to import the pre-trained masked-language modelling architecture trained on PubChem."
"Let's start by importing the MolNet dataloder from `bert-loves-chemistry`, before importing apex and transformers, the tool which will allow us to import the ChemBERTA language model (LM) trained on PubChem-1M."
"model = ClassificationModel('roberta', 'seyonec/SMILES_tokenized_PubChem_shard00_160k', args={'evaluate_each_epoch': True, 'evaluate_during_training_verbose': True, 'no_save': True, 'num_train_epochs': 20, 'auto_weights': True}) # You can set class weights by using the optional weight argument\n"
"model = ClassificationModel('roberta', 'seyonec/SMILES_tokenized_PubChem_shard00_160k', args={'evaluate_each_epoch': True, 'evaluate_during_training_verbose': True, 'no_save': True, 'num_train_epochs': 15, 'auto_weights': True}) # You can set class weights by using the optional weight argument\n"
],
"execution_count": null,
"outputs": [
@@ -8176,7 +8199,7 @@
"id": "dD2FlxhWUqvo"
},
"source": [
"The model performs pretty well, averaging above 96% ROC-PRC after training on only ~1400 data samples and 150 positive leads in a couple of minutes! This model was also trained on 1/10th the amount of pre-training data as the PubChem-10M BPE model we used previously, but it still showcases robust performance. We can clearly see the predictive power of transfer learning, and approaches like these are becoming increasing popular in the pharmaceutical industry where larger datasets are scarce. By training on more epochs and tasks, we can probably boost the accuracy as well!\n",
"The model performs pretty well, averaging above 96% PRC-AUC after training on only ~1400 data samples and 150 positive leads in a couple of minutes! This model was also trained on 1/10th the amount of pre-training data as the PubChem-10M BPE model we used previously, but it still showcases robust performance. We can clearly see the predictive power of transfer learning, and approaches like these are becoming increasing popular in the pharmaceutical industry where larger datasets are scarce. By training on more epochs and tasks, we can probably boost the accuracy as well!\n",
"\n",
"Lets evaluate the model on one last string from ClinTox's test set for toxicity. The model should predict 1, meaning the drug failed clinical trials for toxicity reasons and wasn't approved by the FDA."
