"We want to install NVIDIA's Apex tool, for the training pipeline used by `simple-transformers` and Weights and Biases. This package enables us to use 16-bit training, mixed precision, and distributed training without any changes to our code. Generally GPUs are good at doing 32-bit(single precision) math, not at 16-bit(half) nor 64-bit(double precision). Therefore traditionally deep learning model trainings are done in 32-bit. By switching to 16-bit, we’ll be using half the memory and theoretically less computation at the expense of the available number range and precision. However, pure 16-bit training creates a lot of problems for us (imprecise weight updates, gradient underflow and overflow). **Mixed precision training, with Apex, alleviates these problems**.\n",
"\n",
"\n",
"In order to support SmilesTokenizer, we have to install a fork of the `simple-transformers` library"
"We will be installing `simple-transformers`, a library which builds ontop of HuggingFace's `transformers` package specifically for fine-tuning ChemBERTa. In order to support SmilesTokenizer, we have to install a modified fork of the `simple-transformers` library"
]
},
{
@@ -6835,7 +6835,7 @@
"id": "6JGGgFolTA1m"
},
"source": [
"Now, using `simple-transformer`, let's load the pre-trained model from HuggingFace's useful model-hub. We'll set the number of epochs to 3 in the arguments, but you can train for longer. Also make sure that `auto_weights` is set to True to do automatic weight balancing, as we are dealing with imbalanced toxicity datasets. "
"Now, using `simple-transformer`, let's load the pre-trained model from HuggingFace's useful model-hub. We'll set the number of epochs to 15 in the arguments, but you can train for longer, and pass early-stopping as an argument to prevent overfitting. Also make sure that `auto_weights` is set to True to do automatic weight balancing, as we are dealing with imbalanced toxicity datasets. "
]
},
{
@@ -7662,7 +7662,7 @@
"id": "HCPFrC7mUJYq"
},
"source": [
"Let's install scikit-learn now, to evaluate the model we've trained."
"Let's install scikit-learn now, to evaluate the model we've trained. We will be using the accuracy and PRC-AUC metrics (average precision score)."
]
},
{
@@ -8324,7 +8324,7 @@
"id": "CYLS8A1aP8V-"
},
"source": [
"The model predicts the sample correctly! Some future tasks may include using the same model on multiple tasks (Tox21 provides multiple tasks relating to different biochemical pathways for toxicity, as an example), through multi-task classification, as well as training on a larger dataset such as HIV, one of the other harder tasks in molecular machine learning. This will be expanded on in future work! In the next tutorial, we'll be training a similar variant of ChemBERTa, that utilizes a different tokenizer, the [SmilesTokenizer](https://deepchem.readthedocs.io/en/latest/api_reference/tokenizers.html) which is built-in to DeepChem! Let see if using a tokenizer which splits SMILES sequences into syntatically relevant chemical tokens performs differently, especially on attention visualizaiton tasks!"
"The model predicts the sample correctly! Some future tasks may include using the same model on multiple tasks (Tox21 provides multiple tasks relating to different biochemical pathways for toxicity, as an example), through multi-task classification, as well as training on a larger dataset such as HIV, one of the other harder tasks in molecular machine learning. This will be expanded on in future work!"
