# Tutorial 1: The Basic Tools of the Deep Life Sciences
# Tutorial 1: The Basic Tools of the Deep Life Sciences
Welcome to DeepChem's introductory tutorial for the deep life sciences. This series of notebooks is step-by-step guide for you to get to know the new tools and techniques needed to do deep learning for the life sciences. We'll start from the basics, assuming that you're new to machine learning and the life sciences, and build up a repertoire of tools and techniques that you can use to do meaningful work in the life sciences.
Welcome to DeepChem's introductory tutorial for the deep life sciences. This series of notebooks is step-by-step guide for you to get to know the new tools and techniques needed to do deep learning for the life sciences. We'll start from the basics, assuming that you're new to machine learning and the life sciences, and build up a repertoire of tools and techniques that you can use to do meaningful work in the life sciences.
**Scope:** This tutorial will encompass both the machine learning and data handling needed to build systems for the deep life sciences.
**Scope:** This tutorial will encompass both the machine learning and data handling needed to build systems for the deep life sciences.
## Colab
## Colab
This tutorial and the rest in the sequences are designed to be done in Google colab. If you'd like to open this notebook in colab, you can use the following link.
This tutorial and the rest in the sequences are designed to be done in Google colab. If you'd like to open this notebook in colab, you can use the following link.
[](https://colab.research.google.com/github/deepchem/deepchem/blob/master/examples/tutorials/01_The_Basic_Tools_of_the_Deep_Life_Sciences.ipynb)
[](https://colab.research.google.com/github/deepchem/deepchem/blob/master/examples/tutorials/01_The_Basic_Tools_of_the_Deep_Life_Sciences.ipynb)
## Why do the DeepChem Tutorial?
## Why do the DeepChem Tutorial?
**1) Career Advancement:** Applying AI in the life sciences is a booming
**1) Career Advancement:** Applying AI in the life sciences is a booming
industry at present. There are a host of newly funded startups and initiatives
industry at present. There are a host of newly funded startups and initiatives
at large pharmaceutical and biotech companies centered around AI. Learning and
at large pharmaceutical and biotech companies centered around AI. Learning and
mastering DeepChem will bring you to the forefront of this field and will
mastering DeepChem will bring you to the forefront of this field and will
prepare you to enter a career in this field.
prepare you to enter a career in this field.
**2) Humanitarian Considerations:** Disease is the oldest cause of human
**2) Humanitarian Considerations:** Disease is the oldest cause of human
suffering. From the dawn of human civilization, humans have suffered from pathogens,
suffering. From the dawn of human civilization, humans have suffered from pathogens,
cancers, and neurological conditions. One of the greatest achievements of
cancers, and neurological conditions. One of the greatest achievements of
the last few centuries has been the development of effective treatments for
the last few centuries has been the development of effective treatments for
many diseases. By mastering the skills in this tutorial, you will be able to
many diseases. By mastering the skills in this tutorial, you will be able to
stand on the shoulders of the giants of the past to help develop new
stand on the shoulders of the giants of the past to help develop new
medicine.
medicine.
**3) Lowering the Cost of Medicine:** The art of developing new medicine is
**3) Lowering the Cost of Medicine:** The art of developing new medicine is
currently an elite skill that can only be practiced by a small core of expert
currently an elite skill that can only be practiced by a small core of expert
practitioners. By enabling the growth of open source tools for drug discovery,
practitioners. By enabling the growth of open source tools for drug discovery,
you can help democratize these skills and open up drug discovery to more
you can help democratize these skills and open up drug discovery to more
competition. Increased competition can help drive down the cost of medicine.
competition. Increased competition can help drive down the cost of medicine.
## Getting Extra Credit
## Getting Extra Credit
If you're excited about DeepChem and want to get more more involved, there's a couple of things that you can do right now:
If you're excited about DeepChem and want to get more more involved, there's a couple of things that you can do right now:
* Star DeepChem on GitHub! - https://github.com/deepchem/deepchem
* Star DeepChem on GitHub! - https://github.com/deepchem/deepchem
* Join the DeepChem forums and introduce yourself! - https://forum.deepchem.io
* Join the DeepChem forums and introduce yourself! - https://forum.deepchem.io
* Say hi on the DeepChem gitter - https://gitter.im/deepchem/Lobby
* Say hi on the DeepChem gitter - https://gitter.im/deepchem/Lobby
* Make a YouTube video teaching the contents of this notebook.
* Make a YouTube video teaching the contents of this notebook.
## Prerequisites
## Prerequisites
This tutorial will assume some basic familiarity with the Python data science ecosystem. We will assume that you have familiarity with libraries such as Numpy, Pandas, and TensorFlow. We'll provide some brief refreshers on basics through the tutorial so don't worry if you're not an expert.
This tutorial will assume some basic familiarity with the Python data science ecosystem. We will assume that you have familiarity with libraries such as Numpy, Pandas, and TensorFlow. We'll provide some brief refreshers on basics through the tutorial so don't worry if you're not an expert.
## Setup
## Setup
The first step is to get DeepChem up and running. We recommend using Google Colab to work through this tutorial series. You'll need to run the following commands to get DeepChem installed on your colab notebook. Note that this will take something like 5 minutes to run on your colab instance.
The first step is to get DeepChem up and running. We recommend using Google Colab to work through this tutorial series. You'll need to run the following commands to get DeepChem installed on your colab notebook. Note that this will take something like 5 minutes to run on your colab instance.
add /root/miniconda/lib/python3.6/site-packages to PYTHONPATH
add /root/miniconda/lib/python3.6/site-packages to PYTHONPATH
python version: 3.6.9
python version: 3.6.9
fetching installer from https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
fetching installer from https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
done
done
installing miniconda to /root/miniconda
installing miniconda to /root/miniconda
done
done
installing deepchem
installing deepchem
done
done
/usr/local/lib/python3.6/dist-packages/sklearn/externals/joblib/__init__.py:15: FutureWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
/usr/local/lib/python3.6/dist-packages/sklearn/externals/joblib/__init__.py:15: FutureWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
warnings.warn(msg, category=FutureWarning)
warnings.warn(msg, category=FutureWarning)
WARNING:tensorflow:
WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
The TensorFlow contrib module will not be included in TensorFlow 2.0.
* https://github.com/tensorflow/io (for I/O related ops)
* https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.
If you depend on functionality not listed there, please file an issue.
deepchem-2.3.0 installation finished!
deepchem-2.3.0 installation finished!
CPU times: user 3.15 s, sys: 662 ms, total: 3.81 s
CPU times: user 3.15 s, sys: 662 ms, total: 3.81 s
Wall time: 2min 22s
Wall time: 2min 22s
%% Cell type:markdown id: tags:
%% Cell type:markdown id: tags:
You can of course run this tutorial locally if you prefer. In this case, don't run the above cell since it will download and install Anaconda on your local machine. In either case, we can now import `deepchem` the package to play with.
You can of course run this tutorial locally if you prefer. In this case, don't run the above cell since it will download and install Anaconda on your local machine. In either case, we can now import `deepchem` the package to play with.
%% Cell type:code id: tags:
%% Cell type:code id: tags:
``` python
``` python
# Run this cell to see if things work
# Run this cell to see if things work
importdeepchemasdc
importdeepchemasdc
```
```
%% Cell type:markdown id: tags:
%% Cell type:markdown id: tags:
# Basic Data Handling in DeepChem
# Basic Data Handling in DeepChem
What does it take to do deep learning on the life sciences? Well, the first thing we'll need to do is actually handle some data. How can we start handling some basic data? For beginners, let's just take a look at some synthetic data.
What does it take to do deep learning on the life sciences? Well, the first thing we'll need to do is actually handle some data. How can we start handling some basic data? For beginners, let's just take a look at some synthetic data.
To generate some basic synthetic data, we will use Numpy to create some basic arrays.
To generate some basic synthetic data, we will use Numpy to create some basic arrays.
%% Cell type:code id: tags:
%% Cell type:code id: tags:
``` python
``` python
importnumpyasnp
importnumpyasnp
data=np.random.random((4,4))
data=np.random.random((4,4))
labels=np.random.random((4,))# labels of size 20x1
labels=np.random.random((4,))# labels of size 20x1
```
```
%% Cell type:markdown id: tags:
%% Cell type:markdown id: tags:
We've given these arrays some evocative names: "data" and "labels." For now, don't worry too much about the names, but just note that the arrays have different shapes. Let's take a quick look to get a feeling for these arrays
We've given these arrays some evocative names: "data" and "labels." For now, don't worry too much about the names, but just note that the arrays have different shapes. Let's take a quick look to get a feeling for these arrays
In order to be able to work with this data in DeepChem, we need to wrap these arrays so DeepChem knows how to work with them. DeepChem has a `Dataset` API that it uses to facilitate its handling of datasets. For handling of Numpy datasets, we use DeepChem's `NumpyDataset` object.
In order to be able to work with this data in DeepChem, we need to wrap these arrays so DeepChem knows how to work with them. DeepChem has a `Dataset` API that it uses to facilitate its handling of datasets. For handling of Numpy datasets, we use DeepChem's `NumpyDataset` object.
%% Cell type:code id: tags:
%% Cell type:code id: tags:
``` python
``` python
fromdeepchem.data.datasetsimportNumpyDataset
fromdeepchem.data.datasetsimportNumpyDataset
dataset=NumpyDataset(data,labels)
dataset=NumpyDataset(data,labels)
```
```
%% Cell type:markdown id: tags:
%% Cell type:markdown id: tags:
Ok, now what? We have these arrays in a `NumpyDataset` object. What can we do with it? Let's try printing out the object.
Ok, now what? We have these arrays in a `NumpyDataset` object. What can we do with it? Let's try printing out the object.
%% Cell type:code id: tags:
%% Cell type:code id: tags:
``` python
``` python
dataset
dataset
```
```
%% Output
%% Output
<deepchem.data.datasets.NumpyDataset at 0x7f0c85883ba8>
<deepchem.data.datasets.NumpyDataset at 0x7f0c85883ba8>
%% Cell type:markdown id: tags:
%% Cell type:markdown id: tags:
Ok, that's not terribly informative. It's telling us that `dataset` is a Python object that lives somewhere in memory. Can we recover the two datasets that we used to construct this object? Luckily, the DeepChem API allows us to recover the two original datasets by calling the `dataset.X` and `dataset.y` attributes of the original object.
Ok, that's not terribly informative. It's telling us that `dataset` is a Python object that lives somewhere in memory. Can we recover the two datasets that we used to construct this object? Luckily, the DeepChem API allows us to recover the two original datasets by calling the `dataset.X` and `dataset.y` attributes of the original object.
This set of transformations raises a few questions. First, what was the point of it all? Why would we want to wrap objects this way instead of working with the raw Numpy arrays? The simple answer is for have a unified API for working with larger datasets. Suppose that `X` and `y` are so large that they can't fit easily into memory. What would we do then? Being able to work with an abstract `dataset` object proves very convenient then. In fact, you'll have reason to use this feature of `Dataset` later in the tutorial series.
This set of transformations raises a few questions. First, what was the point of it all? Why would we want to wrap objects this way instead of working with the raw Numpy arrays? The simple answer is for have a unified API for working with larger datasets. Suppose that `X` and `y` are so large that they can't fit easily into memory. What would we do then? Being able to work with an abstract `dataset` object proves very convenient then. In fact, you'll have reason to use this feature of `Dataset` later in the tutorial series.
What else can we do with the `dataset` object? It turns out that it can be useful to be able to walk through the datapoints in the `dataset` one at a time. For that, we can use the `dataset.itersamples()` method.
What else can we do with the `dataset` object? It turns out that it can be useful to be able to walk through the datapoints in the `dataset` one at a time. For that, we can use the `dataset.itersamples()` method.
There are a couple of other fields that the `dataset` object tracks. The first is `dataset.ids`. This is a listing of unique identifiers for the datapoitns in the dataset.
There are a couple of other fields that the `dataset` object tracks. The first is `dataset.ids`. This is a listing of unique identifiers for the datapoints in the dataset.
%% Cell type:code id: tags:
%% Cell type:code id: tags:
``` python
``` python
dataset.ids
dataset.ids
```
```
%% Output
%% Output
array([0, 1, 2, 3], dtype=object)
array([0, 1, 2, 3], dtype=object)
%% Cell type:markdown id: tags:
%% Cell type:markdown id: tags:
In addition, the `dataset` object has a field `dataset.w`. This is the "example weight" associated with each datapoint. Since we haven't explicitly assigned the weights, this is simply going to be all ones.
In addition, the `dataset` object has a field `dataset.w`. This is the "example weight" associated with each datapoint. Since we haven't explicitly assigned the weights, this is simply going to be all ones.
%% Cell type:code id: tags:
%% Cell type:code id: tags:
``` python
``` python
dataset.w
dataset.w
```
```
%% Output
%% Output
array([1., 1., 1., 1.], dtype=float32)
array([1., 1., 1., 1.], dtype=float32)
%% Cell type:markdown id: tags:
%% Cell type:markdown id: tags:
What if we want to set nontrivial weights for a dataset? One time we might want to do this is if we have a dataset where there are only a few positive examples to play with. It's pretty straightforward to do this with DeepChem.
What if we want to set nontrivial weights for a dataset? One time we might want to do this is if we have a dataset where there are only a few positive examples to play with. It's pretty straightforward to do this with DeepChem.
%% Cell type:code id: tags:
%% Cell type:code id: tags:
``` python
``` python
w=np.random.random((4,))# initializing weights with random vector of size 4x1
w=np.random.random((4,))# initializing weights with random vector of size 4x1
Just to get a better understanding, we'll use the venerable MNIST dataset and use `NumpyDataset` to store the data. We're going to make use of the `tensorflow-datasets` package to facilitate our data reading. You'll need to install this package in order to make use of it.
Just to get a better understanding, we'll use the venerable MNIST dataset and use `NumpyDataset` to store the data. We're going to make use of the `tensorflow-datasets` package to facilitate our data reading. You'll need to install this package in order to make use of it.
%% Cell type:code id: tags:
%% Cell type:code id: tags:
``` python
``` python
# Install tensorflow-datasets
# Install tensorflow-datasets
## TODO(rbharath): Switch to stable version on release
## TODO(rbharath): Switch to stable version on release
# TODO(rbharath): This only works on TF2. Uncomment once we've upgraded.
# TODO(rbharath): This only works on TF2. Uncomment once we've upgraded.
WARNING:tensorflow:From <ipython-input-14-7ea47016376a>:3: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
WARNING:tensorflow:From <ipython-input-14-7ea47016376a>:3: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Instructions for updating:
Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/contrib/learn/python/learn/datasets/mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/contrib/learn/python/learn/datasets/mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Instructions for updating:
Please write your own downloading logic.
Please write your own downloading logic.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/contrib/learn/python/learn/datasets/base.py:252: _internal_retry.<locals>.wrap.<locals>.wrapped_fn (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/contrib/learn/python/learn/datasets/base.py:252: _internal_retry.<locals>.wrap.<locals>.wrapped_fn (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/contrib/learn/python/learn/datasets/mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/contrib/learn/python/learn/datasets/mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Instructions for updating:
Please use tf.data to implement this functionality.
Please use tf.data to implement this functionality.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/contrib/learn/python/learn/datasets/mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/contrib/learn/python/learn/datasets/mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Instructions for updating:
Please use tf.data to implement this functionality.
Please use tf.data to implement this functionality.
Extracting MNIST_data/train-labels-idx1-ubyte.gz
Extracting MNIST_data/train-labels-idx1-ubyte.gz
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/contrib/learn/python/learn/datasets/mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/contrib/learn/python/learn/datasets/mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/contrib/learn/python/learn/datasets/mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/contrib/learn/python/learn/datasets/mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Instructions for updating:
Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
%% Cell type:markdown id: tags:
%% Cell type:markdown id: tags:
Let's take a look at some of the data we've loaded so we can visualize our samples.
Let's take a look at some of the data we've loaded so we can visualize our samples.
%% Cell type:code id: tags:
%% Cell type:code id: tags:
``` python
``` python
importmatplotlib.pyplotasplt
importmatplotlib.pyplotasplt
# Visualize one sample
# Visualize one sample
sample=np.reshape(train.X[5],(28,28))
sample=np.reshape(train.X[5],(28,28))
plt.imshow(sample)
plt.imshow(sample)
plt.show()
plt.show()
```
```
%% Output
%% Output
%% Cell type:markdown id: tags:
%% Cell type:markdown id: tags:
## Converting a Numpy Array to tf.data.dataset()
## Converting a Numpy Array to tf.data.dataset()
Let's say you want to use the `tf.data` module instead of DeepChem's data handling library. Doing this is straightforward and is quite similar to getting a `NumpyDataset` object from numpy arrays.
Let's say you want to use the `tf.data` module instead of DeepChem's data handling library. Doing this is straightforward and is quite similar to getting a `NumpyDataset` object from numpy arrays.
In order to extract the numpy array from the `tf.data`, you first need to define an `iterator` to iterate over the `tf.data.Dataset` object and then in the tensorflow session, run over the iterator to get the data instances. Let's have a look at how it's done.
In order to extract the numpy array from the `tf.data`, you first need to define an `iterator` to iterate over the `tf.data.Dataset` object and then in the tensorflow session, run over the iterator to get the data instances. Let's have a look at how it's done.
data_,label_=sess.run(next_element)# data_ contains the data and label_ contains the labels that we fed in the previous step
data_,label_=sess.run(next_element)# data_ contains the data and label_ contains the labels that we fed in the previous step
numpy_data[i,:]=data_
numpy_data[i,:]=data_
numpy_label[i]=label_
numpy_label[i]=label_
print ("Numpy Data")
print ("Numpy Data")
print(numpy_data)
print(numpy_data)
print ("\n Numpy Label")
print ("\n Numpy Label")
print(numpy_label)
print(numpy_label)
```
```
%% Output
%% Output
WARNING:tensorflow:From <ipython-input-17-f67e6d094179>:1: DatasetV1.make_one_shot_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
WARNING:tensorflow:From <ipython-input-17-f67e6d094179>:1: DatasetV1.make_one_shot_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Instructions for updating:
Use `for ... in dataset:` to iterate over a dataset. If using `tf.estimator`, return the `Dataset` object directly from your input function. As a last resort, you can use `tf.compat.v1.data.make_one_shot_iterator(dataset)`.
Use `for ... in dataset:` to iterate over a dataset. If using `tf.estimator`, return the `Dataset` object directly from your input function. As a last resort, you can use `tf.compat.v1.data.make_one_shot_iterator(dataset)`.
In this section we will have a look at the various splitters that are present in deepchem library and how each of them can be used.
In this section we will have a look at the various splitters that are present in deepchem library and how each of them can be used.
### Index Splitter
### Index Splitter
We start with the IndexSplitter. This splitter returns a range object which contains the split according to the fractions provided by the user. The three range objects can then be used to iterate over the dataset as test,valid and Train.
We start with the IndexSplitter. This splitter returns a range object which contains the split according to the fractions provided by the user. The three range objects can then be used to iterate over the dataset as test,valid and Train.
Each of the splitters that will be used has two functions inherited from the main class that are `train_test_split` which can be used to split the data into training and tesing data and the other fucnction is `train_valid_test_split` which is used to split the data to train, validation and test split.
Each of the splitters that will be used has two functions inherited from the main class that are `train_test_split` which can be used to split the data into training and tesing data and the other fucnction is `train_valid_test_split` which is used to split the data to train, validation and test split.
Note: All the splitters have a default percentage of 80,10,10 as train, valid and test respectively. But can be changed by specifying the `frac_train`,`frac_test` and `frac_valid` in the ratio we want to split the data.
Note: All the splitters have a default percentage of 80,10,10 as train, valid and test respectively. But can be changed by specifying the `frac_train`,`frac_test` and `frac_valid` in the ratio we want to split the data.
The next splitter that is present in the library is the specified splitter. This splitter needs a list from the dataset where it is specified which data is for training and which is for validation and testing.
The next splitter that is present in the library is the specified splitter. This splitter needs a list from the dataset where it is specified which data is for training and which is for validation and testing.
When we split the data using the specified splitter it compares the data in each row of the `split_field` which the user has to specify wether the given row should be used as training data, validation data or testing data. The user has to specify as `train`,`test` and `valid` in the `split_field`.
When we split the data using the specified splitter it compares the data in each row of the `split_field` which the user has to specify wether the given row should be used as training data, validation data or testing data. The user has to specify as `train`,`test` and `valid` in the `split_field`.
Note: The input is case insensitive.
Note: The input is case insensitive.
%% Cell type:code id: tags:
%% Cell type:code id: tags:
``` python
``` python
train_data,valid_data,test_data
train_data,valid_data,test_data
```
```
%% Output
%% Output
([0, 1, 2, 3, 4, 5], [6, 7], [8, 9])
([0, 1, 2, 3, 4, 5], [6, 7], [8, 9])
%% Cell type:markdown id: tags:
%% Cell type:markdown id: tags:
## Indice Splitter
## Indice Splitter
Another splitter present in the fraework is `IndiceSplitter`. This splitter takes an input of valid_indices and test_indices which are lists with the indices of validation data and test data in the dataset respectively.
Another splitter present in the fraework is `IndiceSplitter`. This splitter takes an input of valid_indices and test_indices which are lists with the indices of validation data and test data in the dataset respectively.
The splitter which can be used to split the data on the basis of groupings is the `RandomGroupSplitter`. This splitter that splits on groupings.
The splitter which can be used to split the data on the basis of groupings is the `RandomGroupSplitter`. This splitter that splits on groupings.
An example use case is when there are multiple conformations of the same molecule that share the same topology.This splitter subsequently guarantees that resulting splits preserve groupings.
An example use case is when there are multiple conformations of the same molecule that share the same topology.This splitter subsequently guarantees that resulting splits preserve groupings.
Note that it doesn't do any dynamic programming or something fancy to try to maximize the choice such that `frac_train`, `frac_valid`, or `frac_test` is maximized.It simply permutes the groups themselves. As such, use with caution if the number of elements per group varies significantly.
Note that it doesn't do any dynamic programming or something fancy to try to maximize the choice such that `frac_train`, `frac_valid`, or `frac_test` is maximized.It simply permutes the groups themselves. As such, use with caution if the number of elements per group varies significantly.
The parameter that needs to be provided with the splitter is `groups`. This is an array like list of hashables which is the same as the size of the dataset.
The parameter that needs to be provided with the splitter is `groups`. This is an array like list of hashables which is the same as the size of the dataset.
print("Groups present in the training data =",train_data)
print("Groups present in the training data =",train_data)
print("Groups present in the validation data = ",valid_data)
print("Groups present in the validation data = ",valid_data)
print("Groups present in the testing data = ",test_data)
print("Groups present in the testing data = ",test_data)
```
```
%% Output
%% Output
Groups present in the training data = [3, 3, 1, 1, 4, 0, 0, 0]
Groups present in the training data = [3, 3, 1, 1, 4, 0, 0, 0]
Groups present in the validation data = [7]
Groups present in the validation data = [7]
Groups present in the testing data = [2]
Groups present in the testing data = [2]
%% Cell type:markdown id: tags:
%% Cell type:markdown id: tags:
So the `RandomGroupSplitter` when properly assigned the groups, splits the data accordingly and preserves the groupings.
So the `RandomGroupSplitter` when properly assigned the groups, splits the data accordingly and preserves the groupings.
%% Cell type:markdown id: tags:
%% Cell type:markdown id: tags:
## Scaffold Splitter
## Scaffold Splitter
The `ScaffoldSplitter` splits the data based on the scaffold of small molecules. The splitter takes the data and generates scaffolds using the smiles in the data. Then the splitter sorts the data into scaffold sets.
The `ScaffoldSplitter` splits the data based on the scaffold of small molecules. The splitter takes the data and generates scaffolds using the smiles in the data. Then the splitter sorts the data into scaffold sets.
Congratulations on completing this tutorial notebook! If you enjoyed working through the tutorial, and want to continue working with DeepChem, we encourage you to finish the rest of the tutorials in this series. You can also help the DeepChem community in the following ways:
Congratulations on completing this tutorial notebook! If you enjoyed working through the tutorial, and want to continue working with DeepChem, we encourage you to finish the rest of the tutorials in this series. You can also help the DeepChem community in the following ways:
## Star DeepChem on [GitHub](https://github.com/deepchem/deepchem)
## Star DeepChem on [GitHub](https://github.com/deepchem/deepchem)
This helps build awareness of the DeepChem project and the tools for open source drug discovery that we're trying to build.
This helps build awareness of the DeepChem project and the tools for open source drug discovery that we're trying to build.
## Join the DeepChem Gitter
## Join the DeepChem Gitter
The DeepChem [Gitter](https://gitter.im/deepchem/Lobby) hosts a number of scientists, developers, and enthusiasts interested in deep learning for the life sciences. Join the conversation!
The DeepChem [Gitter](https://gitter.im/deepchem/Lobby) hosts a number of scientists, developers, and enthusiasts interested in deep learning for the life sciences. Join the conversation!
