colab

%% Cell type:markdown id: tags:

# Tutorial 1: The Basic Tools of the Deep Life Sciences

Welcome to DeepChem's introductory tutorial for the deep life sciences. This series of notebooks is step-by-step guide for you to get to know the new tools and techniques needed to do deep learning for the life sciences. We'll start from the basics, assuming that you're new to machine learning and the life sciences, and build up a repertoire of tools and techniques that you can use to do meaningful work in the life sciences.

**Scope:** This tutorial will encompass both the machine learning and data handling needed to build systems for the deep life sciences.

## Colab

This tutorial and the rest in the sequences are designed to be done in Google colab. If you'd like to open this notebook in colab, you can use the following link.

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/deepchem/deepchem/blob/master/examples/tutorials/01_The_Basic_Tools_of_the_Deep_Life_Sciences.ipynb)

## Why do the DeepChem Tutorial?

**1) Career Advancement:** Applying AI in the life sciences is a booming

industry at present. There are a host of newly funded startups and initiatives

at large pharmaceutical and biotech companies centered around AI. Learning and

mastering DeepChem will bring you to the forefront of this field and will

prepare you to enter a career in this field.

**2) Humanitarian Considerations:** Disease is the oldest cause of human

suffering. From the dawn of human civilization, humans have suffered from pathogens,

cancers, and neurological conditions. One of the greatest achievements of

the last few centuries has been the development of effective treatments for

many diseases. By mastering the skills in this tutorial, you will be able to

stand on the shoulders of the giants of the past to help develop new

medicine.

**3) Lowering the Cost of Medicine:** The art of developing new medicine is

currently an elite skill that can only be practiced by a small core of expert

practitioners. By enabling the growth of open source tools for drug discovery,

you can help democratize these skills and open up drug discovery to more

competition. Increased competition can help drive down the cost of medicine.

## Getting Extra Credit

If you're excited about DeepChem and want to get more more involved, there's a couple of things that you can do right now:

* Star DeepChem on GitHub! - https://github.com/deepchem/deepchem

* Join the DeepChem forums and introduce yourself! - https://forum.deepchem.io

* Say hi on the DeepChem gitter - https://gitter.im/deepchem/Lobby

* Make a YouTube video teaching the contents of this notebook.

## Prerequisites

This tutorial will assume some basic familiarity with the Python data science ecosystem. We will assume that you have familiarity with libraries such as Numpy, Pandas, and TensorFlow. We'll provide some brief refreshers on basics through the tutorial so don't worry if you're not an expert.

## Setup

The first step is to get DeepChem up and running. We recommend using Google Colab to work through this tutorial series. You'll need to run the following commands to get DeepChem installed on your colab notebook. Note that this will take something like 5 minutes to run on your colab instance.

%% Cell type:code id: tags:

``` python

!wget -c https://repo.anaconda.com/archive/Anaconda3-2019.10-Linux-x86_64.sh

!chmod +x Anaconda3-2019.10-Linux-x86_64.sh

!bash ./Anaconda3-2019.10-Linux-x86_64.sh -b -f -p /usr/local

!conda install -y -c deepchem -c rdkit -c conda-forge -c omnia deepchem-gpu=2.3.0

import sys

sys.path.append('/usr/local/lib/python3.7/site-packages/')

```

%% Cell type:markdown id: tags:

You can of course run this tutorial locally if you prefer. In this case, don't run the above cell since it will download and install Anaconda on your local machine. In either case, we can now import `deepchem` the package to play with.

%% Cell type:code id: tags:

``` python

# Run this cell to see if things work

import deepchem as dc

```

%% Output

    /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/sklearn/externals/joblib/__init__.py:15: FutureWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.

      warnings.warn(msg, category=FutureWarning)

    RDKit WARNING: [16:55:20] Enabling RDKit 2019.09.3 jupyter extensions

    /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

      _np_qint8 = np.dtype([("qint8", np.int8, 1)])

    /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

      _np_quint8 = np.dtype([("quint8", np.uint8, 1)])

    /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

      _np_qint16 = np.dtype([("qint16", np.int16, 1)])

    /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

      _np_quint16 = np.dtype([("quint16", np.uint16, 1)])

    /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

      _np_qint32 = np.dtype([("qint32", np.int32, 1)])

    /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

      np_resource = np.dtype([("resource", np.ubyte, 1)])

    /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

      _np_qint8 = np.dtype([("qint8", np.int8, 1)])

    /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

      _np_quint8 = np.dtype([("quint8", np.uint8, 1)])

    /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

      _np_qint16 = np.dtype([("qint16", np.int16, 1)])

    /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

      _np_quint16 = np.dtype([("quint16", np.uint16, 1)])

    /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

      _np_qint32 = np.dtype([("qint32", np.int32, 1)])

    /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

      np_resource = np.dtype([("resource", np.ubyte, 1)])

%% Cell type:markdown id: tags:

# Basic Data Handling in DeepChem

What does it take to do deep learning on the life sciences? Well, the first thing we'll need to do is actually handle some data. How can we start handling some basic data? For beginners, let's just take a look at some synthetic data.

To generate some basic synthetic data, we will use Numpy to create some basic arrays.

%% Cell type:code id: tags:

``` python

import numpy as np

data = np.random.random((4, 4))

labels = np.random.random((4,)) # labels of size 20x1

```

%% Cell type:markdown id: tags:

We've given these arrays some evocative names: "data" and "labels." For now, don't worry too much about the names, but just note that the arrays have different shapes. Let's take a quick look to get a feeling for these arrays

%% Cell type:code id: tags:

``` python

data, labels

```

%% Output

    (array([[0.8409268 , 0.54779812, 0.15791233, 0.83406089],

            [0.18134816, 0.90800565, 0.73072345, 0.260484  ],

            [0.00528476, 0.6911    , 0.96467034, 0.90761902],

            [0.701121  , 0.43129514, 0.49785863, 0.01989653]]),

     array([0.302203  , 0.43742869, 0.55756899, 0.92048307]))

%% Cell type:markdown id: tags:

In order to be able to work with this data in DeepChem, we need to wrap these arrays so DeepChem knows how to work with them. DeepChem has a `Dataset` API that it uses to facilitate its handling of datasets. For handling of Numpy datasets, we use DeepChem's `NumpyDataset` object.

%% Cell type:code id: tags:

``` python

from deepchem.data.datasets import NumpyDataset

dataset = NumpyDataset(data, labels)

```

%% Cell type:markdown id: tags:

Ok, now what? We have these arrays in a `NumpyDataset` object. What can we do with it? Let's try printing out the object.

%% Cell type:code id: tags:

``` python

dataset

```

%% Output

    <deepchem.data.datasets.NumpyDataset at 0x109845d68>

%% Cell type:markdown id: tags:

Ok, that's not terribly informative. It's telling us that `dataset` is a Python object that lives somewhere in memory. Can we recover the two datasets that we used to construct this object? Luckily, the DeepChem API allows us to recover the two original datasets by calling the `dataset.X` and `dataset.y` attributes of the original object.

%% Cell type:code id: tags:

``` python

dataset.X, dataset.y

```

%% Output

    (array([[0.8409268 , 0.54779812, 0.15791233, 0.83406089],

            [0.18134816, 0.90800565, 0.73072345, 0.260484  ],

            [0.00528476, 0.6911    , 0.96467034, 0.90761902],

            [0.701121  , 0.43129514, 0.49785863, 0.01989653]]),

     array([0.302203  , 0.43742869, 0.55756899, 0.92048307]))

%% Cell type:markdown id: tags:

This set of transformations raises a few questions. First, what was the point of it all? Why would we want to wrap objects this way instead of working with the raw Numpy arrays? The simple answer is for have a unified API for working with larger datasets. Suppose that `X` and `y` are so large that they can't fit easily into memory. What would we do then? Being able to work with an abstract `dataset` object proves very convenient then. In fact, you'll have reason to use this feature of `Dataset` later in the tutorial series.

What else can we do with the `dataset` object? It turns out that it can be useful to be able to walk through the datapoints in the `dataset` one at a time. For that, we can use the `dataset.itersamples()` method.

%% Cell type:code id: tags:

``` python

for x, y, _, _ in dataset.itersamples():

    print(x, y)

```

%% Output

    [0.8409268  0.54779812 0.15791233 0.83406089] 0.3022030049033878

    [0.18134816 0.90800565 0.73072345 0.260484  ] 0.4374286946177691

    [0.00528476 0.6911     0.96467034 0.90761902] 0.5575689868183242

    [0.701121   0.43129514 0.49785863 0.01989653] 0.9204830702131853

%% Cell type:markdown id: tags:

There are a couple of other fields that the `dataset` object tracks. The first is `dataset.ids`. This is a listing of unique identifiers for the datapoitns in the dataset.

%% Cell type:code id: tags:

``` python

dataset.ids

```

%% Output

    array([0, 1, 2, 3], dtype=object)

%% Cell type:markdown id: tags:

In addition, the `dataset` object has a field `dataset.w`. This is the "example weight" associated with each datapoint. Since we haven't explicitly assigned the weights, this is simply going to be all ones.

%% Cell type:code id: tags:

``` python

dataset.w

```

%% Output

    array([1., 1., 1., 1.], dtype=float32)

%% Cell type:markdown id: tags:

What if we want to set nontrivial weights for a dataset? One time we might want to do this is if we have a dataset where there are only a few positive examples to play with. It's pretty straightforward to do this with DeepChem.

%% Cell type:code id: tags:

``` python

w = np.random.random((4,)) # initializing weights with random vector of size 4x1

dataset_with_weights = NumpyDataset(data, labels, w) # creates numpy dataset object

dataset_with_weights.w

```

%% Output

    array([0.28356599, 0.10301754, 0.4463241 , 0.43093856])

%% Cell type:markdown id: tags:

## MNIST Example

Just to get a better understanding, we'll use the venerable MNIST dataset and use `NumpyDataset` to store the data. We're going to make use of the `tensorflow-datasets` package to facilitate our data reading. You'll need to install this package in order to make use of it.

%% Cell type:code id: tags:

``` python

# Install tensorflow-datasets

# TODO(rbharath): Switch to stable version on release

!pip install -q --upgrade tfds-nightly tf-nightly

## TODO(rbharath): Switch to stable version on release

# TODO(rbharath): This only works on TF2. Uncomment once we've upgraded.

#!pip install -q --upgrade tfds-nightly tf-nightly

```

%% Cell type:code id: tags:

``` python

# TODO(rbharath): This cell will only work with TF2 installed. Swap to this as default soon.

import tensorflow_datasets as tfds

#import tensorflow_datasets as tfds

data_dir = '/tmp/tfds'

#data_dir = '/tmp/tfds'

# Fetch full datasets for evaluation

## Fetch full datasets for evaluation

# tfds.load returns tf.Tensors (or tf.data.Datasets if batch_size != -1)

# You can convert them to NumPy arrays (or iterables of NumPy arrays) with tfds.dataset_as_numpy

mnist_data, info = tfds.load(name="mnist", batch_size=-1, data_dir=data_dir, with_info=True)

mnist_data = tfds.as_numpy(mnist_data)

train_data, test_data = mnist_data['train'], mnist_data['test']

num_labels = info.features['label'].num_classes

h, w, c = info.features['image'].shape

num_pixels = h * w * c

# Full train set

train_images, train_labels = train_data['image'], train_data['label']

train_images = np.reshape(train_images, (len(train_images), num_pixels))

train_labels = one_hot(train_labels, num_labels)

# Full test set

test_images, test_labels = test_data['image'], test_data['label']

test_images = np.reshape(test_images, (len(test_images), num_pixels))

test_labels = one_hot(test_labels, num_labels)

#mnist_data, info = tfds.load(name="mnist", batch_size=-1, data_dir=data_dir, with_info=True)

#mnist_data = tfds.as_numpy(mnist_data)

#train_data, test_data = mnist_data['train'], mnist_data['test']

#num_labels = info.features['label'].num_classes

#h, w, c = info.features['image'].shape

#num_pixels = h * w * c

## Full train set

#train_images, train_labels = train_data['image'], train_data['label']

#train_images = np.reshape(train_images, (len(train_images), num_pixels))

#train_labels = one_hot(train_labels, num_labels)

## Full test set

#test_images, test_labels = test_data['image'], test_data['label']

#test_images = np.reshape(test_images, (len(test_images), num_pixels))

#test_labels = one_hot(test_labels, num_labels)

```

%% Output

    ---------------------------------------------------------------------------

    AttributeError                            Traceback (most recent call last)

    <ipython-input-32-db19541136ab> in <module>

    ----> 1 import tensorflow_datasets as tfds

          2

          3 data_dir = '/tmp/tfds'

          4

          5 # Fetch full datasets for evaluation

    ~/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow_datasets/__init__.py in <module>

         44 # needs to happen before anything else, since the imports below will try to

         45 # import tensorflow, too.

    ---> 46 from tensorflow_datasets.core import tf_compat

         47 tf_compat.ensure_tf_install()

         48

    ~/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow_datasets/core/__init__.py in <module>

         19 # import tensorflow, too.

         20 from tensorflow_datasets.core import tf_compat

    ---> 21 tf_compat.ensure_tf_install()

         22

         23 from tensorflow_datasets.core.api_utils import disallow_positional_args  # pylint:disable=g-import-not-at-top

    ~/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow_datasets/core/tf_compat.py in ensure_tf_install()

         64             required="1.13.0",

         65             present=tf.__version__))

    ---> 66   _patch_tf(tf)

         67

         68

    ~/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow_datasets/core/tf_compat.py in _patch_tf(tf)

         78   if v_1_13 <= tf_version < v_2:

         79     TF_PATCH = "tf1_13"

    ---> 80     _patch_for_tf1_13(tf)

         81

         82

    ~/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow_datasets/core/tf_compat.py in _patch_for_tf1_13(tf)

         94   if not hasattr(tf.autograph.experimental, "do_not_convert"):

         95     tf.autograph.experimental.do_not_convert = (

    ---> 96         tf.contrib.autograph.do_not_convert)

         97

         98

    AttributeError: module 'tensorflow.compat.v2' has no attribute 'contrib'

%% Cell type:code id: tags:

``` python

from tensorflow.examples.tutorials.mnist import input_data

mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)

# Load the numpy data of MNIST into NumpyDataset

train = NumpyDataset(mnist.train.images, mnist.train.labels)

valid = NumpyDataset(mnist.validation.images, mnist.validation.labels)

```

%% Output

    WARNING:tensorflow:From <ipython-input-33-7ea47016376a>:3: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.

    Instructions for updating:

    Please use alternatives such as official/mnist/dataset.py from tensorflow/models.

    WARNING:tensorflow:From /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.

    Instructions for updating:

    Please write your own downloading logic.

    WARNING:tensorflow:From /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.

    Instructions for updating:

    Please use tf.data to implement this functionality.

    Extracting MNIST_data/train-images-idx3-ubyte.gz

    WARNING:tensorflow:From /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.

    Instructions for updating:

    Please use tf.data to implement this functionality.

    Extracting MNIST_data/train-labels-idx1-ubyte.gz

    WARNING:tensorflow:From /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.

    Instructions for updating:

    Please use tf.one_hot on tensors.

    Extracting MNIST_data/t10k-images-idx3-ubyte.gz

    Extracting MNIST_data/t10k-labels-idx1-ubyte.gz

    WARNING:tensorflow:From /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.

    Instructions for updating:

    Please use alternatives such as official/mnist/dataset.py from tensorflow/models.

%% Cell type:markdown id: tags:

Let's take a look at some of the data we've loaded so we can visualize our samples.

%% Cell type:code id: tags:

``` python

import matplotlib.pyplot as plt

# Visualize one sample

sample = np.reshape(train.X[5], (28, 28))

plt.imshow(sample)

plt.show()

```

%% Output

%% Cell type:markdown id: tags:

## Converting a Numpy Array to tf.data.dataset()

Let's say you want to use the `tf.data` module instead of DeepChem's data handling library. Doing this is straightforward and is quite similar to getting a `NumpyDataset` object from numpy arrays.

%% Cell type:code id: tags:

``` python

import tensorflow as tf

data_small = np.random.random((4,5))

label_small = np.random.random((4,))

dataset = tf.data.Dataset.from_tensor_slices((data_small, label_small))

print ("Data\n")

print (data_small)

print ("\n Labels")

print (label_small)

```

%% Output

    Data

    [[0.76703639 0.32740734 0.83969256 0.88836494 0.13575798]

     [0.81691255 0.93198925 0.31409331 0.97220957 0.39390713]

     [0.85819869 0.05097942 0.57634862 0.42789271 0.08894737]

     [0.36893407 0.16985698 0.29056158 0.88201174 0.08395812]]

     Labels

    [0.42013801 0.14442763 0.74540663 0.48853841]

%% Cell type:markdown id: tags:

## Extracting the numpy dataset from tf.data

In order to extract the numpy array from the `tf.data`, you first need to define an `iterator` to iterate over the `tf.data.Dataset` object and then in the tensorflow session, run over the iterator to get the data instances. Let's have a look at how it's done.

%% Cell type:code id: tags:

``` python

iterator = dataset.make_one_shot_iterator() # iterator

next_element = iterator.get_next()

numpy_data = np.zeros((4, 5))

numpy_label = np.zeros((4,))

sess = tf.Session() # tensorflow session

for i in range(4):

    data_, label_ = sess.run(next_element) # data_ contains the data and label_ contains the labels that we fed in the previous step

    numpy_data[i, :] = data_

    numpy_label[i] = label_

print ("Numpy Data")

print(numpy_data)

print ("\n Numpy Label")

print(numpy_label)

```

%% Output

    WARNING:tensorflow:From <ipython-input-37-f67e6d094179>:1: DatasetV1.make_one_shot_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.

    Instructions for updating:

    Use `for ... in dataset:` to iterate over a dataset. If using `tf.estimator`, return the `Dataset` object directly from your input function. As a last resort, you can use `tf.compat.v1.data.make_one_shot_iterator(dataset)`.

    Numpy Data

    [[0.76703639 0.32740734 0.83969256 0.88836494 0.13575798]

     [0.81691255 0.93198925 0.31409331 0.97220957 0.39390713]

     [0.85819869 0.05097942 0.57634862 0.42789271 0.08894737]

     [0.36893407 0.16985698 0.29056158 0.88201174 0.08395812]]

     Numpy Label

    [0.42013801 0.14442763 0.74540663 0.48853841]

%% Cell type:markdown id: tags:

Now that you have the numpy arrays of `data` and `labels`, you can convert it to `NumpyDataset`.

%% Cell type:code id: tags:

``` python

dataset_ = NumpyDataset(numpy_data, numpy_label) # convert to NumpyDataset

dataset_.X  # printing just to check if the data is same!!

```

%% Output

    array([[0.76703639, 0.32740734, 0.83969256, 0.88836494, 0.13575798],

           [0.81691255, 0.93198925, 0.31409331, 0.97220957, 0.39390713],

           [0.85819869, 0.05097942, 0.57634862, 0.42789271, 0.08894737],

           [0.36893407, 0.16985698, 0.29056158, 0.88201174, 0.08395812]])

%% Cell type:markdown id: tags:

## Converting NumpyDataset to `tf.data`

This can be easily done by the `make_iterator()` method of `NumpyDataset`. This converts the `NumpyDataset` to `tf.data`. Let's look how it's done!

%% Cell type:code id: tags:

``` python

iterator_ = dataset_.make_iterator() # Using make_iterator for converting NumpyDataset to tf.data

next_element_ = iterator_.get_next()

sess = tf.Session() # tensorflow session

data_and_labels = sess.run(next_element_) # data_ contains the data and label_ contains the labels that we fed in the previous step

print ("Numpy Data")

print(data_and_labels[0])  # Data in the first index

print ("\n Numpy Label")

print(data_and_labels[1])  # Labels in the second index

```

%% Output

    WARNING:tensorflow:From /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.

    Instructions for updating:

    tf.py_func is deprecated in TF V2. Instead, there are two

        options available in V2.

        - tf.py_function takes a python function which manipulates tf eager

        tensors instead of numpy arrays. It's easy to convert a tf eager tensor to

        an ndarray (just call tensor.numpy()) but having access to eager tensors

        means `tf.py_function`s can use accelerators such as GPUs as well as

        being differentiable using a gradient tape.

        - tf.numpy_function maintains the semantics of the deprecated tf.py_func

        (it is not differentiable, and manipulates numpy arrays). It drops the

        stateful argument making all functions stateful.

    Numpy Data

    [[0.81691255 0.93198925 0.31409331 0.97220957 0.39390713]

     [0.36893407 0.16985698 0.29056158 0.88201174 0.08395812]

     [0.85819869 0.05097942 0.57634862 0.42789271 0.08894737]

     [0.76703639 0.32740734 0.83969256 0.88836494 0.13575798]]

     Numpy Label

    [0.14442763 0.48853841 0.74540663 0.42013801]

%% Cell type:markdown id: tags:

# Using Splitters to split DeepChem Datasets

In this section we will have a look at the various splitters that are present in deepchem library and how each of them can be used.

### Index Splitter

We start with the IndexSplitter. This splitter returns a range object which contains the split according to the fractions provided by the user. The three range objects can then be used to iterate over the dataset as test,valid and Train.

Each of the splitters that will be used has two functions inherited from the main class that are `train_test_split` which can be used to split the data into training and tesing data and the other fucnction is `train_valid_test_split` which is used to split the data to train, validation and test split.

Note: All the splitters have a default percentage of 80,10,10 as train, valid and test respectively. But can be changed by specifying the `frac_train`,`frac_test` and `frac_valid` in the ratio we want to split the data.

%% Cell type:code id: tags:

``` python

!wget https://raw.githubusercontent.com/deepchem/deepchem/master/deepchem/models/tests/example.csv

```

%% Output

    --2020-03-05 18:21:06--  https://raw.githubusercontent.com/deepchem/deepchem/master/deepchem/models/tests/example.csv

    Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.40.133

    Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.40.133|:443... connected.

    HTTP request sent, awaiting response... 200 OK

    Length: 568 [text/plain]

    Saving to: ‘example.csv’

    example.csv         100%[===================>]     568  --.-KB/s    in 0s

    2020-03-05 18:21:06 (24.6 MB/s) - ‘example.csv’ saved [568/568]

%% Cell type:code id: tags:

``` python

import os

current_dir=os.path.dirname(os.path.realpath('__file__'))

input_data=os.path.join(current_dir,'example.csv')

```

%% Cell type:markdown id: tags:

We then featurize the data using any one of the featurizers present.

%% Cell type:code id: tags:

``` python

import deepchem as dc

tasks=['log-solubility']

featurizer=dc.feat.CircularFingerprint(size=1024)

loader = dc.data.CSVLoader(tasks=tasks, smiles_field="smiles",featurizer=featurizer)

dataset=loader.featurize(input_data)

```

%% Output

    /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/sklearn/externals/joblib/__init__.py:15: FutureWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.

      warnings.warn(msg, category=FutureWarning)

    RDKit WARNING: [18:23:34] Enabling RDKit 2019.09.3 jupyter extensions

    /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

      _np_qint8 = np.dtype([("qint8", np.int8, 1)])

    /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

      _np_quint8 = np.dtype([("quint8", np.uint8, 1)])

    /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

      _np_qint16 = np.dtype([("qint16", np.int16, 1)])

    /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

      _np_quint16 = np.dtype([("quint16", np.uint16, 1)])

    /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

      _np_qint32 = np.dtype([("qint32", np.int32, 1)])

    /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

      np_resource = np.dtype([("resource", np.ubyte, 1)])

    /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

      _np_qint8 = np.dtype([("qint8", np.int8, 1)])

    /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

      _np_quint8 = np.dtype([("quint8", np.uint8, 1)])

    /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

      _np_qint16 = np.dtype([("qint16", np.int16, 1)])

    /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

      _np_quint16 = np.dtype([("quint16", np.uint16, 1)])

    /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

      _np_qint32 = np.dtype([("qint32", np.int32, 1)])

    /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

      np_resource = np.dtype([("resource", np.ubyte, 1)])

    Loading raw samples now.

    shard_size: 8192

    About to start loading CSV from /Users/bharath/Code/deepchem/examples/tutorials/example.csv

    Loading shard 1 of size 8192.

    Featurizing sample 0

    TIMING: featurizing shard 0 took 0.023 s

    TIMING: dataset construction took 0.036 s

    Loading dataset from disk.

%% Cell type:code id: tags:

``` python

from deepchem.splits.splitters import IndexSplitter

```

%% Cell type:code id: tags:

``` python

splitter=IndexSplitter()

train_data,valid_data,test_data=splitter.split(dataset)

```

%% Cell type:code id: tags:

``` python

train_data=[i for i in train_data]

valid_data=[i for i in valid_data]

test_data=[i for i in test_data]

```

%% Cell type:code id: tags:

``` python

len(train_data),len(valid_data),len(test_data)

```

%% Output

    (8, 1, 1)

%% Cell type:markdown id: tags:

As we can see that without providing the user specifications on how to split the data, the data was split into a default of 80,10,10.

But when we specify the parameters the dataset can be split according to our specificaitons.

%% Cell type:code id: tags:

``` python

train_data,valid_data,test_data=splitter.split(dataset,frac_train=0.7,frac_valid=0.2,frac_test=0.1)

train_data=[i for i in train_data]

valid_data=[i for i in valid_data]

test_data=[i for i in test_data]

len(train_data),len(valid_data),len(test_data)

```

%% Output

    (7, 2, 1)

%% Cell type:markdown id: tags:

## Specified Splitter

The next splitter that is present in the library is the specified splitter. This splitter needs a list from the dataset where it is specified which data is for training and which is for validation and testing.

%% Cell type:code id: tags:

``` python

from deepchem.splits.splitters import SpecifiedSplitter

current_dir=os.path.dirname(os.path.realpath('__file__'))

input_file=os.path.join('../../deepchem/models/tests/user_specified_example.csv')

tasks=['log-solubility']

featurizer=dc.feat.CircularFingerprint(size=1024)

loader = dc.data.CSVLoader(tasks=tasks, smiles_field="smiles",featurizer=featurizer)

dataset=loader.featurize(input_file)

split_field='split'

splitter=SpecifiedSplitter(input_file,split_field)

```

%% Output

    Loading raw samples now.

    shard_size: 8192

    About to start loading CSV from ../../deepchem/models/tests/user_specified_example.csv

    Loading shard 1 of size 8192.

    Featurizing sample 0

    TIMING: featurizing shard 0 took 0.020 s

    TIMING: dataset construction took 0.039 s

    Loading dataset from disk.

%% Cell type:code id: tags:

``` python

train_data,valid_data,test_data=splitter.split(dataset)

```

%% Cell type:markdown id: tags:

When we split the data using the specified splitter it compares the data in each row of the `split_field` which the user has to specify wether the given row should be used as training data, validation data or testing data. The user has to specify as `train`,`test` and `valid` in the `split_field`.

Note: The input is case insensitive.

%% Cell type:code id: tags:

``` python

train_data,valid_data,test_data

```

%% Output

    ([0, 1, 2, 3, 4, 5], [6, 7], [8, 9])

%% Cell type:markdown id: tags:

## Indice Splitter

Another splitter present in the fraework is `IndiceSplitter`. This splitter takes an input of valid_indices and test_indices which are lists with the indices of validation data and test data in the dataset respectively.

%% Cell type:code id: tags:

``` python

from deepchem.splits.splitters import IndiceSplitter

splitter=IndiceSplitter(valid_indices=[7],test_indices=[9])

splitter.split(dataset)

```

%% Output

    ([0, 1, 2, 3, 4, 5, 6, 8], [7], [9])

%% Cell type:markdown id: tags:

## RandomGroupSplitter

The splitter which can be used to split the data on the basis of groupings is the `RandomGroupSplitter`. This splitter that splits on groupings.

An example use case is when there are multiple conformations of the same molecule that share the same topology.This splitter subsequently guarantees that resulting splits preserve groupings.

Note that it doesn't do any dynamic programming or something fancy to try to maximize the choice such that `frac_train`, `frac_valid`, or `frac_test` is maximized.It simply permutes the groups themselves. As such, use with caution if the number of elements per group varies significantly.

The parameter that needs to be provided with the splitter is `groups`. This is an array like list of hashables which is the same as the size of the dataset.

%% Cell type:code id: tags:

``` python

from deepchem.splits.splitters import RandomGroupSplitter

groups = [0, 4, 1, 2, 3, 7, 0, 3, 1, 0]

solubility_dataset=dc.data.tests.load_solubility_data()

splitter=RandomGroupSplitter(groups=groups)

train_idxs, valid_idxs, test_idxs = splitter.split(

        solubility_dataset)

```

%% Output

    Loading raw samples now.

    shard_size: 8192

    About to start loading CSV from /Users/bharath/Code/deepchem/deepchem/data/tests/../../models/tests/example.csv

    Loading shard 1 of size 8192.

    Featurizing sample 0