Updated tutorial and IndiceSplitter

      self.valid_indices = []

    if self.test_indices is None:

      self.test_indices = []

    valid_test = self.valid_indices

    valid_test = list(self.valid_indices)

    valid_test.extend(self.test_indices)

    for indice in indices:

      if not indice in valid_test:

      assert len(np.where(~w.any(axis=1))[0]) == 0

  def test_indice_split(self):

    solubility_dataset = dc.data.tests.load_solubility_data()

    random_splitter = dc.splits.IndiceSplitter(valid_indices=[7],test_indices=[8])

    train_data, valid_data, test_data = \

      random_splitter.split(

        solubility_dataset)

    assert len(train_data) == 8

    assert len(valid_data) == 1

    assert len(test_data) == 1

if __name__ == "__main__":

  import nose

%% Cell type:markdown id: tags:

# Using Splitters

%% Cell type:markdown id: tags:

In this tutorial we will have a look at the various splitters that are present in deepchem library and how each of them can be used.

%% Cell type:code id: tags:

``` python

import deepchem as dc

import pandas as pd

import os

```

%% Output

    /home/ab/anaconda3/envs/deepchem/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.

      from ._conv import register_converters as _register_converters

%% Cell type:markdown id: tags:

## Index Splitter

%% Cell type:markdown id: tags:

We start with the IndexSplitter. This splitter returns a range object which contains the split according to the fractions provided by the user. The three range objects can then be used to iterate over the dataset as test,valid and Train.

Each of the splitters that will be used has two functions inherited from the main class that are `train_test_split` which can be used to split the data into training and tesing data and the other fucnction is `train_valid_test_split` which is used to split the data to train, validation and test split.

Note: All the splitters have a default percentage of 80,10,10 as train, valid and test respectively. But can be changed by specifying the `frac_train`,`frac_test` and `frac_valid` in the ratio we want to split the data.

%% Cell type:code id: tags:

``` python

current_dir=os.path.dirname(os.path.realpath('__file__'))

input_data=os.path.join(current_dir,'../../deepchem/models/tests/example.csv')

```

%% Cell type:markdown id: tags:

We then featurize the data using any one of the featurizers present.

%% Cell type:code id: tags:

``` python

tasks=['log-solubility']

featurizer=dc.feat.CircularFingerprint(size=1024)

loader = dc.data.CSVLoader(tasks=tasks, smiles_field="smiles",featurizer=featurizer)

dataset=loader.featurize(input_data)

```

%% Output

    Loading raw samples now.

    shard_size: 8192

    About to start loading CSV from /home/ab/deepchem/examples/notebooks/../../deepchem/models/tests/example.csv

    Loading shard 1 of size 8192.

    Featurizing sample 0

    TIMING: featurizing shard 0 took 0.033 s

    TIMING: dataset construction took 0.042 s

    Loading dataset from disk.

%% Cell type:code id: tags:

``` python

from deepchem.splits.splitters import IndexSplitter

```

%% Cell type:code id: tags:

``` python

splitter=IndexSplitter()

train_data,valid_data,test_data=splitter.split(dataset)

```

%% Cell type:code id: tags:

``` python

train_data=[i for i in train_data]

valid_data=[i for i in valid_data]

test_data=[i for i in test_data]

```

%% Cell type:code id: tags:

``` python

len(train_data),len(valid_data),len(test_data)

```

%% Output

    (8, 1, 1)

%% Cell type:markdown id: tags:

As we can see that without providing the user specifications on how to split the data, the data was split into a default of 80,10,10.

But when we specify the parameters the dataset can be split according to our specificaitons.

%% Cell type:code id: tags:

``` python

train_data,valid_data,test_data=splitter.split(dataset,frac_train=0.7,frac_valid=0.2,frac_test=0.1)

```

%% Cell type:code id: tags:

``` python

train_data=[i for i in train_data]

valid_data=[i for i in valid_data]

test_data=[i for i in test_data]

```

%% Cell type:code id: tags:

``` python

len(train_data),len(valid_data),len(test_data)

```

%% Output

    (7, 2, 1)

%% Cell type:markdown id: tags:

## Specified Splitter

%% Cell type:markdown id: tags:

The next splitter that is present in the library is the specified splitter. This splitter needs a list from the dataset where it is specified which data is for training and which is for validation and testing.

%% Cell type:code id: tags:

``` python

from deepchem.splits.splitters import SpecifiedSplitter

current_dir=os.path.dirname(os.path.realpath('__file__'))

input_file=os.path.join('../../deepchem/models/tests/user_specified_example.csv')

tasks=['log-solubility']

featurizer=dc.feat.CircularFingerprint(size=1024)

loader = dc.data.CSVLoader(tasks=tasks, smiles_field="smiles",featurizer=featurizer)

dataset=loader.featurize(input_file)

split_field='split'

splitter=SpecifiedSplitter(input_file,split_field)

```

%% Output

    Loading raw samples now.

    shard_size: 8192

    About to start loading CSV from ../../deepchem/models/tests/user_specified_example.csv

    Loading shard 1 of size 8192.

    Featurizing sample 0

    TIMING: featurizing shard 0 took 0.030 s

    TIMING: dataset construction took 0.040 s

    Loading dataset from disk.

%% Cell type:code id: tags:

``` python

train_data,valid_data,test_data=splitter.split(dataset)

```

%% Cell type:markdown id: tags:

When we split the data using the specified splitter it compares the data in each row of the `split_field` which the user has to specify wether the given row should be used as training data, validation data or testing data. The user has to specify as `train`,`test` and `valid` in the `split_field`.

Note: The input is case insensitive.

%% Cell type:code id: tags:

``` python

train_data,valid_data,test_data

```

%% Output

    ([0, 1, 2, 3, 4, 5], [6, 7], [8, 9])

%% Cell type:markdown id: tags:

## Indice Splitter

%% Cell type:markdown id: tags:

Another splitter present in the fraework is `IndiceSplitter`. This splitter takes an input of valid_indices and test_indices which are lists with the indices of validation data and test data in the dataset respectively.

%% Cell type:code id: tags:

``` python

from deepchem.splits.splitters import IndiceSplitter

splitter=IndiceSplitter(valid_indices=[7],test_indices=[9])

```

%% Cell type:code id: tags:

``` python

splitter.split(dataset)

```

%% Output

    ([0, 1, 2, 3, 4, 5, 6, 8], [7, 9], [9])

%% Cell type:markdown id: tags:

## RandomGroupSplitter

%% Cell type:markdown id: tags:

The splitter which can be used to split the data on the basis of groupings is the `RandomGroupSplitter`. This splitter that splits on groupings.

An example use case is when there are multiple conformations of the same molecule that share the same topology.This splitter subsequently guarantees that resulting splits preserve groupings.

Note that it doesn't do any dynamic programming or something fancy to try to maximize the choice such that `frac_train`, `frac_valid`, or `frac_test` is maximized.It simply permutes the groups themselves. As such, use with caution if the number of elements per group varies significantly.

The parameter that needs to be provided with the splitter is `groups`. This is an array like list of hashables which is the same as the size of the dataset.

%% Cell type:code id: tags:

``` python

from deepchem.splits.splitters import RandomGroupSplitter

group=[i for i in range(len(dataset))]

splitter=RandomGroupSplitter(groups=group)

data=pd.read_csv(input_data)

```

%% Cell type:code id: tags:

``` python

splitter.split(dataset,frac_train=0.7,frac_valid=0.2,frac_test=0.1)

```

%% Output

    ([7, 0, 8, 6, 1, 2, 3], [4, 9], [5])

%% Cell type:code id: tags:

``` python

```