Commit 0fe4ad1e authored by seyonechithrananda's avatar seyonechithrananda
Browse files

add PreTrainedTokenizers link

parent 1c2db085
Loading
Loading
Loading
Loading
+1 −2
Original line number Diff line number Diff line
@@ -5,7 +5,7 @@ A tokenizer is in charge of preparing the inputs for a model. The HuggingFace tr

The base classes PreTrainedTokenizer and PreTrainedTokenizerFast implements the common methods for encoding string inputs in model inputs and instantiating/saving python tokenizers either from a local file or directory or from a pretrained tokenizer provided by the library (downloaded from HuggingFace’s AWS S3 repository).

PreTrainedTokenizer and PreTrainedTokenizerFast thus implements the main methods for using all the tokenizers:
PreTrainedTokenizer `(transformers.PreTrainedTokenizer) <https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer>`_ thus implements the main methods for using all the tokenizers:

- Tokenizing (spliting strings in sub-word token strings), converting tokens strings to ids and back, and encoding/decoding (i.e. tokenizing + convert to integers),

@@ -17,7 +17,6 @@ BatchEncoding holds the output of the tokenizer’s encoding methods (__call__,

For more details on the base tokenizers which the DeepChem tokenizers inherit from, please refer to the following: `HuggingFace tokenizers docs <https://huggingface.co/transformers/main_classes/tokenizer.html>`_


SmilesTokenizer
^^^^^^^^^^^^^^^