a2rl.Tokenizer#

class a2rl.Tokenizer[source]#

Bases: ABC

Abstract base class of tokenizers.

Methods

`fit`(df)	Fit this tokenizer.
`fit_transform`(df)	Call `fit()` then `transform()`.
`inverse_transform`(df)	Revert the tokenized data frame back to their original space.
`transform`(df)	Tokenize a data frame.
`valid_tokens`(col)	Get the valid tokens for column `col`.
`valid_tokens_of_col_idx`(col_idx)	Get the valid tokens for column index `col_idx`.
`valid_tokens_of_col_name`(col_name)	Get the valid tokens for column name `col_name`.