generative-ai-cdk-constructs

@cdklabs/generative-ai-cdk-constructs


@cdklabs/generative-ai-cdk-constructs / opensearchserverless / TokenizerType

Enumeration: TokenizerType

Enumeration Members

ICU_TOKENIZER

ICU_TOKENIZER: "icu_tokenizer"

ICU tokenizer is used for Unicode text segmentation based on UAX #29 rules


KUROMOJI_TOKENIZER

KUROMOJI_TOKENIZER: "kuromoji_tokenizer"

Kuromoji tokenizer is used for Japanese text analysis and segmentation


NORI_TOKENIZER

NORI_TOKENIZER: "nori_tokenizer"

Nori tokenizer is used for Korean text analysis and segmentation