@cdklabs/generative-ai-cdk-constructs
@cdklabs/generative-ai-cdk-constructs / opensearchserverless / TokenizerType
ICU_TOKENIZER:
"icu_tokenizer"
ICU tokenizer is used for Unicode text segmentation based on UAX #29 rules
KUROMOJI_TOKENIZER:
"kuromoji_tokenizer"
Kuromoji tokenizer is used for Japanese text analysis and segmentation
NORI_TOKENIZER:
"nori_tokenizer"
Nori tokenizer is used for Korean text analysis and segmentation