tokenize
Splits and parses text into tokens based on the specified tokenizer.
Syntaxβ
ARRRY<VARCHAR> tokenize(VARCHAR tokenizer_name, VARCHAR content);
Parametersβ
-
tokenizer_name
: The tokenizer to use. Valid values:english
,standard
, andchinese
. -
content
: The text to be tokenized. This item can be a constant string or a column name. If a column is specified, it must be of the STRING or VARCHAR type.
Return valueβ
Returns a array of VARCHAR.
Examplesβ
MYSQL > SELECT tokenize('english', 'Today is saturday');
+------------------------------------------+
| tokenize('english', 'Today is saturday') |
+------------------------------------------+
| ["today","is","saturday"] |
+------------------------------------------+
keywordβ
TOKENIZE