SubwordTokenizer
Defined in: parser.ts:31
Minimal sub-word tokenizer contract (satisfied by @huggingface/transformers).
SubwordTokenizer(
text,opts):object
Defined in: parser.ts:32
Minimal sub-word tokenizer contract (satisfied by @huggingface/transformers).
Parameters
Section titled “Parameters”string
add_special_tokens
Section titled “add_special_tokens”boolean
Returns
Section titled “Returns”object
input_ids
Section titled “input_ids”input_ids:
object
input_ids.data
Section titled “input_ids.data”data:
ArrayLike<number|bigint>
Properties
Section titled “Properties”cls_token_id?
Section titled “cls_token_id?”
optionalcls_token_id?:number
Defined in: parser.ts:33
unk_token_id?
Section titled “unk_token_id?”
optionalunk_token_id?:number
Defined in: parser.ts:34