sentencepiece
Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.
v1.0.01 install
Signing
SignedSLSA L2- Signed by
- skills-hub.ai distributor
- Method
- Distributor-signed by skills-hub.aiCryptographically signed by the skills-hub.ai distributor key at publish time.
- Signed
Install this skill
Run this command in your terminal. No account required — it auto-detects your AI tool and installs the skill file.
npx @skills-hub-ai/cli install ai-research-sentencepieceOr download directly:
Browse all CLI commands →Setup by platform
Install
One-click setup for your editorRun in your project root
npx @skills-hub-ai/cli install ai-research-sentencepiece --target claude-codeInstructions
Security
Loading security scan...