#tokenization Repositories
Discover 33 active open source GitHub repositories tagged with #tokenization, accumulating 183,037 total stars (averaging 5,546 stars per project). Leading this topic is shiyu-coder/Kronos with 48,042 stars.
33 repositories total
๐ Token-Oriented Object Notation (TOON) โ compact, human-readable serialization of JSON data for LLM prompts. TypeScript SDK, CLI, benchmarks.
Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
Core protocol repository for Ridera โ Mobility RWA on Base. Includes docs, architecture, smart contracts, and protocol design.
LunaSec - Dependency Security Scanner that automatically notifies you about vulnerabilities like Log4Shell or node-ipc in your Pull Requests and Builds. Protect yourself in 30 seconds with the LunaTrace GitHub App: https://github.com/marketplace/lunatrace-by-lunasec/
Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript
:snake: mecab-python. you can find original version here:http://taku910.github.io/mecab/
Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing
Reverse Engineering of Supervised Semantic Speech Tokenizer (S3Tokenizer) proposed in CosyVoice
PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language
A layer 1 daemon, for the Taproot Assets Protocol specification, written in Go (golang)
Static webpage that allows you to use your OpenAI API key for the same experience as you get with ChatGPT!
Frequently Asked Questions
What is the most popular #tokenization repository on GitHub?
The top ranked open source #tokenization project on GitHub is shiyu-coder/Kronos with 48,042 stars. Kronos: A Foundation Model for the Language of Financial Markets
Which programming languages are most used in #tokenization repositories?
Among the top ranked #tokenization projects on GitHub, the most common programming languages are Python, TypeScript, PHP, Rust.
How is the #tokenization repository ranking calculated?
We track 33 active open source repositories tagged with #tokenization (500+ stars), totaling 183,037 stars with an average of 5,546 stars per repository. Rankings update daily from the GitHub API.