#tokenizer Repositories
Discover 29 active open source GitHub repositories tagged with #tokenizer, accumulating 145,054 total stars (averaging 5,001 stars per project). Leading this topic is rasbt/LLMs-from-scratch with 108,745 stars.
29 repositories total
A small library for converting tokenized PHP source code into XML (and potentially other formats)
A micro-library of stream components for building custom JSON and JSONC processing pipelines with a minimal memory footprint โ parse, filter, and transform JSON far larger than available memory with a SAX-inspired token API, on Node.js or Web Streams.
This is a Python library for Korean natural language processing. It provides functions for word extraction / tokenizer / POS tagging / preprocessing.
SQLite fts5 full-text search extension supporting Chinese and PinYin | A SQLite3 fts5 tokenizer which supports Chinese and PinYin
The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.
An Integrated Corpus Tool With Multilingual Support for the Study of Language, Literature, and Translation
Data Labeling is a specialized tool for processing and labeling text data. By simplifying quick text labeling processes and dynamic algorithm feedback, it supports users in quickly labeling keywords and can continuously reduce the cost and time of manual labeling through algorithms. The data labeling process first builds the foundation through manual labeling, then automatic labeling feeds back to manual labeling, and finally manual labeling is used for correction, thereby greatly improving the accuracy and efficiency of labeling. Data Labeling requires relying on an open-source digital base for personnel position management.
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
Open Korean Text Processor - An Open-source Korean Text Processor
Achieve the llama3 inference step-by-step, grasp the core concepts, master the process derivation, implement the code.
Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript
[NeurIPS 2025 Spotlight] A Unified Tokenizer for Visual Generation and Understanding
A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.
Frequently Asked Questions
What is the most popular #tokenizer repository on GitHub?
The top ranked open source #tokenizer project on GitHub is rasbt/LLMs-from-scratch with 108,745 stars. Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
Which programming languages are most used in #tokenizer repositories?
Among the top ranked #tokenizer projects on GitHub, the most common programming languages are Python, TypeScript, JavaScript, C++.
How is the #tokenizer repository ranking calculated?
We track 29 active open source repositories tagged with #tokenizer (500+ stars), totaling 145,054 stars with an average of 5,001 stars per repository. Rankings update daily from the GitHub API.