Git Stars Git Stars

#tokenization Repositories

Discover 33 active open source GitHub repositories tagged with #tokenization, accumulating 183,037 total stars (averaging 5,546 stars per project). Leading this topic is shiyu-coder/Kronos with 48,042 stars.

33
Total Repositories
183,037
Total Stars
5,546
Avg Stars/Repo
1
shiyu-coder/Kronos

Kronos: A Foundation Model for the Language of Financial Markets

shiyu-coder
โญ 48,042 Python ๐Ÿ‘ค shiyu-coder User ๐Ÿ• Updated 2026-04-13 ๐Ÿ“… Created 2025-07-01
2
explosion/spaCy

๐Ÿ’ซ Industrial-strength Natural Language Processing (NLP) in Python

explosion
โญ 34,082 Python ๐Ÿ‘ค explosion Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2014-07-03
3
toon-format/toon

๐ŸŽ’ Token-Oriented Object Notation (TOON) โ€“ compact, human-readable serialization of JSON data for LLM prompts. TypeScript SDK, CLI, benchmarks.

toon-format
โญ 25,642 TypeScript ๐Ÿ‘ค toon-format Organization ๐Ÿ• Updated 2026-11-11 ๐Ÿ“… Created 2025-10-22
4
openai/tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

openai
โญ 19,698 Python ๐Ÿ‘ค openai Organization ๐Ÿ• Updated 2026-11-08 ๐Ÿ“… Created 2022-12-01
5
doctrine/lexer

Base library for a lexer that can be used in Top-Down, Recursive Descent Parsers.

doctrine
โญ 11,141 PHP ๐Ÿ‘ค doctrine Organization ๐Ÿ• Updated 2026-11-15 ๐Ÿ“… Created 2013-01-12
6
karpathy/minbpe

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

karpathy
โญ 10,241 Python ๐Ÿ‘ค karpathy User ๐Ÿ• Updated 2024-07-01 ๐Ÿ“… Created 2024-02-16
7
openai/harmony

Renderer for the harmony response format to be used with gpt-oss

openai
โญ 4,548 Rust ๐Ÿ‘ค openai Organization ๐Ÿ• Updated 2026-04-08 ๐Ÿ“… Created 2025-07-31
8
marcelroed/gigatoken

Language model tokenization at GB/s

marcelroed
โญ 4,157 Rust ๐Ÿ‘ค marcelroed User ๐Ÿ• Updated 2026-11-02 ๐Ÿ“… Created 2025-11-10
9
AgentOps-AI/tokencost

Easy token price estimates for 400+ LLMs. TokenOps.

AgentOps-AI
โญ 2,007 Python ๐Ÿ‘ค AgentOps-AI Organization ๐Ÿ• Updated 2025-09-05 ๐Ÿ“… Created 2023-12-03
10
apple-aiml-research/ml-4m

4M: Massively Multimodal Masked Modeling

apple-aiml-research
โญ 1,814 Python ๐Ÿ‘ค apple-aiml-research Organization ๐Ÿ• Updated 2026-09-11 ๐Ÿ“… Created 2024-04-08
11
rspeer/wordfreq

Access a database of word frequencies, in various natural languages.

rspeer
โญ 1,664 Python ๐Ÿ‘ค rspeer User ๐Ÿ• Updated 2025-01-04 ๐Ÿ“… Created 2013-10-28
12
Ridera-Rwa/Ridera-core

Core protocol repository for Ridera โ€” Mobility RWA on Base. Includes docs, architecture, smart contracts, and protocol design.

Ridera-Rwa
โญ 1,593 Solidity ๐Ÿ‘ค Ridera-Rwa User ๐Ÿ• Updated 2025-12-15 ๐Ÿ“… Created 2025-11-24
13
shikijs/shiki-magic-move

Smoothly animated code blocks with Shiki

shikijs
โญ 1,511 TypeScript ๐Ÿ‘ค shikijs Organization ๐Ÿ• Updated 2026-05-19 ๐Ÿ“… Created 2024-02-22
14
securitybunker/databunker

Secure Vault for Customer PII/PHI/PCI/KYC Records

securitybunker
โญ 1,498 Go ๐Ÿ‘ค securitybunker Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2019-12-08
15
lunasec-io/lunasec

LunaSec - Dependency Security Scanner that automatically notifies you about vulnerabilities like Log4Shell or node-ipc in your Pull Requests and Builds. Protect yourself in 30 seconds with the LunaTrace GitHub App: https://github.com/marketplace/lunatrace-by-lunasec/

lunasec-io
โญ 1,461 TypeScript ๐Ÿ‘ค lunasec-io Organization ๐Ÿ• Updated 2024-05-02 ๐Ÿ“… Created 2021-03-16
16
huggingface/swift-transformers

Swift Package to implement a transformers-like API in Swift

huggingface
โญ 1,366 Swift ๐Ÿ‘ค huggingface Organization ๐Ÿ• Updated 2026-11-06 ๐Ÿ“… Created 2023-05-11
17
tensorflow/text

Making text a first-class citizen in TensorFlow.

tensorflow
โญ 1,299 C++ ๐Ÿ‘ค tensorflow Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2019-05-29
18
RavenProject/Ravencoin

Ravencoin Core integration/staging tree

RavenProject
โญ 1,122 C ๐Ÿ‘ค RavenProject Organization ๐Ÿ• Updated 2026-08-18 ๐Ÿ“… Created 2017-07-09
19
mistralai/mistral-common

Official inference library for pre-processing of Mistral models

mistralai
โญ 965 Python ๐Ÿ‘ค mistralai Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2024-04-15
20
servo/rust-cssparser

Rust implementation of CSS Syntax Level 3

servo
โญ 875 Rust ๐Ÿ‘ค servo Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2013-07-10
21
KennFalcon/elasticsearch-analysis-hanlp

HanLP Analyzer for Elasticsearch

KennFalcon
โญ 854 Java ๐Ÿ‘ค KennFalcon User ๐Ÿ• Updated 2024-07-12 ๐Ÿ“… Created 2017-03-04
22
explosion/spacy-streamlit

๐Ÿ‘‘ spaCy building blocks and visualizers for Streamlit apps

explosion
โญ 849 Python ๐Ÿ‘ค explosion Organization ๐Ÿ• Updated 2024-07-29 ๐Ÿ“… Created 2020-06-23
23
nlp-uoregon/trankit

Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing

nlp-uoregon
โญ 795 Python ๐Ÿ‘ค nlp-uoregon User ๐Ÿ• Updated 2025-07-22 ๐Ÿ“… Created 2021-01-08
24
angeloskath/php-nlp-tools

Natural Language Processing Tools in PHP

angeloskath
โญ 762 PHP ๐Ÿ‘ค angeloskath User ๐Ÿ• Updated 2024-07-22 ๐Ÿ“… Created 2012-08-22
25
cbaziotis/ekphrasis

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

cbaziotis
โญ 676 Python ๐Ÿ‘ค cbaziotis User ๐Ÿ• Updated 2025-06-02 ๐Ÿ“… Created 2017-02-07
26
alasdairforsythe/tokenmonster

Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript

alasdairforsythe
โญ 626 Go ๐Ÿ‘ค alasdairforsythe User ๐Ÿ• Updated 2026-07-10 ๐Ÿ“… Created 2023-05-12
27
SamuraiT/mecab-python3

:snake: mecab-python. you can find original version here:http://taku910.github.io/mecab/

SamuraiT
โญ 579 C++ ๐Ÿ‘ค SamuraiT User ๐Ÿ• Updated 2025-11-25 ๐Ÿ“… Created 2014-05-31
28
adobe/NLP-Cube

Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing

adobe
โญ 561 HTML ๐Ÿ‘ค adobe Organization ๐Ÿ• Updated 2024-11-03 ๐Ÿ“… Created 2018-03-13
29
cl-tohoku/bert-japanese

BERT models for Japanese text.

cl-tohoku
โญ 540 Python ๐Ÿ‘ค cl-tohoku Organization ๐Ÿ• Updated 2024-03-23 ๐Ÿ“… Created 2019-11-14
30
xingchensong/S3Tokenizer

Reverse Engineering of Supervised Semantic Speech Tokenizer (S3Tokenizer) proposed in CosyVoice

xingchensong
โญ 536 Python ๐Ÿ‘ค xingchensong User ๐Ÿ• Updated 2026-11-01 ๐Ÿ“… Created 2024-09-09
31
yooper/php-text-analysis

PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language

yooper
โญ 531 PHP ๐Ÿ‘ค yooper User ๐Ÿ• Updated 2024-12-28 ๐Ÿ“… Created 2012-05-21
32
lightninglabs/taproot-assets

A layer 1 daemon, for the Taproot Assets Protocol specification, written in Go (golang)

lightninglabs
โญ 529 Go ๐Ÿ‘ค lightninglabs Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2022-04-11
33
patrikzudel/PatrikZeros-ChatGPT-API-UI

Static webpage that allows you to use your OpenAI API key for the same experience as you get with ChatGPT!

patrikzudel
โญ 508 Svelte ๐Ÿ‘ค patrikzudel User ๐Ÿ• Updated 2024-01-08 ๐Ÿ“… Created 2023-03-07

Frequently Asked Questions

1

What is the most popular #tokenization repository on GitHub?

The top ranked open source #tokenization project on GitHub is shiyu-coder/Kronos with 48,042 stars. Kronos: A Foundation Model for the Language of Financial Markets

2

Which programming languages are most used in #tokenization repositories?

Among the top ranked #tokenization projects on GitHub, the most common programming languages are Python, TypeScript, PHP, Rust.

3

How is the #tokenization repository ranking calculated?

We track 33 active open source repositories tagged with #tokenization (500+ stars), totaling 183,037 stars with an average of 5,546 stars per repository. Rankings update daily from the GitHub API.