Git Stars Git Stars

#tokenizer Repositories

Discover 29 active open source GitHub repositories tagged with #tokenizer, accumulating 145,054 total stars (averaging 5,001 stars per project). Leading this topic is rasbt/LLMs-from-scratch with 108,745 stars.

29
Total Repositories
145,054
Total Stars
5,001
Avg Stars/Repo
1
rasbt/LLMs-from-scratch

Implement a ChatGPT-like LLM in PyTorch from scratch, step by step

rasbt
โญ 108,745 Jupyter Notebook ๐Ÿ‘ค rasbt User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2023-07-23
2
theseer/tokenizer

A small library for converting tokenized PHP source code into XML (and potentially other formats)

theseer
โญ 5,190 PHP ๐Ÿ‘ค theseer User ๐Ÿ• Updated 2026-02-03 ๐Ÿ“… Created 2017-04-05
3
marcelroed/gigatoken

Language model tokenization at GB/s

marcelroed
โญ 4,157 Rust ๐Ÿ‘ค marcelroed User ๐Ÿ• Updated 2026-11-02 ๐Ÿ“… Created 2025-11-10
4
andialbrecht/sqlparse

A non-validating SQL parser module for Python

andialbrecht
โญ 4,023 Python ๐Ÿ‘ค andialbrecht User ๐Ÿ• Updated 2026-11-01 ๐Ÿ“… Created 2012-04-18
5
Chevrotain/chevrotain

Parser Building Toolkit for JavaScript

Chevrotain
โญ 2,820 TypeScript ๐Ÿ‘ค Chevrotain Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2015-04-15
6
dqbd/tiktokenizer

Online playground for OpenAPI tokenizers

dqbd
โญ 1,605 TypeScript ๐Ÿ‘ค dqbd User ๐Ÿ• Updated 2025-04-24 ๐Ÿ“… Created 2023-03-02
7
roshan-research/hazm

Persian NLP Toolkit

roshan-research
โญ 1,435 Python ๐Ÿ‘ค roshan-research Organization ๐Ÿ• Updated 2026-04-01 ๐Ÿ“… Created 2013-10-29
8
natasha/natasha

Solves basic Russian NLP tasks, API for lower level Natasha projects

natasha
โญ 1,364 Python ๐Ÿ‘ค natasha Organization ๐Ÿ• Updated 2026-10-14 ๐Ÿ“… Created 2016-08-03
9
datalust/superpower

A C# parser construction toolkit with high-quality error reporting

datalust
โญ 1,345 C# ๐Ÿ‘ค datalust Organization ๐Ÿ• Updated 2026-05-20 ๐Ÿ“… Created 2016-09-01
10
uhop/stream-json

A micro-library of stream components for building custom JSON and JSONC processing pipelines with a minimal memory footprint โ€” parse, filter, and transform JSON far larger than available memory with a SAX-inspired token API, on Node.js or Web Streams.

uhop
โญ 1,209 JavaScript ๐Ÿ‘ค uhop User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2013-08-11
11
lovit/soynlp

This is a Python library for Korean natural language processing. It provides functions for word extraction / tokenizer / POS tagging / preprocessing.

lovit
โญ 996 Python ๐Ÿ‘ค lovit User ๐Ÿ• Updated 2026-10-04 ๐Ÿ“… Created 2017-05-13
12
ikawaha/kagome

Self-contained Japanese Morphological Analyzer written in pure Go

ikawaha
โญ 988 Go ๐Ÿ‘ค ikawaha User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2014-06-26
13
no-context/moo

Optimised tokenizer/lexer generator! ๐Ÿ„ Uses /y for performance. Moo.

no-context
โญ 884 JavaScript ๐Ÿ‘ค no-context Organization ๐Ÿ• Updated 2026-02-27 ๐Ÿ“… Created 2017-03-10
14
wangfenjin/simple

SQLite fts5 full-text search extension supporting Chinese and PinYin | A SQLite3 fts5 tokenizer which supports Chinese and PinYin

wangfenjin
โญ 877 C++ ๐Ÿ‘ค wangfenjin User ๐Ÿ• Updated 2026-05-17 ๐Ÿ“… Created 2020-03-03
15
niieani/gpt-tokenizer

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.

niieani
โญ 862 TypeScript ๐Ÿ‘ค niieani User ๐Ÿ• Updated 2026-11-10 ๐Ÿ“… Created 2023-03-22
16
BLKSerene/Wordless

An Integrated Corpus Tool With Multilingual Support for the Study of Language, Literature, and Translation

BLKSerene
โญ 762 Python ๐Ÿ‘ค BLKSerene User ๐Ÿ• Updated 2025-10-30 ๐Ÿ“… Created 2018-08-23
17
lindera/lindera

A multilingual morphological analysis library.

lindera
โญ 708 Rust ๐Ÿ‘ค lindera Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2020-01-22
18
risesoft-y9/Data-Labeling

Data Labeling is a specialized tool for processing and labeling text data. By simplifying quick text labeling processes and dynamic algorithm feedback, it supports users in quickly labeling keywords and can continuously reduce the cost and time of manual labeling through algorithms. The data labeling process first builds the foundation through manual labeling, then automatic labeling feeds back to manual labeling, and finally manual labeling is used for correction, thereby greatly improving the accuracy and efficiency of labeling. Data Labeling requires relying on an open-source digital base for personnel position management.

risesoft-y9
โญ 695 Java ๐Ÿ‘ค risesoft-y9 Organization ๐Ÿ• Updated 2025-06-23 ๐Ÿ“… Created 2024-08-26
19
cbaziotis/ekphrasis

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

cbaziotis
โญ 676 Python ๐Ÿ‘ค cbaziotis User ๐Ÿ• Updated 2025-06-02 ๐Ÿ“… Created 2017-02-07
20
open-korean-text/open-korean-text

Open Korean Text Processor - An Open-source Korean Text Processor

open-korean-text
โญ 651 Scala ๐Ÿ‘ค open-korean-text Organization ๐Ÿ• Updated 2024-03-12 ๐Ÿ“… Created 2017-01-24
21
jflex-de/jflex

The fast scanner generator for Javaโ„ข with full Unicode support

jflex-de
โญ 637 Java ๐Ÿ‘ค jflex-de Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2015-02-15
22
therealoliver/Deepdive-llama3-from-scratch

Achieve the llama3 inference step-by-step, grasp the core concepts, master the process derivation, implement the code.

therealoliver
โญ 629 Jupyter Notebook ๐Ÿ‘ค therealoliver User ๐Ÿ• Updated 2025-02-24 ๐Ÿ“… Created 2025-02-19
23
alasdairforsythe/tokenmonster

Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript

alasdairforsythe
โญ 626 Go ๐Ÿ‘ค alasdairforsythe User ๐Ÿ• Updated 2026-07-10 ๐Ÿ“… Created 2023-05-12
24
glayzzle/php-parser

NodeJS PHP Parser - extract AST or tokens

glayzzle
โญ 564 JavaScript ๐Ÿ‘ค glayzzle Organization ๐Ÿ• Updated 2026-10-30 ๐Ÿ“… Created 2014-12-07
25
lydell/js-tokens

Tiny JavaScript tokenizer.

lydell
โญ 557 JavaScript ๐Ÿ‘ค lydell User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2014-03-08
26
FoundationVision/UniTok

[NeurIPS 2025 Spotlight] A Unified Tokenizer for Visual Generation and Understanding

FoundationVision
โญ 537 Python ๐Ÿ‘ค FoundationVision Organization ๐Ÿ• Updated 2025-11-14 ๐Ÿ“… Created 2025-02-27
27
polm/fugashi

A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.

polm
โญ 536 C++ ๐Ÿ‘ค polm User ๐Ÿ• Updated 2025-10-24 ๐Ÿ“… Created 2019-10-14
28
leodevbro/vscode-blockman

VSCode extension to highlight nested code blocks

leodevbro
โญ 514 TypeScript ๐Ÿ‘ค leodevbro User ๐Ÿ• Updated 2026-11-07 ๐Ÿ“… Created 2021-05-12
29
NLPOptimize/flash-tokenizer

EFFICIENT AND OPTIMIZED TOKENIZER ENGINE FOR LLM INFERENCE SERVING

NLPOptimize
โญ 509 C++ ๐Ÿ‘ค NLPOptimize Organization ๐Ÿ• Updated 2026-02-02 ๐Ÿ“… Created 2025-03-09

Frequently Asked Questions

1

What is the most popular #tokenizer repository on GitHub?

The top ranked open source #tokenizer project on GitHub is rasbt/LLMs-from-scratch with 108,745 stars. Implement a ChatGPT-like LLM in PyTorch from scratch, step by step

2

Which programming languages are most used in #tokenizer repositories?

Among the top ranked #tokenizer projects on GitHub, the most common programming languages are Python, TypeScript, JavaScript, C++.

3

How is the #tokenizer repository ranking calculated?

We track 29 active open source repositories tagged with #tokenizer (500+ stars), totaling 145,054 stars with an average of 5,001 stars per repository. Rankings update daily from the GitHub API.