Git Stars Git Stars

#text-tokenization Repositories

Discover 3 active open source GitHub repositories tagged with #text-tokenization, accumulating 3,189 total stars (averaging 1,063 stars per project). Leading this topic is microsoft/BlingFire with 1,877 stars.

3
Total Repositories
3,189
Total Stars
1,063
Avg Stars/Repo
Programming Languages: C++ Rust Go

3 repositories total

1
microsoft/BlingFire

A lightning fast Finite State machine and REgular expression manipulation library.

microsoft
⭐ 1,877 C++ 👤 microsoft Organization 🕐 Updated 2024-12-08 📅 Created 2019-03-13
2
unicode-rs/unicode-segmentation

Grapheme Cluster and Word boundaries according to UAX#29 rules

unicode-rs
⭐ 687 Rust 👤 unicode-rs Organization 🕐 Updated 2026-11-10 📅 Created 2015-04-13
3
alasdairforsythe/tokenmonster

Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript

alasdairforsythe
⭐ 626 Go 👤 alasdairforsythe User 🕐 Updated 2026-07-10 📅 Created 2023-05-12

Frequently Asked Questions

1

What is the most popular #text-tokenization repository on GitHub?

The top ranked open source #text-tokenization project on GitHub is microsoft/BlingFire with 1,877 stars. A lightning fast Finite State machine and REgular expression manipulation library.

2

Which programming languages are most used in #text-tokenization repositories?

Among the top ranked #text-tokenization projects on GitHub, the most common programming languages are C++, Rust, Go.

3

How is the #text-tokenization repository ranking calculated?

We track 3 active open source repositories tagged with #text-tokenization (500+ stars), totaling 3,189 stars with an average of 1,063 stars per repository. Rankings update daily from the GitHub API.