Git Stars Git Stars

#tokenize Repositories

Discover 4 active open source GitHub repositories tagged with #tokenize, accumulating 5,793 total stars (averaging 1,448 stars per project). Leading this topic is micromark/micromark with 2,247 stars.

4
Total Repositories
5,793
Total Stars
1,448
Avg Stars/Repo
Programming Languages: JavaScript Rust Go

4 repositories total

1
micromark/micromark

small, safe, and great commonmark (optionally gfm, mdx) compliant markdown parser

micromark
⭐ 2,247 JavaScript 👤 micromark Organization 🕐 Updated 2026-09-26 📅 Created 2018-11-13
2
wooorm/markdown-rs

CommonMark compliant markdown parser in Rust with ASTs and extensions

wooorm
⭐ 1,530 Rust 👤 wooorm User 🕐 Updated 2025-04-23 📅 Created 2022-06-08
3
winkjs/wink-nlp

Developer friendly Natural Language Processing ✨

winkjs
⭐ 1,391 JavaScript 👤 winkjs Organization 🕐 Updated 2026-05-27 📅 Created 2018-12-15
4
alasdairforsythe/tokenmonster

Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript

alasdairforsythe
⭐ 626 Go 👤 alasdairforsythe User 🕐 Updated 2026-07-10 📅 Created 2023-05-12

Frequently Asked Questions

1

What is the most popular #tokenize repository on GitHub?

The top ranked open source #tokenize project on GitHub is micromark/micromark with 2,247 stars. small, safe, and great commonmark (optionally gfm, mdx) compliant markdown parser

2

Which programming languages are most used in #tokenize repositories?

Among the top ranked #tokenize projects on GitHub, the most common programming languages are JavaScript, Rust, Go.

3

How is the #tokenize repository ranking calculated?

We track 4 active open source repositories tagged with #tokenize (500+ stars), totaling 5,793 stars with an average of 1,448 stars per repository. Rankings update daily from the GitHub API.