Git Stars Git Stars

#flash-attention Repositories

Discover 48 active open source GitHub repositories tagged with #flash-attention, accumulating 156,291 total stars (averaging 3,256 stars per project). Leading this topic is deepseek-ai/DeepSeek-OCR with 24,180 stars.

48
Total Repositories
156,291
Total Stars
3,256
Avg Stars/Repo
1
deepseek-ai/DeepSeek-OCR

Contexts Optical Compression

deepseek-ai
โญ 24,180 Python ๐Ÿ‘ค deepseek-ai Organization ๐Ÿ• Updated 2026-01-27 ๐Ÿ“… Created 2025-10-17
2
QwenLM/Qwen

The official repo of Qwen (้€šไน‰ๅƒ้—ฎ) chat & pretrained large language model proposed by Alibaba Cloud.

QwenLM
โญ 22,063 Python ๐Ÿ‘ค QwenLM Organization ๐Ÿ• Updated 2026-03-05 ๐Ÿ“… Created 2023-08-03
3
xlite-dev/LeetCUDA

An Open-source book with Modern CUDA learning notes for beginners โ€” BF16/FP8/FP4, HGEMM, CuTe, Flash-Attention, etc.

xlite-dev
โญ 12,208 TeX ๐Ÿ‘ค xlite-dev Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2022-12-17
โญ 9,620 Python ๐Ÿ‘ค meituan-longcat Organization ๐Ÿ• Updated 2026-11-15 ๐Ÿ“… Created 2025-10-25
5
InternLM/InternLM

Official release of InternLM series (InternLM, InternLM2, InternLM2.5, InternLM3).

InternLM
โญ 7,280 Python ๐Ÿ‘ค InternLM Organization ๐Ÿ• Updated 2025-10-30 ๐Ÿ“… Created 2023-07-06
6
ymcui/Chinese-LLaMA-Alpaca-2

Chinese LLaMA-2 & Alpaca-2 LLMs Phase II + 64K long context models (Chinese LLaMA-2 & Alpaca-2 LLMs with 64K long context models)

ymcui
โญ 7,098 Python ๐Ÿ‘ค ymcui User ๐Ÿ• Updated 2026-04-19 ๐Ÿ“… Created 2023-07-18
7
karminski/one-small-step

This is a simple technical็ง‘ๆ™ฎ tutorial project, mainly focusing on explaining some interesting, cutting-edge technical concepts and principles. Each article strives to be completed within 5 minutes of reading.

karminski
โญ 7,006 Python ๐Ÿ‘ค karminski User ๐Ÿ• Updated 2026-03-08 ๐Ÿ“… Created 2025-01-16
8
KellerJordan/modded-nanogpt

NanoGPT (124M) in 90 seconds

KellerJordan
โญ 6,058 Python ๐Ÿ‘ค KellerJordan User ๐Ÿ• Updated 2026-09-28 ๐Ÿ“… Created 2024-06-01
9
xlite-dev/Awesome-LLM-Inference

๐Ÿ“šA curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.๐ŸŽ‰

xlite-dev
โญ 5,565 Python ๐Ÿ‘ค xlite-dev Organization ๐Ÿ• Updated 2026-10-20 ๐Ÿ“… Created 2023-08-27
10
turboderp-org/exllamav2

A fast inference library for running LLMs locally on modern consumer-class GPUs

turboderp-org
โญ 4,653 Python ๐Ÿ‘ค turboderp-org Organization ๐Ÿ• Updated 2026-03-04 ๐Ÿ“… Created 2023-08-30
11
QwenLM/Qwen3-ASR

Qwen3-ASR is an open-source series of ASR models developed by the Qwen team at Alibaba Cloud, supporting stable multilingual speech/music/song recognition, language detection and timestamp prediction.

QwenLM
โญ 3,933 Python ๐Ÿ‘ค QwenLM Organization ๐Ÿ• Updated 2026-06-26 ๐Ÿ“… Created 2026-01-28
12
HazyResearch/ThunderKittens

Tile primitives for speedy kernels

HazyResearch
โญ 3,805 Cuda ๐Ÿ‘ค HazyResearch Organization ๐Ÿ• Updated 2026-09-12 ๐Ÿ“… Created 2024-03-04
13
guandeh17/Self-Forcing

Official codebase for "Self Forcing: Bridging Training and Inference in Autoregressive Video Diffusion" (NeurIPS 2025 Spotlight)

guandeh17
โญ 3,511 Python ๐Ÿ‘ค guandeh17 User ๐Ÿ• Updated 2025-09-12 ๐Ÿ“… Created 2025-06-09
14
ikawrakow/ik_llama.cpp

llama.cpp fork with additional SOTA quants and improved performance

ikawrakow
โญ 3,411 C++ ๐Ÿ‘ค ikawrakow User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-06-27
โญ 2,509 Python ๐Ÿ‘ค HiDream-ai Organization ๐Ÿ• Updated 2025-07-16 ๐Ÿ“… Created 2025-04-06
16
MoonshotAI/MoBA

MoBA: Mixture of Block Attention for Long-Context LLMs

MoonshotAI
โญ 2,122 Python ๐Ÿ‘ค MoonshotAI Organization ๐Ÿ• Updated 2025-04-03 ๐Ÿ“… Created 2025-02-17
17
Ucas-HaoranWei/Vary

[ECCV 2024] Official code implementation of Vary: Scaling Up the Vision Vocabulary of Large Vision Language Models.

Ucas-HaoranWei
โญ 1,888 Python ๐Ÿ‘ค Ucas-HaoranWei User ๐Ÿ• Updated 2024-12-30 ๐Ÿ“… Created 2023-12-07
โญ 1,850 Python ๐Ÿ‘ค Tele-AI Organization ๐Ÿ• Updated 2026-07-09 ๐Ÿ“… Created 2024-01-07
19
allenai/Olmo-core

PyTorch building blocks for the OLMo ecosystem

allenai
โญ 1,810 Python ๐Ÿ‘ค allenai Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-03-23
20
mjun0812/flash-attention-prebuild-wheels

Provide with pre-build flash-attention 2 and 3 package wheels on Linux and Windows using GitHub Actions

mjun0812
โญ 1,798 Python ๐Ÿ‘ค mjun0812 User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-10-26
21
Soul-AILab/SoulX-FlashTalk

SoulX-FlashTalk is the first 14B model to achieve sub-second start-up latency (0.87s) while maintaining a real-time throughput of 32 FPS on an 8xH800 node.

Soul-AILab
โญ 1,572 Python ๐Ÿ‘ค Soul-AILab Organization ๐Ÿ• Updated 2026-07-30 ๐Ÿ“… Created 2025-12-25
โญ 1,472 Python ๐Ÿ‘ค Om-Alve User ๐Ÿ• Updated 2025-02-15 ๐Ÿ“… Created 2025-01-12
23
X-Square-Robot/wall-x

Building General-Purpose Robots Based on Embodied Foundation Model

X-Square-Robot
โญ 1,337 Python ๐Ÿ‘ค X-Square-Robot Organization ๐Ÿ• Updated 2026-09-18 ๐Ÿ“… Created 2025-09-04
24
weiruihhh/cs336_note_and_hw

Record my notes and assignments while studying CS336.

weiruihhh
โญ 1,305 Python ๐Ÿ‘ค weiruihhh User ๐Ÿ• Updated 2026-10-10 ๐Ÿ“… Created 2025-06-01
25
zhuzilin/ring-flash-attention

Ring attention implementation with flash attention

zhuzilin
โญ 1,054 Python ๐Ÿ‘ค zhuzilin User ๐Ÿ• Updated 2025-09-10 ๐Ÿ“… Created 2024-02-21
26
tspeterkim/flash-attention-minimal

Flash Attention in ~100 lines of CUDA (forward pass only)

tspeterkim
โญ 1,036 Cuda ๐Ÿ‘ค tspeterkim User ๐Ÿ• Updated 2024-12-30 ๐Ÿ“… Created 2024-03-07
27
NVIDIA/cudnn-frontend

cuDNN Frontend is NVIDIA's modern, open-source entry point to the cuDNN library and a growing collection of high-performance open-source kernels.

NVIDIA
โญ 1,008 Cuda ๐Ÿ‘ค NVIDIA Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2021-01-22
28
SandAI-org/MagiAttention

A Distributed Attention Towards Linear Scalability for Ultra-Long Context, Heterogeneous Data Training

SandAI-org
โญ 975 Python ๐Ÿ‘ค SandAI-org Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-04-20
29
Alpha-VLLM/Lumina-Image-2.0

Lumina-Image 2.0: A Unified and Efficient Image Generative Framework

Alpha-VLLM
โญ 904 Python ๐Ÿ‘ค Alpha-VLLM Organization ๐Ÿ• Updated 2026-05-22 ๐Ÿ“… Created 2025-01-25
30
TRI-ML/prismatic-vlms

A flexible and efficient codebase for training visually-conditioned language models (VLMs)

TRI-ML
โญ 894 Python ๐Ÿ‘ค TRI-ML Organization ๐Ÿ• Updated 2024-07-04 ๐Ÿ“… Created 2024-02-13
31
RUCAIBox/LLMBox

A comprehensive library for implementing LLMs, including a unified training pipeline and comprehensive model evaluation.

RUCAIBox
โญ 849 Python ๐Ÿ‘ค RUCAIBox Organization ๐Ÿ• Updated 2025-06-16 ๐Ÿ“… Created 2023-10-25
32
NVlabs/rcm

rCM & Causal-rCM: Leading and Unified Algorithms/Infrastructures for Bidirectional/Autoregressive Video Diffusion Distillation at Scale

NVlabs
โญ 832 Python ๐Ÿ‘ค NVlabs Organization ๐Ÿ• Updated 2026-06-25 ๐Ÿ“… Created 2025-10-09
โญ 789 Python ๐Ÿ‘ค HiDream-ai Organization ๐Ÿ• Updated 2025-07-17 ๐Ÿ“… Created 2025-04-14
35
exiawsh/StreamPETR

[ICCV 2023] StreamPETR: Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection

exiawsh
โญ 754 Python ๐Ÿ‘ค exiawsh User ๐Ÿ• Updated 2024-06-26 ๐Ÿ“… Created 2023-03-21
โญ 752 Python ๐Ÿ‘ค Open-Magic-Video Organization ๐Ÿ• Updated 2025-02-18 ๐Ÿ“… Created 2025-02-10
37
jzhang38/EasyContext

Memory optimization and training recipes to extrapolate language models' context length to 1 million tokens, with minimal hardware.

jzhang38
โญ 751 Python ๐Ÿ‘ค jzhang38 User ๐Ÿ• Updated 2024-09-27 ๐Ÿ“… Created 2024-04-05
38
HKUSTDial/flash-sparse-attention

Trainable fast and memory-efficient sparse attention

HKUSTDial
โญ 750 Python ๐Ÿ‘ค HKUSTDial Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2025-05-12
โญ 715 Python ๐Ÿ‘ค SkunkworksAI Organization ๐Ÿ• Updated 2024-03-06 ๐Ÿ“… Created 2023-10-10
40
yihao-meng/HoloCine

[CVPR 2026 Highlight] Official Implementations for Paper - HoloCine: Holistic Generation of Cinematic Multi-Shot Long Video Narratives

yihao-meng
โญ 707 Python ๐Ÿ‘ค yihao-meng User ๐Ÿ• Updated 2025-11-26 ๐Ÿ“… Created 2025-10-22
41
penghao-wu/vstar

PyTorch Implementation of "V* : Guided Visual Search as a Core Mechanism in Multimodal LLMs"

penghao-wu
โญ 684 Python ๐Ÿ‘ค penghao-wu User ๐Ÿ• Updated 2024-01-07 ๐Ÿ“… Created 2023-12-15
42
louaaron/Score-Entropy-Discrete-Diffusion

[ICML 2024 Best Paper] Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution (https://arxiv.org/abs/2310.16834)

louaaron
โญ 681 Python ๐Ÿ‘ค louaaron User ๐Ÿ• Updated 2024-02-29 ๐Ÿ“… Created 2024-02-21
43
Tencent/WeDLM

WeDLM: The fastest diffusion language model with standard causal attention and native KV cache compatibility, delivering real speedups over vLLM-optimized baselines.

Tencent
โญ 659 Python ๐Ÿ‘ค Tencent Organization ๐Ÿ• Updated 2026-03-03 ๐Ÿ“… Created 2025-12-24
44
zengxiao-he/tessera

From teacher to tiles โ€” a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.

zengxiao-he
โญ 556 Python ๐Ÿ‘ค zengxiao-he User ๐Ÿ• Updated 2026-10-09 ๐Ÿ“… Created 2026-06-05
โญ 549 Python ๐Ÿ‘ค rwitten User ๐Ÿ• Updated 2024-07-11 ๐Ÿ“… Created 2024-01-28
46
mit-han-lab/duo-attention

[ICLR 2025] DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads

mit-han-lab
โญ 541 Python ๐Ÿ‘ค mit-han-lab Organization ๐Ÿ• Updated 2025-02-10 ๐Ÿ“… Created 2024-10-15
47
DachengLi1/LongChat

Official repository for LongChat and LongEval

DachengLi1
โญ 533 Python ๐Ÿ‘ค DachengLi1 User ๐Ÿ• Updated 2024-05-24 ๐Ÿ“… Created 2023-04-21
48
bofenghuang/vigogne

French instruction-following and chat models

bofenghuang
โญ 506 Jupyter Notebook ๐Ÿ‘ค bofenghuang User ๐Ÿ• Updated 2024-12-05 ๐Ÿ“… Created 2023-03-24

Frequently Asked Questions

1

What is the most popular #flash-attention repository on GitHub?

The top ranked open source #flash-attention project on GitHub is deepseek-ai/DeepSeek-OCR with 24,180 stars. Contexts Optical Compression

2

Which programming languages are most used in #flash-attention repositories?

Among the top ranked #flash-attention projects on GitHub, the most common programming languages are Python, Cuda, TeX, C++.

3

How is the #flash-attention repository ranking calculated?

We track 48 active open source repositories tagged with #flash-attention (500+ stars), totaling 156,291 stars with an average of 3,256 stars per repository. Rankings update daily from the GitHub API.