Git Stars Git Stars

#attention-mechanism Repositories

Discover 50 active open source GitHub repositories tagged with #attention-mechanism, accumulating 274,514 total stars (averaging 5,490 stars per project). Leading this topic is rasbt/LLMs-from-scratch with 108,745 stars.

50
Total Repositories
274,514
Total Stars
5,490
Avg Stars/Repo
1
rasbt/LLMs-from-scratch

Implement a ChatGPT-like LLM in PyTorch from scratch, step by step

rasbt
โญ 108,745 Jupyter Notebook ๐Ÿ‘ค rasbt User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2023-07-23
2
lucidrains/vit-pytorch

Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

lucidrains
โญ 25,588 Python ๐Ÿ‘ค lucidrains User ๐Ÿ• Updated 2026-11-07 ๐Ÿ“… Created 2020-10-03
3
Dao-AILab/flash-attention

Fast and memory-efficient exact attention

Dao-AILab
โญ 25,330 Python ๐Ÿ‘ค Dao-AILab Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2022-05-19
4
nndl/nndl

Qiu Xipengโ€™s Neural Networks and Deep Learning, Second Edition and General Education Edition: E-book, Table of Contents, Learning Resources, and Errata

nndl
โญ 19,627 ๐Ÿ‘ค nndl Organization ๐Ÿ• Updated 2026-09-06 ๐Ÿ“… Created 2016-09-06
5
BlinkDL/RWKV-LM

RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.

BlinkDL
โญ 14,786 Python ๐Ÿ‘ค BlinkDL User ๐Ÿ• Updated 2026-11-04 ๐Ÿ“… Created 2021-08-08
6
facebookresearch/xformers

Hackable and optimized Transformers building blocks, supporting a composable construction.

facebookresearch
โญ 10,579 Python ๐Ÿ‘ค facebookresearch Organization ๐Ÿ• Updated 2026-11-13 ๐Ÿ“… Created 2021-10-13
7
lucidrains/x-transformers

A concise but complete full-attention transformer with a set of promising experimental features from various papers

lucidrains
โญ 5,966 Python ๐Ÿ‘ค lucidrains User ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2020-10-24
8
lucidrains/DALLE-pytorch

Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch

lucidrains
โญ 5,632 Python ๐Ÿ‘ค lucidrains User ๐Ÿ• Updated 2024-02-17 ๐Ÿ“… Created 2021-01-05
9
cmhungsteve/Awesome-Transformer-Attention

An ultimately comprehensive paper list of Vision Transformer/Attention, including papers, codes, and related websites

cmhungsteve
โญ 4,987 ๐Ÿ‘ค cmhungsteve User ๐Ÿ• Updated 2024-07-30 ๐Ÿ“… Created 2021-09-15
10
ruvnet/RuVector

RuVector provides high-performance, real-time decision-making and agent memory, self-learning AI, and a vector GNN database built in Rust.

ruvnet
โญ 4,713 Rust ๐Ÿ‘ค ruvnet User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-11-19
11
raiyanyahya/how-to-train-your-gpt

Build a modern LLM from scratch. Every line commented. Explained like we are five.

raiyanyahya
โญ 4,183 Jupyter Notebook ๐Ÿ‘ค raiyanyahya User ๐Ÿ• Updated 2026-10-26 ๐Ÿ“… Created 2026-05-03
12
MLNLP-World/LLMs-from-scratch-CN

LLMs-from-scratch project Chinese translation

MLNLP-World
โญ 3,100 Jupyter Notebook ๐Ÿ‘ค MLNLP-World Organization ๐Ÿ• Updated 2026-04-19 ๐Ÿ“… Created 2025-01-20
13
sunsmarterjie/yolov12

[NeurIPS 2025] YOLOv12: Attention-Centric Real-Time Object Detectors

sunsmarterjie
โญ 2,967 Python ๐Ÿ‘ค sunsmarterjie User ๐Ÿ• Updated 2026-10-25 ๐Ÿ“… Created 2025-02-06
14
linto-ai/whisper-timestamped

Multilingual Automatic Speech Recognition with word-level timestamps and confidence

linto-ai
โญ 2,860 Python ๐Ÿ‘ค linto-ai Organization ๐Ÿ• Updated 2026-09-28 ๐Ÿ“… Created 2023-01-13
15
philipperemy/keras-attention

Keras Attention Layer (Luong and Bahdanau scores).

philipperemy
โญ 2,809 Python ๐Ÿ‘ค philipperemy User ๐Ÿ• Updated 2026-03-12 ๐Ÿ“… Created 2017-05-23
16
aburkov/theLMbook

This is the official repository for The Hundred-Page Language Models Book by Andriy Burkov

aburkov
โญ 2,194 Jupyter Notebook ๐Ÿ‘ค aburkov User ๐Ÿ• Updated 2026-02-08 ๐Ÿ“… Created 2024-06-28
17
flybirdxx/ComfyUI-Qwen-TTS

A Simple Implementation of Qwen3-TTS's ComfyUI

flybirdxx
โญ 1,981 Python ๐Ÿ‘ค flybirdxx User ๐Ÿ• Updated 2026-11-10 ๐Ÿ“… Created 2026-01-22
18
amitshekhariitbhu/llm-internals

Learn LLM internals step by step - from tokenization to attention to inference optimization.

amitshekhariitbhu
โญ 1,774 ๐Ÿ‘ค amitshekhariitbhu User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2026-04-12
19
lucidrains/soundstorm-pytorch

Implementation of SoundStorm, Efficient Parallel Audio Generation from Google Deepmind, in Pytorch

lucidrains
โญ 1,545 Python ๐Ÿ‘ค lucidrains User ๐Ÿ• Updated 2025-04-24 ๐Ÿ“… Created 2023-05-17
20
yulunzhang/RCAN

PyTorch code for our ECCV 2018 paper "Image Super-Resolution Using Very Deep Residual Channel Attention Networks"

yulunzhang
โญ 1,497 Python ๐Ÿ‘ค yulunzhang User ๐Ÿ• Updated 2025-04-01 ๐Ÿ“… Created 2018-07-03
22
meta-pytorch/attention-gym

Helpful tools and examples for working with flex-attention

meta-pytorch
โญ 1,274 Python ๐Ÿ‘ค meta-pytorch Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-07-31
23
lucidrains/perceiver-pytorch

Implementation of Perceiver, General Perception with Iterative Attention, in Pytorch

lucidrains
โญ 1,223 Python ๐Ÿ‘ค lucidrains User ๐Ÿ• Updated 2026-10-26 ๐Ÿ“… Created 2021-03-05
24
awslabs/sockeye

Sequence-to-sequence framework with a focus on Neural Machine Translation based on PyTorch

awslabs
โญ 1,217 Python ๐Ÿ‘ค awslabs Organization ๐Ÿ• Updated 2024-10-24 ๐Ÿ“… Created 2017-06-08
25
chengtan9907/OpenSTL

OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learning

chengtan9907
โญ 1,165 Python ๐Ÿ‘ค chengtan9907 User ๐Ÿ• Updated 2026-03-01 ๐Ÿ“… Created 2022-07-27
26
SuperBruceJia/EEG-DL

A Deep Learning library for EEG Tasks (Signals) Classification, based on TensorFlow.

SuperBruceJia
โญ 1,162 Python ๐Ÿ‘ค SuperBruceJia User ๐Ÿ• Updated 2025-07-20 ๐Ÿ“… Created 2020-04-30
27
cedrickchee/awesome-transformer-nlp

A curated list of NLP resources focused on Transformer networks, attention mechanism, GPT, BERT, ChatGPT, LLMs, and transfer learning.

cedrickchee
โญ 1,124 ๐Ÿ‘ค cedrickchee User ๐Ÿ• Updated 2024-10-27 ๐Ÿ“… Created 2019-01-13
28
lucidrains/tab-transformer-pytorch

Implementation of TabTransformer, attention network for tabular data, in Pytorch

lucidrains
โญ 1,097 Python ๐Ÿ‘ค lucidrains User ๐Ÿ• Updated 2026-01-08 ๐Ÿ“… Created 2020-12-15
29
qiuzh20/gated_attention

The official implementation for [NeurIPS2025 Oral] Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free

qiuzh20
โญ 994 Jupyter Notebook ๐Ÿ‘ค qiuzh20 User ๐Ÿ• Updated 2025-12-20 ๐Ÿ“… Created 2025-05-09
30
AMLab-Amsterdam/AttentionDeepMIL

Implementation of Attention-based Deep Multiple Instance Learning in PyTorch

AMLab-Amsterdam
โญ 965 Python ๐Ÿ‘ค AMLab-Amsterdam Organization ๐Ÿ• Updated 2024-01-08 ๐Ÿ“… Created 2018-02-26
31
lucidrains/linear-attention-transformer

Transformer based on a variant of attention that is linear complexity in respect to sequence length

lucidrains
โญ 821 Python ๐Ÿ‘ค lucidrains User ๐Ÿ• Updated 2024-05-05 ๐Ÿ“… Created 2020-06-04
32
mit-han-lab/fastcomposer

[IJCV] FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention

mit-han-lab
โญ 716 Python ๐Ÿ‘ค mit-han-lab Organization ๐Ÿ• Updated 2025-01-10 ๐Ÿ“… Created 2023-05-17
33
swz30/MIRNet

[ECCV 2020] Learning Enriched Features for Real Image Restoration and Enhancement. SOTA results for image denoising, super-resolution, and image enhancement.

swz30
โญ 708 Python ๐Ÿ‘ค swz30 User ๐Ÿ• Updated 2024-04-18 ๐Ÿ“… Created 2020-03-11
34
positive666/yolo_research

based on yolo-high-level project (detect\pose\classify\segment\):include yolov5\yolov7\yolov8\ core ,improvement research ,SwintransformV2 and Attention Series. training skills, business customization, engineering deployment C

positive666
โญ 667 Python ๐Ÿ‘ค positive666 User ๐Ÿ• Updated 2026-04-27 ๐Ÿ“… Created 2021-02-25
35
Canjie-Luo/MORAN_v2

MORAN: A Multi-Object Rectified Attention Network for Scene Text Recognition

Canjie-Luo
โญ 648 Python ๐Ÿ‘ค Canjie-Luo User ๐Ÿ• Updated 2024-07-25 ๐Ÿ“… Created 2019-01-09
36
zjx0101/ObjectClear

[CVPR'26] ObjectClear: Precise Object and Effect Removal with Adaptive Target-Aware Attention

zjx0101
โญ 634 Python ๐Ÿ‘ค zjx0101 User ๐Ÿ• Updated 2026-08-03 ๐Ÿ“… Created 2025-05-28
37
therealoliver/Deepdive-llama3-from-scratch

Achieve the llama3 inference step-by-step, grasp the core concepts, master the process derivation, implement the code.

therealoliver
โญ 629 Jupyter Notebook ๐Ÿ‘ค therealoliver User ๐Ÿ• Updated 2025-02-24 ๐Ÿ“… Created 2025-02-19
38
kyegomez/zeta

Build high-performance AI models with modular building blocks

kyegomez
โญ 605 Python ๐Ÿ‘ค kyegomez User ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2023-07-09
39
bhoov/exbert

A Visual Analysis Tool to Explore Learned Representations in Transformers Models

bhoov
โญ 602 Python ๐Ÿ‘ค bhoov User ๐Ÿ• Updated 2024-02-07 ๐Ÿ“… Created 2019-10-14
40
prs-eth/OverlapPredator

[CVPR 2021, Oral] PREDATOR: Registration of 3D Point Clouds with Low Overlap.

prs-eth
โญ 599 Python ๐Ÿ‘ค prs-eth Organization ๐Ÿ• Updated 2025-02-20 ๐Ÿ“… Created 2020-11-24
41
Haiyang-W/TokenFormer

[ICLR2025 Spotlight๐Ÿ”ฅ] Official Implementation of TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters

Haiyang-W
โญ 590 Python ๐Ÿ‘ค Haiyang-W User ๐Ÿ• Updated 2025-02-11 ๐Ÿ“… Created 2024-10-29
43
lucidrains/enformer-pytorch

Implementation of Enformer, Deepmind's attention network for predicting gene expression, in Pytorch

lucidrains
โญ 576 Python ๐Ÿ‘ค lucidrains User ๐Ÿ• Updated 2026-06-26 ๐Ÿ“… Created 2021-10-23
44
philipturner/metal-flash-attention

FlashAttention (Metal Port)

philipturner
โญ 570 Swift ๐Ÿ‘ค philipturner User ๐Ÿ• Updated 2024-09-22 ๐Ÿ“… Created 2023-06-23
45
wgcban/ChangeFormer

[IGARSS'22]: A Transformer-Based Siamese Network for Change Detection

wgcban
โญ 555 Python ๐Ÿ‘ค wgcban User ๐Ÿ• Updated 2024-01-31 ๐Ÿ“… Created 2021-12-02
46
lucidrains/ring-attention-pytorch

Implementation of ๐Ÿ’ Ring Attention, from Liu et al. at Berkeley AI, in Pytorch

lucidrains
โญ 547 Python ๐Ÿ‘ค lucidrains User ๐Ÿ• Updated 2025-05-16 ๐Ÿ“… Created 2024-02-14
47
kyegomez/RT-2

Democratization of RT-2 "RT-2: New model translates vision and language into action"

kyegomez
โญ 539 Python ๐Ÿ‘ค kyegomez User ๐Ÿ• Updated 2024-07-26 ๐Ÿ“… Created 2023-07-28
48
WenjieDu/SAITS

The official PyTorch implementation of the paper "SAITS: Self-Attention-based Imputation for Time Series". A fast and state-of-the-art (SOTA) deep-learning neural network model for efficient time-series imputation (impute multivariate incomplete time series containing NaN missing data/values with machine learning). https://arxiv.org/abs/2202.08516

WenjieDu
โญ 521 Python ๐Ÿ‘ค WenjieDu User ๐Ÿ• Updated 2026-08-25 ๐Ÿ“… Created 2021-12-07
49
zhengchen1999/DAT

[ICCV'23] Dual Aggregation Transformer for Image Super-Resolution

zhengchen1999
โญ 517 Python ๐Ÿ‘ค zhengchen1999 User ๐Ÿ• Updated 2024-11-26 ๐Ÿ“… Created 2022-11-10
50
lucidrains/slot-attention

Implementation of Slot Attention from Google AI

lucidrains
โญ 502 Python ๐Ÿ‘ค lucidrains User ๐Ÿ• Updated 2026-06-06 ๐Ÿ“… Created 2020-06-29

Frequently Asked Questions

1

What is the most popular #attention-mechanism repository on GitHub?

The top ranked open source #attention-mechanism project on GitHub is rasbt/LLMs-from-scratch with 108,745 stars. Implement a ChatGPT-like LLM in PyTorch from scratch, step by step

2

Which programming languages are most used in #attention-mechanism repositories?

Among the top ranked #attention-mechanism projects on GitHub, the most common programming languages are Python, Jupyter Notebook, Rust, Swift.

3

How is the #attention-mechanism repository ranking calculated?

We track 50 active open source repositories tagged with #attention-mechanism (500+ stars), totaling 274,514 stars with an average of 5,490 stars per repository. Rankings update daily from the GitHub API.