MiniMind-3 (64M): Train Your Own LLM from Scratch in Native PyTorch
Breaking down jingyaogong/minimind (67.5K stars): 64M Dense vs. 198M-A64M MoE specs, what the 2-hour RTX 3090 claim actually covers, and the full Pretrain-to-Agentic RL pipeline.

Q. Can MiniMind really train a custom LLM from scratch in 2 hours on a single consumer GPU? Yes, with one important caveat. The "2 hours and 3 RMB (~$0.43)" benchmark in the official README measures 1 epoch of SFT (
sft_t2t_mini) onminimind-3(64M) using a single NVIDIA RTX 3090, not the entire multi-stage pipeline. Even if you run Pretrain, SFT, ToolCall, and RLAIF end to end, each stage takes roughly 1 to 1.5 hours, making it practical for an individual developer to inspect every line of code and finish a full run.
⚠️ Accuracy guardrail: This guide was written on November 20, 2026 by directly inspecting the jingyaogong/minimind GitHub repository (67,494 Stars, Apache-2.0) and the November 18 Toris Studio GitHub Trending Top 10 video. GPU rental estimates (~1.3 RMB/hour) and training durations are empirical single-RTX 3090 figures reported by the repository and vary with hardware and dataset size.
30-second summary
- Zero high-level wrapper dependencies: Built from scratch in native PyTorch without relying on
transformers,trl, orpeftabstractions—covering tokenizer training, Dense/MoE models, LoRA, DPO, PPO, GRPO, CISPO, and Agentic RL. - Ultra-lightweight
minimind-3specs: Ships a 64M Dense model (d_model=768,n_layers=8,q_heads=8,kv_heads=4, RoPE + YaRN 32K) and a 198M-A64M MoE variant (4 experts / top-1), aligned with theQwen3ecosystem. - Stage-by-stage cost on 1x RTX 3090: With the
minidatasets, Pretrain takes ~1.21h (~1.57 RMB), Full SFT ~1.10h (~1.43 RMB), ToolCall ~0.90h (~1.17 RMB), and RLAIF ~1.10h (~1.43 RMB). - Drop-in local inference: Trained weights run directly via
ollama run jingyaogong/minimind-3,vLLM,llama.cpp, an OpenAI-compatible server (serve_openai_api.py), and a Streamlit WebUI.
Why MiniMind avoids transformers and peft
MiniMind is designed not just to ship a tiny checkpoint, but to expose the entire mechanics of building an LLM from scratch without hiding the math behind framework wrappers.
In typical fine-tuning workflows, transformers, peft, and trl let you train a model in a dozen lines of code. That convenience is great for production, yet it leaves attention masking, RoPE rotary embeddings, LoRA adapter injection, and RL policy gradients inside a black box. jingyaogong/minimind removes those abstractions by implementing the core model (model_minimind.py) and every training loop (trainer/) directly in native PyTorch.

Released on April 1, 2026, minimind-3 also made a practical architectural shift: it aligned its parameter naming and decoder structure with Qwen3 / Qwen3-MoE. As a result, weights you train from scratch can be exported to standard inference engines such as llama.cpp, vLLM, and Ollama without custom glue code.
Model lineup and architecture specs
The current main branch centers on two configurations: minimind-3 (64M Dense) and minimind-3-moe (198M-A64M).
| Model | Params | Vocab size | Max context | Layers / d_model |
Attention heads | Notes |
|---|---|---|---|---|---|---|
minimind-3 |
64M | 6,400 | 32,768 (YaRN) | 8 / 768 | Q=8, KV=4 | Main Dense model |
minimind-3-moe |
198M-A64M | 6,400 | 32,768 (YaRN) | 8 / 768 | Q=8, KV=4 | 4 experts / top-1 |
minimind2 |
104M | 6,400 | 32,768 (YaRN) | 16 / 768 | Q=8, KV=2 | 2025.04 legacy |
minimind2-small |
26M | 6,400 | 32,768 (YaRN) | 8 / 512 | Q=8, KV=2 | 2025.04 legacy |
One notable engineering trade-off appears in minimind-3-moe. While MoE speeds up inference via sparse activation, pure PyTorch training without fused Triton or DeepSpeed-MoE kernels incurs heavy token-dispatch scheduling overhead as expert counts grow. To preserve readable native PyTorch code, the author removed shared experts and settled on 4 experts / top-1 routing, keeping MoE training roughly 50% slower than the Dense baseline.
What '2 hours and 3 RMB' actually covers
Taking "train an LLM in 2 hours" at face value can mislead beginners into thinking dataset download through RL alignment all finish inside 120 minutes. The official README footnote and benchmark table spell out the exact scope.

| Model | Pretrain (mini, 1.2GB) |
Full SFT (mini, 1.6GB) |
ToolCall | RLAIF (PPO/GRPO) |
|---|---|---|---|---|
minimind-3 (64M) |
≈ 1.21h (≈ 1.57 RMB) | ≈ 1.10h (≈ 1.43 RMB) | ≈ 0.90h (≈ 1.17 RMB) | ≈ 1.10h (≈ 1.43 RMB) |
minimind-3-moe (198M-A64M) |
≈ 1.69h (≈ 2.20 RMB) | ≈ 1.54h (≈ 2.00 RMB) | ≈ 1.26h (≈ 1.64 RMB) | ≈ 1.54h (≈ 2.00 RMB) |
Basis: Single NVIDIA RTX 3090 (24GB), estimated rental cost ~1.3 RMB/h (~7 RMB ≈ $1 USD), 1 epoch on
pretrain_t2t_mini.jsonlandsft_t2t_mini.jsonl.
In short, Pretrain(mini) and SFT(mini) each take around 1 to 1.2 hours—roughly 2.3 hours (~3 RMB) combined for a basic reproducible chat model (MiniMind Zero). Training on the full pretrain_t2t.jsonl (10GB) and sft_t2t.jsonl (14GB) datasets or adding DPO and Agentic RL scales proportionally. Still, running the full lifecycle from Pretrain to RL on a single consumer GPU for less than the price of a coffee is rare.
End-to-end training pipeline: Pretrain to Agentic RL
The main reason minimind crossed 67,000 GitHub stars is that it implements modern post-training and RLHF/RLAIF algorithms in plain PyTorch.

- Pretrain & Full SFT (
train_pretrain.py,train_full_sft.py): Uses a BPE + ByteLevel tokenizer (len_vocab=6400) to learn base language modeling, then aligns multi-turn chat,<think>adaptive reasoning, and<tool_call>formatting onsft_t2t. - LoRA & Distillation (
train_lora.py,train_distillation.py): Implements low-rank adaptation from scratch (even fast enough to test on CPU) alongside white-box CE + KL distillation from a teacher model. - DPO · PPO · GRPO · CISPO (
train_dpo.py,train_ppo.py,train_grpo.py): Lets you compare offline preference optimization (DPO), actor-critic PPO, critic-free group-relative GRPO, and CISPO (loss_type=cispo), which preserves gradient flow when importance ratios are clipped. - Agentic RL (
train_agent.py): Added in March 2026, it optimizes multi-turn tool-use trajectories with delayed rewards and supports decoupling rollout generation via ansglangserver—mirroring the architecture of large-scale RL frameworks like OpenRLHF and verl in miniature.
Q. Can a 64M MiniMind model replace production chatbots or coding assistants? No. Even the README's own Zero sample shows hallucinations on cross-lingual prompts, reflecting the natural ceiling of 64M parameters. Use it for learning LLM internals, testing ultra-small domain classifiers/routers, or prototyping agentic RL loops, not general-purpose reasoning.
If you are also exploring agentic developer tooling, check out Toris Studio's guides on Plannotator for browser-based plan and diff reviews and CLI-Anything for turning software into agent-native CLIs.
Toris's take
If you want to move from calling LLM APIs to understanding how models are actually trained, this is the single best repository to clone first.
- Best for: Engineers who want executable PyTorch implementations of Transformers, RoPE, MoE, GRPO, and CISPO instead of paper equations, and solo builders with one RTX 3090/4090 or a cheap cloud GPU.
- Skip if: You need an accurate out-of-the-box assistant for production QA or coding—fine-tuning a 7B+ Qwen/Llama/Gemma model or using frontier APIs is far more practical there.
Start small: download only pretrain_t2t_mini.jsonl and sft_t2t_mini.jsonl, run train_pretrain.py --from_resume 1, and serve your finished checkpoint with ollama run.
Sources
Advertisement