Skip to content
JournalTORIS / FIELD NOTES

← BlogAI insights

MiniMind-3 (64M): Train Your Own LLM from Scratch in Native PyTorch

Breaking down jingyaogong/minimind (67.5K stars): 64M Dense vs. 198M-A64M MoE specs, what the 2-hour RTX 3090 claim actually covers, and the full Pretrain-to-Agentic RL pipeline.

Read the Korean original

MiniMind-3 (64M): Train Your Own LLM from Scratch in Native PyTorch

Q. Can MiniMind really train a custom LLM from scratch in 2 hours on a single consumer GPU? Yes, with one important caveat. The "2 hours and 3 RMB (~$0.43)" benchmark in the official README measures 1 epoch of SFT (sft_t2t_mini) on minimind-3 (64M) using a single NVIDIA RTX 3090, not the entire multi-stage pipeline. Even if you run Pretrain, SFT, ToolCall, and RLAIF end to end, each stage takes roughly 1 to 1.5 hours, making it practical for an individual developer to inspect every line of code and finish a full run.

⚠️ Accuracy guardrail: This guide was written on November 20, 2026 by directly inspecting the jingyaogong/minimind GitHub repository (67,494 Stars, Apache-2.0) and the November 18 Toris Studio GitHub Trending Top 10 video. GPU rental estimates (~1.3 RMB/hour) and training durations are empirical single-RTX 3090 figures reported by the repository and vary with hardware and dataset size.

30-second summary

  • Zero high-level wrapper dependencies: Built from scratch in native PyTorch without relying on transformers, trl, or peft abstractions—covering tokenizer training, Dense/MoE models, LoRA, DPO, PPO, GRPO, CISPO, and Agentic RL.
  • Ultra-lightweight minimind-3 specs: Ships a 64M Dense model (d_model=768, n_layers=8, q_heads=8, kv_heads=4, RoPE + YaRN 32K) and a 198M-A64M MoE variant (4 experts / top-1), aligned with the Qwen3 ecosystem.
  • Stage-by-stage cost on 1x RTX 3090: With the mini datasets, Pretrain takes ~1.21h (~1.57 RMB), Full SFT ~1.10h (~1.43 RMB), ToolCall ~0.90h (~1.17 RMB), and RLAIF ~1.10h (~1.43 RMB).
  • Drop-in local inference: Trained weights run directly via ollama run jingyaogong/minimind-3, vLLM, llama.cpp, an OpenAI-compatible server (serve_openai_api.py), and a Streamlit WebUI.

Why MiniMind avoids transformers and peft

MiniMind is designed not just to ship a tiny checkpoint, but to expose the entire mechanics of building an LLM from scratch without hiding the math behind framework wrappers.

In typical fine-tuning workflows, transformers, peft, and trl let you train a model in a dozen lines of code. That convenience is great for production, yet it leaves attention masking, RoPE rotary embeddings, LoRA adapter injection, and RL policy gradients inside a black box. jingyaogong/minimind removes those abstractions by implementing the core model (model_minimind.py) and every training loop (trainer/) directly in native PyTorch.

Official jingyaogong/minimind GitHub repository

Released on April 1, 2026, minimind-3 also made a practical architectural shift: it aligned its parameter naming and decoder structure with Qwen3 / Qwen3-MoE. As a result, weights you train from scratch can be exported to standard inference engines such as llama.cpp, vLLM, and Ollama without custom glue code.

Advertisement

Model lineup and architecture specs

The current main branch centers on two configurations: minimind-3 (64M Dense) and minimind-3-moe (198M-A64M).

Model Params Vocab size Max context Layers / d_model Attention heads Notes
minimind-3 64M 6,400 32,768 (YaRN) 8 / 768 Q=8, KV=4 Main Dense model
minimind-3-moe 198M-A64M 6,400 32,768 (YaRN) 8 / 768 Q=8, KV=4 4 experts / top-1
minimind2 104M 6,400 32,768 (YaRN) 16 / 768 Q=8, KV=2 2025.04 legacy
minimind2-small 26M 6,400 32,768 (YaRN) 8 / 512 Q=8, KV=2 2025.04 legacy

One notable engineering trade-off appears in minimind-3-moe. While MoE speeds up inference via sparse activation, pure PyTorch training without fused Triton or DeepSpeed-MoE kernels incurs heavy token-dispatch scheduling overhead as expert counts grow. To preserve readable native PyTorch code, the author removed shared experts and settled on 4 experts / top-1 routing, keeping MoE training roughly 50% slower than the Dense baseline.

What '2 hours and 3 RMB' actually covers

Taking "train an LLM in 2 hours" at face value can mislead beginners into thinking dataset download through RL alignment all finish inside 120 minutes. The official README footnote and benchmark table spell out the exact scope.

Developer workstation running local LLM training logs on an RTX 3090 GPU

Model Pretrain (mini, 1.2GB) Full SFT (mini, 1.6GB) ToolCall RLAIF (PPO/GRPO)
minimind-3 (64M) ≈ 1.21h (≈ 1.57 RMB) ≈ 1.10h (≈ 1.43 RMB) ≈ 0.90h (≈ 1.17 RMB) ≈ 1.10h (≈ 1.43 RMB)
minimind-3-moe (198M-A64M) ≈ 1.69h (≈ 2.20 RMB) ≈ 1.54h (≈ 2.00 RMB) ≈ 1.26h (≈ 1.64 RMB) ≈ 1.54h (≈ 2.00 RMB)

Basis: Single NVIDIA RTX 3090 (24GB), estimated rental cost ~1.3 RMB/h (~7 RMB ≈ $1 USD), 1 epoch on pretrain_t2t_mini.jsonl and sft_t2t_mini.jsonl.

In short, Pretrain(mini) and SFT(mini) each take around 1 to 1.2 hours—roughly 2.3 hours (~3 RMB) combined for a basic reproducible chat model (MiniMind Zero). Training on the full pretrain_t2t.jsonl (10GB) and sft_t2t.jsonl (14GB) datasets or adding DPO and Agentic RL scales proportionally. Still, running the full lifecycle from Pretrain to RL on a single consumer GPU for less than the price of a coffee is rare.

End-to-end training pipeline: Pretrain to Agentic RL

The main reason minimind crossed 67,000 GitHub stars is that it implements modern post-training and RLHF/RLAIF algorithms in plain PyTorch.

MiniMind end-to-end pipeline from Pretrain to SFT, RLHF/RLAIF, and local inference

  1. Pretrain & Full SFT (train_pretrain.py, train_full_sft.py): Uses a BPE + ByteLevel tokenizer (len_vocab=6400) to learn base language modeling, then aligns multi-turn chat, <think> adaptive reasoning, and <tool_call> formatting on sft_t2t.
  2. LoRA & Distillation (train_lora.py, train_distillation.py): Implements low-rank adaptation from scratch (even fast enough to test on CPU) alongside white-box CE + KL distillation from a teacher model.
  3. DPO · PPO · GRPO · CISPO (train_dpo.py, train_ppo.py, train_grpo.py): Lets you compare offline preference optimization (DPO), actor-critic PPO, critic-free group-relative GRPO, and CISPO (loss_type=cispo), which preserves gradient flow when importance ratios are clipped.
  4. Agentic RL (train_agent.py): Added in March 2026, it optimizes multi-turn tool-use trajectories with delayed rewards and supports decoupling rollout generation via an sglang server—mirroring the architecture of large-scale RL frameworks like OpenRLHF and verl in miniature.

Q. Can a 64M MiniMind model replace production chatbots or coding assistants? No. Even the README's own Zero sample shows hallucinations on cross-lingual prompts, reflecting the natural ceiling of 64M parameters. Use it for learning LLM internals, testing ultra-small domain classifiers/routers, or prototyping agentic RL loops, not general-purpose reasoning.

If you are also exploring agentic developer tooling, check out Toris Studio's guides on Plannotator for browser-based plan and diff reviews and CLI-Anything for turning software into agent-native CLIs.

Toris's take

If you want to move from calling LLM APIs to understanding how models are actually trained, this is the single best repository to clone first.

  • Best for: Engineers who want executable PyTorch implementations of Transformers, RoPE, MoE, GRPO, and CISPO instead of paper equations, and solo builders with one RTX 3090/4090 or a cheap cloud GPU.
  • Skip if: You need an accurate out-of-the-box assistant for production QA or coding—fine-tuning a 7B+ Qwen/Llama/Gemma model or using frontier APIs is far more practical there.

Start small: download only pretrain_t2t_mini.jsonl and sft_t2t_mini.jsonl, run train_pretrain.py --from_resume 1, and serve your finished checkpoint with ollama run.

Sources

Advertisement