Small Models Lag on Agent Benchmarks
Small models under 7 billion parameters usually choke on complex agent tasks. Nanbeige4.2-3B is trying to change that story. Benchmarks like SWE-Bench Verified and GDPval rubrics tend to reward models at 9B parameters and up. A developer who wants a decent local coding agent has had two choices: put up with slower performance, or pay per API call to a bigger hosted model. The model, released on July 22, 2026 by BOSS Zhipin’s Nanbeige LLM Lab, is built to get around that trade-off.
The team trained a Looped Transformer from scratch on 28 trillion tokens, then ran supervised fine-tuning across real-world and synthetic environments. The arXiv paper lays out a pipeline that mixes office document workflows, repository-level code edits and multi-step tool use. A reinforcement learning stage adds outcome and process rewards to keep long trajectories from falling apart. What comes out is a generalist agent that fits on one consumer GPU and handles tasks that normally need twice the parameters or more.
This compact model is not built for one narrow job. The model card reports solid numbers on math reasoning, scientific benchmarks and instruction following. It is meant to support personal assistant workflows when paired with a scaffold like OpenClaw. The lab frames it as a compact local agent that runs where the data already lives, not as another hosted API product.
Nanbeige4.2-3B Looped Transformer Reuses Layers
The architecture is the main reason a 3B model like this one can compete with rivals three to four times its size. Instead of stacking 40 or 60 transformer layers, it pushes hidden states through the same layer stack twice on every forward pass. The first pass builds an intermediate representation. That output then loops back into the identical blocks for a second pass. The reuse adds effective depth without adding a single new weight matrix.
This builds on older recurrent transformer ideas and keeps the memory footprint in check. The lab pretrained the base model from scratch on 28 trillion tokens, up from the 23 trillion used for earlier models. The corpus was upsampled for math, code, STEM and synthetic Q&A. That gave the model a reasoning backbone before it ever saw an agent training example.
A modeling script released alongside the checkpoint hints at what is coming next: LoopSplit, mHC with depth attention and concatenated n-gram embeddings. Those are slated for Nanbeige4.5 later in 2026. For now, the core trick in version 4.2 is layer looping, and it already produces results that break the usual size-to-performance curve.
Benchmark Scores Exceed Larger Models
The model card on Hugging Face lists numbers that go against expectations. On SWE-Bench Verified, which measures the ability to fix real GitHub issues, Nanbeige4.2-3B scores 63.6 percent. Qwen3.5-9B manages 53.1 and Gemma4-12B reaches 44.2. On the tougher SWE-Bench Pro it hits 46.9 against 33.8 and 21.9. On Terminal-Bench 2.0, a command-line agent test, it sits at 44.1 while the larger rivals stay in the 20s.
General agent evaluations tell the same story. On GDPval rubrics, which test complex tool use and office document handling, it posts 74.3. Qwen3.5-9B and Gemma4-12B get 61.9 and 68.5. On Pinch-Bench-V2 it records 74.7 against 68.2 and 53.8. Even on pure reasoning tasks it holds up: 87.4 on GPQA-Diamond and 82.8 on the HMMT-Feb-2026 math contest.
All the agent evaluations run in thinking mode with `preserve_thinking=true`. Office tasks use the lab’s in-house scaffold, while code agent benchmarks run on OpenHands or SWE-agent frameworks. The scores are self-reported and no independent evaluator has replicated them yet. That gap is exactly where community testing needs to step in.
Independent Verification Still Needed
Every performance claim on the model card comes from the Nanbeige team itself. No outside researcher has confirmed the SWE-Bench score or the GDPval result. Compact models often shine on curated benchmarks, then stumble on long multi-turn sessions that expose fragile planning or repetition loops. Without third-party reproduction, these numbers stay promising, not proven.
The practical upside is immediate, though. A 3B model with a 256K context window that runs locally opens up real options for on-device assistants. The model card ships with quantization recipes down to Q4_K_M for CPU-only setups. Day-one support for Ollama and llama.cpp means any developer can try it within minutes on modest VRAM. The Apache 2.0 license allows commercial use with attribution.
The compact agent space is moving fast right now. If these scores hold up under independent evaluation and long-horizon workflows, the Looped Transformer approach could change how teams build small models going forward. For now, it is on the open-source community to stress-test the model exactly where it claims to be strongest.
Frequently Asked Questions
How does Nanbeige4.2-3B beat much bigger models?
The model uses a Looped Transformer, feeding hidden states back into the same layers for a second pass. This reuse increases effective capacity without adding parameters, letting a 3B backbone perform at the level of 9B to 12B rivals on agent tasks.
How do I run Nanbeige4.2-3B locally?
The Hugging Face model card includes recipes for vLLM, SGLang, llama.cpp and Ollama. With BF16 weights the model fits on a single consumer GPU. Quantized GGUF builds run on CPU and small VRAM configurations.
Is Nanbeige4.2-3B open source?
Yes. BOSS Zhipin’s Nanbeige LLM Lab released the model under the Apache 2.0 license, which permits commercial use with attribution. Both the instruction-tuned and base checkpoints are available for download.
