LFM3-1.7B With a Python Tool at Full Context: Same Accuracy, Fewer Tokens

Austin Dixson, October 30, 2026

Abstract

We ran full-precision (BF16) LFM3-1.7B on all 60 AIME 2025 and 2026 problems and the full MATH-500, in three setups: no tools, a run_python tool, and run_python plus a prompt that asks the model to check its answer in code. Each run had the model's full 262,144-token context, Liquid AI's recommended sampling, and one sample per problem, on a Hugging Face A100 with SGLang. Without tools the model scored 50/60 on AIME (83.3%, Wilson 95% CI 72.0 to 90.7) and 486/500 on MATH-500 (97.2%, CI 95.4 to 98.3). With Python it scored 47/60 and 486/500, and with Python plus the check it scored 50/60 and 481/500. No paired difference was significant (McNemar exact p between 0.27 and 1.0). The tool did cut length: tokens per solved AIME problem fell from 21,776 to 16,688 (−23.4%) with Python and to 17,495 (−19.7%) with the check prompt. On October 28 the same tool looked like an accuracy gain (AIME 2025 17/30 to 24/30 under a 25,000-token cap). At full context that gain did not appear, so we now read it as an effect of the cap. The useful result is speed: the 4-bit build of this model, a 1.05 GB file (about 1.2 GB in memory), runs at a median 153 tokens per second on a 2021 MacBook Pro M1 Max, and the tool lets it reach about the same accuracy with a fifth to a quarter fewer tokens.

1 Introduction

Two days ago we published a measurement of LFM3-1.7B with a Python sandbox on an M1 Max Mac [1]. Those runs used greedy decoding and a token cap. A follow-up run on AIME 2025 with a 25,000-token cap made the tool look like it raised accuracy, from 17/30 to 24/30 with a check-in-code prompt (McNemar p = 0.039) [2].

A cap changes what a benchmark measures. If the model runs out of tokens before it boxes an answer, the item counts as wrong, and anything that shortens the answer will look like an accuracy gain. The question for this run was simple: with no cap that matters, does the tool still raise accuracy, or does it only make answers shorter?

2 Setup

Model: LiquidAI/LFM3-1.7B in BF16 [3]. Server: SGLang (lmsysorg/sglang:dev-lfm3) with --context-length 262144 --dtype bfloat16, on one Hugging Face Jobs a100-large instance per setup. The three setups ran as three parallel jobs on October 30, 2026, between 2:23 and 2:34 PM CT.

Sampling followed the model card's math settings: temperature 1.0, top_p 0.95, top_k 20, min_p 0, thinking on. The seed for each item was the CRC32 of its id, the same in every setup. Each problem got one sample.

Problems: AIME 2025 (30), AIME 2026 (30), and MATH-500 (500), 560 per setup.

System prompts:

• No tools: "Solve the math problem. Reason step by step, then put the final answer in \boxed{}."
• Python: the same, plus "You have a run_python tool (max 6 calls) that executes Python 3 code (sympy available if installed) and returns stdout; use it to compute or verify."
• Python + check: the Python prompt, plus "Before you give the final answer, you must run Python code that independently checks it (substitute back, brute-force small cases, or recompute another way). Only box an answer that your code confirmed."

On Hugging Face the tool ran each call in a fresh Python process as the unprivileged nobody user, with at most 6 calls per problem and a 10 s timeout. This was not the tool's network-denied sandbox. A network probe at the start of each job returned NET_OK, so code the model ran in these cloud runs had network access. Answers were scored with math_verify against the reference. We also scored them with the October 28 string checker for comparison.

Confidence intervals are Wilson score intervals at 95%. For k correct out of n, with p = k/n and z = 1.96, the center is (p + z²/2n) / (1 + z²/n) and the half-width is z · sqrt(p(1 − p)/n + z²/4n²) / (1 + z²/n). For 50/60: p = 0.8333 and z²/n = 3.8416/60 = 0.0640. The center is (0.8333 + 0.0320) / 1.0640 = 0.8133. The half-width is 1.96 · sqrt(0.002315 + 0.000267) / 1.0640 = 1.96 · 0.0508 / 1.0640 = 0.0936. The interval is 72.0 to 90.7.

Paired comparisons use the exact McNemar test. If b items are right only in setup A and c items are right only in setup B, with n = b + c, then p = 2 · Σ_{k ≤ min(b,c)} C(n, k) / 2^n, capped at 1.

3 Results

3.1 Accuracy

Ranges in parentheses are Wilson 95% intervals. No answer in any setup ran out of context, and no request errored.

3.2 Paired tests

Worked case, Python vs no tools on AIME 60: Python alone solved 3 problems and no tools alone solved 6. With n = 9, p = 2 · (C(9,0) + C(9,1) + C(9,2) + C(9,3)) / 2^9 = 2 · (1 + 9 + 36 + 84) / 512 = 260 / 512 = 0.51.

None of the six comparisons, or the per-year AIME comparisons in the data files, reaches p < 0.05. With 60 AIME problems a difference needs to be large to show up, so the honest statement is that we could not detect an accuracy effect in either direction, and the point estimates sit within 5 points of each other.

3.3 Length

Arithmetic:

• AIME per solved, Python: (16,688 − 21,776) / 21,776 = −5,088 / 21,776 = −23.4%.
• AIME per solved, check: (17,495 − 21,776) / 21,776 = −4,281 / 21,776 = −19.7%.
• AIME mean, Python: (13,072 − 18,147) / 18,147 = −5,075 / 18,147 = −28.0%.
• MATH-500 per solved, Python: (3,221 − 3,752) / 3,752 = −531 / 3,752 = −14.2%.
• Total, Python: (2,349,720 − 2,912,297) / 2,912,297 = −562,577 / 2,912,297 = −19.3%.

On the A100 the measured time per solved AIME problem went from 287.1 s to 246.9 s with Python, (246.9 − 287.1) / 287.1 = −14.0%. On MATH-500 it went from 74.9 s to 77.7 s, +3.7%, because short problems save few tokens and still pay for the tool round trip. Each job ran all 560 problems at once on one GPU, so these per-item times include queueing and are inflated. Token counts are the better guide.

3.4 What this means on a Mac

On October 30 we measured the 4-bit build, LFM3-1.7B-QAD-Q4_0.gguf (a 1,049,615,680-byte file, 1.05 GB, about 1.2 GB in memory), with llama.cpp llama-server build 11938 on a 2021 MacBook Pro M1 Max with 64 GB. Over four 400-token trials it generated 137.6 to 157.2 tokens per second, median 153, and processed a short prompt at 687 to 729 tokens per second.

At that speed, decoding the no-tools AIME length of 21,776 tokens per solved problem would take about 21,776 / 153 = 142 s, and the Python length of 16,688 tokens about 16,688 / 153 = 109 s. That is arithmetic, not a measurement: it applies BF16 token counts to the 4-bit model and ignores prompt processing and tool time.

A separate tool-use test on the same Mac did measure time directly. Elon, an agent on our team, gave the same 4-bit model a run_python tool (a fresh process per call under sandbox-exec, network denied, 10 s timeout, 2,000-character output cap) and 27 made-up tasks covering math, unit conversion, dates and combinations. Sampling was temperature 0.1, top_k 50, repeat_penalty 1.1, with at most 8,192 output tokens. With Python the model got 27/27 on each of three seeds (81/81). Without tools it got 21/27. Mean time per task on seed 0 was 4.90 s with Python and 27.57 s without, 27.57 / 4.90 = 5.6 times faster, and mean output tokens were 556 vs 3,465, 3,465 / 556 = 6.2 times fewer. All 120 tool calls were valid JSON with the right name and arguments. All 6 no-tool misses hit the 8,192-token cap without an answer, so the accuracy gap in that test is a cap effect too. The speed gap is the result that carries over.

3.5 The October 28 gain was a cap effect

Without tools the score moved from 17/30 under the cap to 24/30 at full context, the same score the check-in-code prompt reached under the cap. The tool setups barely moved. The simplest reading is that the cap was cutting off long no-tool answers, and the tool helped under the cap by making answers shorter. Once the cap is gone the shorter answers are still shorter, but they are no more often right.

The two runs also differ in precision (Q4_0 vs BF16), decoding (greedy vs sampled) and hardware, so this comparison is not a controlled test of the cap alone. The October 28 paper on 30 MATH-500 problems [1] found the same pattern at a smaller scale: all six no-tool misses under a 4,096-token cap were cut-off answers, and all six were solved with 25,000 tokens and no tools.

3.6 Reference point

The Qwen3.6-35B-A3B model card lists 92.7 on AIME 2026 [4]. That is their reported number from their own setup, and the card does not say whether tools were used. LFM3-1.7B scored 86.7 (26/30) on AIME 2026 without tools here, 6 points lower, with about 1/20 of the total parameters (1.7B vs 35B). We did not run Qwen3.6-35B-A3B on this set, so the two numbers are not a controlled comparison.

4 Cost

The three A100 jobs cost $0.464 (no tools, 668 s), $0.394 (Python, 568 s) and $0.424 (Python + check, 611 s). A 4-item smoke test cost $0.158. The total was 0.158 + 0.464 + 0.394 + 0.424 = $1.44.

Job costs include server startup and model load.

5 Limitations

One sample per problem. Sampled decoding at temperature 1.0 means a rerun could move a few items in each setup. We did not measure that variance.

The sample is small for AIME. Sixty problems give Wilson intervals about 19 points wide around 83%.

We used Liquid's recommended sampling instead of the greedy decoding of October 28, so the two runs differ in more than the cap.

The October 30 cloud runs did not use the tool's network-denied sandbox. On the Mac, run_python runs under sandbox-exec with network denied. On Hugging Face, Python ran as the nobody user with network access, and the logged network probe returned NET_OK in all three jobs. We did not check whether any of the model's code used the network, so these runs do not rule it out.

Wall times on the A100 are inflated by running 560 requests at once and should be read as rough.

Grading uses math_verify. The October 28 string checker gave the same AIME counts but disagreed with it on 20, 22 and 19 items per setup (no tools, Python, Python + check), which shows up as lower MATH-500 counts under that checker (466, 464 and 462). We report math_verify.

AIME 2025, AIME 2026 and MATH-500 are public, so some items may have been seen in training.

The Mac speed figures are for the 4-bit build; the accuracy figures are for BF16. We did not run the full AIME and MATH-500 sets on the Mac.

6 Conclusion

With its full context, LFM3-1.7B solves 83% of recent AIME problems and 97% of MATH-500 without help, and a Python tool does not change those numbers by a measurable amount. What the tool changes is length, by about 20 to 28% on AIME, which on a laptop is time saved. For a model that ships as a 1.05 GB file, takes about 1.2 GB in memory, and runs at about 150 tokens per second on a 2021 M1 Max, that is the reason to use it.

Data

• Raw outputs, scorer, per-item results and the full report: https://huggingface.co/austindixson/lfm3-smallbig-distill/tree/main/math_bf16_1030 (score_report.md, score_summary.json).
• The tool and proxy: https://huggingface.co/austindixson/python-sandbox-math-tool.

References

[1] A. Dixson. A 1 GB Model With a Python Sandbox: Measuring Tool Use for LFM3-1.7B on an M1 Mac. https://www.destroyrebuild.xyz/blog/lfm3-python-sandbox, 2026.

[2] A. Dixson. python-sandbox-math-tool, results for October 28, 2026. https://huggingface.co/austindixson/python-sandbox-math-tool, 2026.

[3] Liquid AI. LFM3-1.7B model card. https://huggingface.co/LiquidAI/LFM3-1.7B, 2026.

[4] Qwen Team. Qwen3.6-35B-A3B model card. https://huggingface.co/Qwen/Qwen3.6-35B-A3B, 2026.