A 1 GB Model With a Python Sandbox: Measuring Tool Use for LFM3-1.7B on an M1 Mac

Austin Dixson, October 28, 2026

Correction, October 30, 2026: a full-context rerun of BF16 LFM3-1.7B on AIME 2025, AIME 2026 and all of MATH-500 found no accuracy gain from the Python tool. The 24/30 to 28/30 gain under the 4,096-token cap below was a cap effect, as was a later AIME gain under a 25,000-token cap. The token savings held. See the October 30 follow-up.

Abstract

Liquid AI released LFM3-1.7B on October 27, 2026. We ran its 4-bit QAD-Q4_0 build (a 1.05 GB file) with llama.cpp on an Apple M1 Max and measured what happens when the model can call a sandboxed Python interpreter with no network access. We used 30 problems from MATH-500, greedy decoding, and two output budgets (4,096 and 25,000 tokens). Under the 4,096-token cap the model scored 24/30 on MATH without tools, and all six misses were answers cut off by the cap. With Python it scored 28/30, fixing four of those six and breaking none. With a 25,000-token budget the model solved all 30 without tools (Wilson 95% CI 88.6 to 100), and with Python it reached the same answers using 33% fewer tokens (3,036.8 to 2,022.8 per item) and 31% less time (24.18 to 16.64 s per item). Letting a model run code for math is an established idea (PAL, PoT, ToRA). What we add is a measurement of that idea on a 1.7B model at 4-bit, on a 2021-generation M1 Max Mac, with all the numbers and failure cases published.

1 Introduction

LFM3-1.7B is a 1.73B-parameter model from Liquid AI, released on October 27, 2026 [1]. Its layers mix 13 gated short-convolution blocks, 5 sliding-window attention layers and 6 global attention layers. The model card says that at 4-bit precision it takes about 1 GB. The card reports tool-calling and reasoning scores (BFCL V4 71.39, GPQA Diamond 63.37, LiveCodeBench v6 73.46) and does not report MATH-500 or MMLU-Pro.

A model this small is cheap to run on a laptop. The question we had was practical. If you give it a Python sandbox on the same laptop, does it get better at math, and what does that cost in time?

The first version of this test used a 4,096-token output cap, and the scores it produced were wrong in an instructive way. Every math miss was a cut-off answer. So we reran the model with a 25,000-token budget and report both runs here. The difference between them is the first finding.

2 Related work

Using code execution to help language models with math is not new. Program-aided Language Models (PAL) had the model write a Python program and let the interpreter compute the answer [2]. Program of Thoughts (PoT) prompting did the same thing and reported large gains on math word problems and financial QA [3]. ToRA trained models to interleave natural-language reasoning with tool calls for competition math [4]. Toolformer showed a model could learn by itself when to call a calculator and other APIs [5]. Our setup is a plain version of these ideas: one system-prompt sentence tells the model a tool exists, and there is no fine-tuning.

The benchmark comes from MATH [6], using the 500-problem MATH-500 subset of Lightman et al. [7]. We also report a no-tools MMLU-Pro [8] baseline as a knowledge reference.

For reference points we used published numbers and did not rerun them. The Qwen3.6-35B-A3B card reports MMLU-Pro 85.2 and GPQA 86.0 for Qwen3.6-35B-A3B, and MMLU-Pro 82.6 for Gemma-4-26B-A4B [9]. It does not list MATH-500. Liquid's LFM2.5-24B-A2B card reports MATH-500 of 96.04 for LFM2.5, 96.73 for Qwen3.6, 96.40 for Nemotron-3.5-Lightning (30B-A3B) and 95.66 for Gemma-4-26B-A4B-it, all from Liquid's own evaluation with vLLM, temperature 0.6 and up to 32,768 output tokens [10]. The Qwen MATH-500 figure in this paper therefore comes from Liquid's card and not from Qwen.

So the answer to "is the Python tool an original discovery?" is no. The contribution here is a measurement. To our knowledge nobody had yet published what this tool loop does for LFM3-1.7B at 4-bit on consumer Apple hardware, one day after release.

3 Setup

Hardware was an Apple M1 Max with 64 GB of memory. We ran the model with llama.cpp build 11938 (llama-server, Metal backend, 65,536-token context, one slot, --jinja chat template). The weights were LFM3-1.7B-QAD-Q4_0.gguf, 1,049,615,680 bytes (1.05 GB). A llama-bench run measured prompt processing at 1,985.42 ± 0.16 tokens/s (pp512) and generation at 168.03 ± 0.15 tokens/s (tg128). During the long evaluation runs the server log showed generation near 135 tokens/s. For comparison, the Q4_K_M build of the same model generated at 139.39 tokens/s, and an MLX 4-bit run generated at 179.8 tokens/s with 1.715 GB (about 1.7 GB) peak memory.

The test set was drawn once with random.Random(0).sample: 30 problems from MATH-500 and, for the no-tools knowledge baseline, 30 questions from the MMLU-Pro test split. Decoding was greedy (temperature 0, top_k 1) with thinking turned on. Liquid recommends temperature 1.0, top_p 0.95 and top_k 20 for math, so our setting is a deliberate departure that makes runs repeatable.

The math system prompt was "Solve the math problem. Reason step by step, then put the final answer in \boxed{}." The MMLU-Pro prompt asked for a last line of the form Answer: <letter>. In the tool runs one more sentence told the model it could call tools, up to six calls per item. This paper reports tool-run results for MATH only.

The tool is run_python. It ran the model's code in python -I under macOS sandbox-exec with the profile (version 1)(allow default)(deny network*), a 10-second timeout and output clipped at 2,000 characters. The harness also exposed one non-code tool, which is outside the scope of this paper. The model never called it on a MATH item: every MATH tool call in both tool runs (29 calls at 4,096 tokens, 35 at 25,000) was run_python.

Math answers were graded by taking the last \boxed{}, normalizing common LaTeX (\dfrac, \text, degree signs, a leading x=), then accepting an exact string match, an unordered match on comma lists, or a numeric match within 1e-6. MMLU-Pro was graded by a regex on the final Answer: letter.

Confidence intervals are Wilson score intervals at 95%. For k correct out of n, with p = k/n and z = 1.96:

center = (p + z²/2n) / (1 + z²/n)

half-width = z · sqrt( p(1 − p)/n + z²/4n² ) / (1 + z²/n)

As a worked case, 24/30 gives p = 0.8 and z²/n = 3.8416/30 = 0.1281. The center is (0.8 + 0.0640) / 1.1281 = 0.766. The half-width is 1.96 · sqrt(0.00533 + 0.00107) / 1.1281 = 1.96 · 0.0800 / 1.1281 = 0.139. The interval is 0.627 to 0.905.

4 Results

4.1 Main table

In the 4,096 runs, the no-tools MATH run hit the cap 6 times and the Python run 2 times. The 25,000-token runs never hit the cap on MATH. The model called Python on 24 of 30 MATH items at 4,096 and 28 of 30 at 25,000.

As a knowledge baseline without tools, the same model scored 12/30 on MMLU-Pro at 4,096 tokens (CI 24.6 to 57.7; 2,443.3 tokens and 18.42 s per item; 8 cap hits) and 15/30 at 25,000 tokens (CI 33.2 to 66.8; 4,611.8 tokens and 36.98 s per item).

For context, the same 4,096-token harness gave LFM2.5-24B-A2B 26/30 on MATH and 15/30 on MMLU-Pro, and Qwen3.6-35B-A3B with thinking off 29/30 and 21/30. Those runs were capped and are not comparable to the published numbers for those models.

4.2 Findings

(i) The 4,096-token cap caused every math miss. The six MATH problems missed without tools at 4,096 (math-71, 128, 298, 401, 432 and 494) all ended at the cap with no boxed answer. With 25,000 tokens and no tools the model solved all six. They needed 9,140, 14,025, 5,453, 7,526, 9,942 and 16,716 tokens. The longest correct answer in the study, math-494, took 16,716 tokens and 137.01 seconds. A 1.7B reasoning model with thinking on needs a large budget, and a 4,096 cap measures the budget more than the model.

(ii) With enough budget, the 1.7B model solved every MATH problem in the sample. 30/30 has a Wilson interval of 88.6 to 100, which contains the 96.73 that Liquid reports for Qwen3.6-35B-A3B. Thirty problems cannot separate the two models. They can only show the small model is in the same range on this kind of problem.

(iii) Python kept accuracy and cut cost. At 25,000 tokens, tools reached the same 30 correct answers (after the hand check in 4.3) while dropping mean tokens from 3,036.8 to 2,022.8, a 33.4% cut, and mean time from 24.18 to 16.64 seconds, a 31.2% cut. Under the 4,096 cap the tool raised the score from 24/30 to 28/30. It fixed math-71, 128, 298 and 401 and broke nothing. The two it did not fix, math-432 and math-494, made no tool calls and ran out of tokens again.

(iv) Knowledge is the weak spot, and a code tool does not address it. Without tools at 25,000 tokens the model scored 15/30 on MMLU-Pro (50%, CI 33.2 to 66.8), far below the 85.2 published for Qwen3.6-35B-A3B. This paper does not test a tool for knowledge questions.

4.3 Worked examples

math-128

"Suzanne walks four miles every third day. What is the fewest number of miles she can walk in February?" The gold answer is 36. Without tools at 4,096 the model reasoned until the cap (4,096 tokens, 30.44 s). Without tools at 25,000 it got 36 after 14,025 tokens and 118.29 seconds. With Python it wrote this program (the comment is the model's own):

def total_miles(days):
    return 4 * (days // 3)  # number of multiples of 3 up to days inclusive? Actually floor division gives count of integers k such that 3k <= days.
for n in [28, 29]:
    print(n, total_miles(n))

The output was 28 36 and 29 36, and it answered 36 after 1,506 tokens and 12.16 seconds. That is 9.3 times fewer tokens and 9.7 times less time than the no-tools answer. The program checks both February lengths, which is the step the unaided model kept circling.

math-71

"The binary number 10101001110₂ is equal to what number in base eight?" The gold answer is 2516₈. Without tools at 4,096 the model hit the cap (28.59 s). At 25,000 it was correct after 9,140 tokens and 70.66 seconds. With Python, the first call was:

binary = '10101001110'
decimal = int(binary, 2)
octal = oct(decimal)[2:]  # remove '0o' prefix
print(decimal, octal)

It printed 1358 2516. A second call printed the decimal 1358 and octal 0o2516. In its final text the model tried to group the bits by hand, got tangled ("Wait, let's do it correctly..."), then wrote "using Python we already have the correct octal digits 2516. Let's trust the computed result" and answered \boxed{2516_8}. The 25,000-token tools run took 3,933 tokens and 30.72 seconds. The model trusted the tool over its own arithmetic, which is the behavior you want.

math-248, a grader miss

In the 25,000 tools run the model answered \begin{pmatrix} -1 & 0 \\[2pt] 0 & -1 \end{pmatrix} after a SymPy call returned Matrix([[-1, 0], [0, -1]]). The gold matrix is the same but written without the [2pt] spacing hint. Our normalizer did not strip it, so the grader marked a correct answer wrong. We report the graded 29/30 in the table and figure and the hand-checked 30/30 in the text.

5 Limitations

The sample is small. Thirty items gives intervals about 30 points wide in the middle of the range, and most differences in the main table sit inside overlapping intervals. Only the no-tools MATH change from 24/30 to 30/30 and the four tool fixes under the cap are clear item-level effects.

Each condition ran once with greedy decoding. This makes runs repeatable but leaves sampling variance unmeasured, and it differs from Liquid's recommended math settings (temperature 1.0) and its eval settings (temperature 0.6). Scores at those settings could be higher or lower.

The grader has at least one false negative (math-248), and there may be others we did not catch. The MMLU-Pro baseline grading depends on the model writing the exact Answer: line.

MATH-500 and MMLU-Pro are public, and the model was trained on 22 trillion tokens, so some test items may have been seen in training. Nothing here rules that out.

The comparison numbers for Qwen3.6 and LFM2.5 come from model cards and were produced with different harnesses, temperatures and budgets. They are reference points only.

The sandbox blocks network access but otherwise uses (allow default), so it is not a hardened security boundary. It is suitable for a benchmark on a personal machine and should not be used to run untrusted code.

6 Next steps

The next experiment is routing. The 1.7B model is fast and good at math with Python. On knowledge questions it scores about 50%. A simple router could send math and code to LFM3-1.7B and send knowledge-heavy questions to a larger local model, and we want to measure the accuracy and time of that split against each model alone.

References

[1] Liquid AI. LFM3-1.7B model card. https://huggingface.co/LiquidAI/LFM3-1.7B, 2026.

[2] L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, G. Neubig. PAL: Program-aided Language Models. arXiv:2211.10435, 2022.

[3] W. Chen, X. Ma, X. Wang, W. W. Cohen. Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks. arXiv:2211.12588, 2022.

[4] Z. Gou et al. ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving. arXiv:2309.17452, 2023.

[5] T. Schick et al. Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv:2302.04761, 2023.

[6] D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, J. Steinhardt. Measuring Mathematical Problem Solving With the MATH Dataset. arXiv:2103.03874, 2021.

[7] H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, K. Cobbe. Let's Verify Step by Step. arXiv:2305.20050, 2023.

[8] Y. Wang et al. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. arXiv:2406.01574, 2024.

[9] Qwen Team. Qwen3.6-35B-A3B model card. https://huggingface.co/Qwen/Qwen3.6-35B-A3B, 2026.

[10] Liquid AI. LFM2.5-24B-A2B model card. https://huggingface.co/LiquidAI/LFM2.5-24B-A2B, 2026.