Needle 2 Setup: A 14MB Tool-Calling Model, Fully Offline

Needle 2 is a 45-million-parameter tool-calling model that ships as a single 14MB binary and runs a full session in roughly 28MB of RAM, and installing it takes one pip command. The repository added about 547 stars today on its way past 6,200 total, which is what happens when a project claims the thing everyone assumed impossible: function calling that never touches a network and still returns valid JSON every time.

The trick is not that the model is smart. It is that the model is not allowed to be wrong about shape. Needle compiles your tool schemas into a byte-level grammar and decodes inside it, so a malformed call is not unlikely, it is unrepresentable. That single design choice is why a model this small is usable at all.

RelatedPrime Agent Setup: A Self-Improving Terminal Coding Agent

• Install is pip install cactus-needle; the inference engine downloads once from Hugging Face and caches, then inference is fully offline.
• Weights are baked into the 14MB engine, so there is no separate model file to manage and nothing to compile.
• Every response carries a calibrated confidence score, so you can act above a threshold and escalate to a bigger model below it.
• MIT licensed, Python 3.9 or newer, current PyPI release 2.0.5, with LoRA fine-tuning and export built into the same CLI.

What is Needle 2 and why is it trending?

Needle 2 comes from Cactus Compute, and it is built for tool calling, device control and structured extraction rather than conversation. The repository you install is the Python package: inference, LoRA fine-tuning and export. Under the hood it is what the team calls a Simple Attention Network, a dense small-model recipe using a Hadamard MLP in place of the feed-forward block, grouped-query attention, engram key-value memory and multi-lane hyper-connections, described in arXiv:2607.18363. It is compressed to CQ2-bit with the team's own quantizer and baked into a purpose-built engine.

The claim drawing attention is the size-quality trade. On the project's own benchmarks Needle 2 trades wins with small models like FunctionGemma 270M, LFM2.5 230M and Apple FM, while being 5x to 70x smaller and running at 2 bits against their 16-bit floats. Treat those as the maker's numbers until someone reruns them, but the shape of the claim is checkable in ten minutes on your own laptop, which is a large part of why the stars are moving.

How do you install it on Windows, macOS or Linux?

One command, and the same one everywhere. There are no platform-specific installers because the engine is fetched as a prebuilt binary matched to your machine:

# installs the Python package; needs Python 3.9+
$ pip install cactus-needle

The engine itself downloads on first use and lands in ~/.cache/cactus-needle/<engine version>/. Prebuilt engine targets cover macOS on Apple Silicon, Linux on x86-64 and aarch64, musl builds, and Windows on both amd64 and arm64. After that first fetch, inference never opens a socket.

Now the smallest useful program. Decorate a function, hand it to an agent, and run() closes the loop: the model picks the call, Needle executes your Python, feeds the result back, and returns the response with the executed results attached.

# quickstart.py
import needle

@needle.tool
def get_weather(city: str):
    "Get the current weather for a city."
    return {"city": city, "temp_c": 27, "sky": "clear"}

agent = needle.Needle(tools=[get_weather])
print(agent.run("what's it like in Lagos right now?")["results"])

The docstring is the tool description and the signature is the argument schema, which means the quality of your docstrings is the quality of your agent. That is not a throwaway detail. Needle reads those descriptions to decide what to call and how to fill it, so a vague docstring degrades the model in a way no amount of prompt engineering recovers.

How do you poke at it before writing any code?

There is a local playground, which is the fastest way to decide whether the model is good enough for your use case:

# base model, serves on http://127.0.0.1:7860
$ needle playground
# or point it at weights you tuned yourself
$ needle playground --weights my.cact

The server downloads and initializes the model before it starts serving, so the first query is instant instead of stalling on a cold load. Pick a preset, edit the tools, run a query, then keep going in the same conversation.

What about extraction and structured output?

Extraction is not a separate mode here. It is tool calling with exactly one tool declared, which means the grammar admits only that one call shape and conformance is guaranteed rather than requested. Pass a Pydantic model and you get a typed object back:

from pydantic import BaseModel

class Invoice(BaseModel):
    vendor: str
    total: float
    due_date: str

invoice = needle.extract("Invoice from Acme Corp, $1,200.00, due 2026-09-01", Invoice)
print(invoice.vendor, invoice.total)

Two behaviours are worth internalising because they are unusual. A request no declared tool can serve comes back as an empty call [] rather than an apologetic paragraph, and an optional field with no evidence in the input is omitted rather than guessed. There is no free-text fallback anywhere in the contract.

How does it compare to the usual options?

Read that table as a scoping exercise, not a scoreboard. Needle is not competing with a frontier model on reasoning, and it will lose that fight badly. It is competing with the API call you currently make to turn "remind me at seven" into a JSON object, on a device that may be a wearable, a robot, or a phone in airplane mode.

Relatedjcode Setup: The Rust Coding Agent That Barely Uses RAM

Can you fine-tune it on your own tools?

Yes, and the pipeline is three commands. Synthesis is optional and needs an OpenRouter key; the fine-tune itself does not.

# 1. synthesize training data from your tool schemas (optional)
$ export OPENROUTER_API_KEY=sk-or-...
$ needle generate-data --tools my_tools.json --num-samples 500 --output data.jsonl

# 2. LoRA fine-tune; base checkpoint auto-downloads
$ needle finetune data.jsonl --epochs 10

# 3. merge the adapter and quantize into a single .cact
$ needle build checkpoints/needle2.pkl --lora checkpoints/needle_lora.pkl --out my_needle.cact

Training is plain JAX, so it runs on whatever accelerator JAX supports. On an Nvidia box pip install "cactus-needle[gpu]" puts the same command on the GPU, and on Apple Silicon the equivalent extra is [metal]. The output is one .cact file that runs on the unmodified engine, no recompilation involved.

What are the gotchas before you rely on it?

Four, and the first one bites people who fine-tune first and read later. Confidence calibration holds for the base model only, so an agent running tuned weights reports confidence as None and warns once at construction. If your escalation logic keys off that score, tuning silently removes your safety net.

Second, tool retrieval is a hard filter, not a ranking hint. Declare more than five tools and only the top five per turn enter the context, with the grammar rebuilt over just that subset. An unselected tool is unreachable, not merely unlikely, so a badly described tool in a large catalogue may never fire at all.

Third, the context is a 256-token sliding window with tools pinned as key-value sinks. That is what keeps memory flat no matter how long the conversation runs, and it is also why this is not the model for long documents. Fourth, air-gapped deployment needs planning: needle fetch pulls the engine for the current machine and prints the path, --platform-tag grabs a build for a different device, and on a device that must never attempt the network you also want HF_HUB_OFFLINE=1 so a missing engine fails loudly instead of hanging on a download.

# prepare an air-gapped install from a connected machine
$ needle fetch --platform-tag manylinux2014_aarch64
$ pip download cactus-needle
# then, on the target device
$ pip install --no-index --find-links <dir> cactus-needle

Our take

Most on-device AI projects shrink a general model and hope the losses land somewhere harmless. Needle does the opposite: it narrows the job until a tiny model is genuinely sufficient, then makes the output format structurally impossible to break. You cannot ask it to write your email. You can ask it to turn a sentence into a function call on a device with no connectivity and a battery budget, which is a real problem that currently gets solved with a round trip to someone else's datacentre.

The honest caveat is that the benchmarks are self-reported and the confidence head does not survive tuning, so the safest way in is the base model plus a threshold plus an escalation path to something larger. Try it for an hour with the playground before you design anything around it. At 14MB and one pip command, the cost of finding out is about as low as it gets.

Original analysis by GenZTech. Tool documentation: cactus-compute/needle on GitHub.