NUMALLAMA
NUMALLAMA is a lightweight C inference engine designed to run modern Large Language Models directly on CPUs — without GPUs, CUDA, or heavyweight runtime dependencies.
Its primary goal is to make large multi-socket ARM servers effective LLM inference platforms by taking full advantage of their CPU cores and memory bandwidth.
Source code: github.com/miloserdev/numallama
Why I created NUMALLAMA
Modern LLM inference is often associated with GPUs. However, many servers already have hundreds of CPU cores, substantial RAM capacity, and considerable memory bandwidth.
The problem is that multi-socket systems are not uniform memory architectures. Each CPU socket has its own local memory, and accessing memory attached to another socket introduces additional latency and consumes inter-socket bandwidth.
During testing on a dual-socket Ampere Altra Max server, I found that simply increasing the number of CPU cores did not guarantee better inference performance. In fact, llama.cpp performed better on one socket than on both sockets for several tested configurations.
I created NUMALLAMA to address this problem.
Instead of treating the entire server as one large CPU, NUMALLAMA explicitly organizes inference around NUMA topology, keeping computation and memory access close to the socket that performs the work.
Key advantages
NUMA-aware execution
- Distributes model weights and KV cache across NUMA nodes.
- Uses per-socket work queues and buffers.
- Keeps CPU workers close to the memory they access.
- Reduces expensive cross-socket memory traffic.
ARM-optimized computation
- NEON vectorized kernels.
- ARM dot-product instructions (SDOT).
- Repacked weights for efficient matrix operations.
- Batched prompt processing and multi-token decoding.
Efficient memory usage
- Q8 and asymmetric Q4 quantization.
- Importance-matrix-based quantization.
- Support for large models and Mixture-of-Experts architectures.
Inference features
- Persistent thread pool and tree-based synchronization barriers.
- Exact speculative decoding using prompt lookup or a draft model.
- OpenAI-compatible HTTP API.
- Streaming responses and tool calling.
- Prompt-prefix cache.
- CLI chat with sandboxed tool execution.
Supported models
NUMALLAMA supports several modern model families:
- Llama 3.x
- Qwen 2.5
- Qwen 3
- Qwen 3 MoE
- Qwen 3.5 / Qwen 3.8 hybrid models
Supported architectures and quantization formats depend on the model implementation.
Benchmark results
The primary test platform:
| Component | Configuration |
|---|---|
| CPU | 2 × Ampere Altra Max M128-30 |
| Architecture | Arm Neoverse N1 |
| CPU cores | 256 |
| Clock | 3.0 GHz |
| RAM | 1 TB |
| NUMA nodes | 2 |
| Memory bandwidth | ~180 GB/s per socket |
Decode performance
| Model | NUMALLAMA (2 sockets) | llama.cpp (best, 1 socket) |
|---|---|---|
| Llama 3.2 3B Q8 | 76 tok/s | 36 tok/s |
| Llama 3.2 3B Q4 | 99 tok/s | 64 tok/s |
| Llama 3.1 8B Q8 | 39 tok/s | 19 tok/s |
| Llama 3.1 8B Q4 | 59 tok/s | 29 tok/s |
Larger models
| Model | Quantization | Decode | Prefill |
|---|---|---|---|
| Qwen3-30B-A3B (MoE) | Q4 | 91 tok/s | ~500 tok/s |
| Qwen3.8-27B | Q4 | 17 tok/s | 172 tok/s |
These results are measurements from one specific server configuration, not universal performance guarantees. Performance depends on hardware, model, quantization, and runtime configuration.
Full benchmark methodology and results:
Building from source
NUMALLAMA is primarily written in C and uses a compact inference runtime.
On Linux, clone the repository and build it:
git clone https://github.com/miloserdev/numallama.git
cd numallama
make
The optimized kernels target ARM64 systems with NEON and dot-product instruction support, including Ampere Altra and other Neoverse-based processors.
x86-64 builds use portable C fallbacks and are not currently optimized.
Preparing a model
The following example prepares Qwen3-30B-A3B-Instruct-2507 using the included export scripts.
Export the model weights:
python3 scripts/export_hf_q8.py \
hf/Qwen3-30B-A3B-Instruct-2507 \
qwen3_30b_q80.bin
Export the tokenizer:
python3 scripts/export_tokenizer.py \
hf/Qwen3-30B-A3B-Instruct-2507 \
qwen3_tokenizer.bin
The model directory should contain the Hugging Face model files required by the export scripts.
Running the inference server
Start the server:
./runq qwen3_30b_q80.bin \
-z qwen3_tokenizer.bin \
-m server \
-P 8080
The server exposes an OpenAI-compatible API:
http://127.0.0.1:8080/v1
You can connect existing applications and tools that support the OpenAI API format.
For example, using Python:
from openai import OpenAI
client = OpenAI(
base_url="http://127.0.0.1:8080/v1",
api_key="not-needed"
)
response = client.chat.completions.create(
model="qwen3",
messages=[
{"role": "user", "content": "Explain NUMA architecture."}
]
)
print(response.choices[0].message.content)
The model value should match the model identifier exposed by the running server.
Limitations
- Linux only.
- Optimized kernels currently target ARM64 with NEON and dot-product instructions.
- No GPU acceleration or GPU offloading.
- One active request at a time; additional requests wait in a queue.
- No multi-user continuous batching.
- Text-only inference.
- Model support is limited to implemented architectures.
Project
NUMALLAMA is an open-source project focused on CPU inference, ARM optimization, and NUMA-aware execution.
The project explores an alternative approach to LLM infrastructure: using the CPU resources already available in large servers instead of requiring dedicated GPU hardware for every inference workload.
- GitHub: miloserdev/numallama
- Benchmarks: BENCHMARKS.md