teenygrad 0.1.1: the first release

teenygrad 0.1.1 is out. Twelve crates, Apache 2.0, on crates.io. It is the first version we would suggest anyone actually install — earlier tags exist only because publishing a workspace of interdependent crates takes a dry run or two.

What it is: a Rust-native ML compiler and runtime. You write a model and its kernels in Rust, and teenyc lowers them through MIR, MLIR, Triton and LLVM IR to PTX. What lands on the device is a binary. There is no interpreter on the board, no ONNX round-trip at deploy time, no vendor runtime to keep matched against a driver.

This post covers where the project is going, what is in the release, what is demonstrably not finished, and — since it is the question we get most — why any of this is written in Rust.

The vision

Empowering the next generation of AI everywhere — uncompromised performance, unparalleled accessibility.

We want a world where machine learning is genuinely ubiquitous: from the smallest embedded sensor to the largest distributed cluster, every device able to run AI efficiently, safely, and without asking permission from a vendor.

Three things follow from that, and they are the standard we hold the codebase to:

• Safety meets scalability. Memory-safe, concurrent and extensible by design, so that pushing performance does not mean trading away reliability.
• Every device is an AI device. From microcontrollers to data centres, performance limited by the silicon rather than by legacy software bloat.
• Innovation is democratised. Open development, a permissive licence, and a backend interface anyone can implement for their own hardware.

The gap this addresses is a real one. The frameworks that dominate today are excellent at what they were built for — large-scale training and serving on homogeneous infrastructure you control. They are a poor fit for a camera on a production line, a drone with a power budget, or an inspection rig that has to run unattended for three years. The closest thing to a good answer has historically been TensorFlow Lite, and it is written in C.

The mission

teenygrad is a high-performance Rust ML training and inference library in the spirit of PyTorch and tinygrad. Concretely, we are building for:

• Memory safety and concurrency, without a runtime tax.
• A low memory footprint — small enough to run on the smallest devices.
• Statically typed kernels — the ML algorithms and the GPU kernels are both checked by the compiler.
• No performance compromises — hardware-accelerated wherever the hardware allows it.
• Broad hardware support — not only NVIDIA and AMD.
• An extensible backend — so a vendor or a team can bring up their own accelerator without forking the framework.
• no_std compatibility in the core training and inference path.
• Full async support, including on embedded targets.
• Multi-threaded by default.

Why Rust

This is a design decision with consequences in every direction, so it is worth arguing properly rather than asserting.

Memory safety without a garbage collector

Edge and embedded AI is still overwhelmingly C and C++. That is not an accident — those languages give you the control over layout, allocation and lifetime that the domain demands. It is also the reason a large share of the CVEs in deployed device software are memory-safety failures: use-after-free in a teardown path, an off-by-one in a buffer of image data, a data race in a producer/consumer pipeline that only shows up on a busy machine.

The usual escape from that class of bug is a managed runtime, and a managed runtime is disqualifying here. Garbage collection means non-deterministic pauses and heap headroom you have to over-provision. If your inspection camera has a 30 ms frame budget, a collector that occasionally takes 40 ms is not a performance problem; it is a correctness problem.

Rust removes the entire class of bug at compile time, with no collector, no runtime and no pauses. Send and Sync extend the same guarantee to concurrency, which matters more than it first appears: the interesting work in an inference pipeline is overlapping capture, preprocessing, kernel launches and postprocessing across cores, and that is exactly the code where data races are easiest to write and hardest to reproduce.

The device is not a workstation

teeny-core is no_std by default — std is an opt-in feature, not something you have to fight your way out of. The tensor types, the computational graph and the layer definitions are all built on that basis, because a design that assumes an allocator, a filesystem and threads is very hard to walk back later.

That matters because of what the alternative actually looks like on a device. The conventional edge path is roughly:

PyTorch → ONNX export → engine build → vendor runtime → Python dependencies → device

Five handoffs, five artifacts, and — more to the point — five version pairs you have to keep matched in the field. Every dashed line in that chain is a place a deployment breaks for reasons that have nothing to do with your model. Anyone who has debugged a mismatch between an ONNX opset, an engine built on a different minor driver version, and a Python wheel that silently pulled a new numpy knows the failure mode.

The teenygrad path is one language and one compiler:

Rust → MIR → MLIR → Triton → LLVM IR → PTX → binary

One artifact. The interpreter, the packages and the version drift stay on your workstation, where they are somebody’s problem during CI rather than at 3am on a customer site.

Portability without rewriting the model

The backend is a trait. Kernels are written generically over it, so retargeting is a change of type parameter rather than a rewrite. This is a real kernel from teeny-kernels, not a simplification:

#[kernel]
pub fn matmul_forward<T: Triton, D: Float, const BLOCK_K: i32>(
    a_ptr: T::Pointer<D>,
    b_ptr: T::Pointer<D>,
    c_ptr: T::Pointer<D>,
    M: i32,
    N: i32,
    K: i32,
) {
    let pid = T::program_id(Axis::X);
    let n = pid % N;
    let m = pid / N;
    if m >= M {
        return;
    }

    let k_offsets = T::arange(0, BLOCK_K);
    // Block-tiled loads, accumulate, store — plain Rust.
}

T is the backend, D is the dtype, and BLOCK_K is a tile size fixed at compile time. The pointers are pointers rather than tensors because that keeps the generated code close to what the GPU actually wants. Nothing in the body names CUDA. When SPIR-V lands, this kernel does not change.

Cross-compilation gets the same treatment. Targeting a Jetson from an x86 workstation is rustup target add aarch64-unknown-linux-gnu and a build flag — the same mechanism the rest of the Rust ecosystem uses, not a bespoke second build system bolted on for the model.

Static typing all the way down to the GPU

The kernel above is checked by the Rust compiler. Dtype mismatches, tile sizes that do not divide, pointer arithmetic that does not typecheck — these are build failures on your workstation, not a launch failure on a board in a cabinet you need a ladder to reach. Const generics carry tile sizes into the type system, so the shapes the compiler reasons about are the shapes the kernel runs with.

This is the part of the design that pays back most over time. The dynamic, runtime-checked approach is genuinely better for research iteration, and worse for everything that happens after the model is chosen.

No vendor toolchain lock-in

The lowering path has nothing proprietary in the middle of it. MLIR, Triton and LLVM are open, the compiler is Apache 2.0, and the backend interface is a public trait rather than a plugin API we control access to. If you are bringing up an accelerator, you implement the trait; you do not negotiate.

The practical version of this argument: your CI is cargo test. Your debugger is the one you already use. Your static analysis is the borrow checker and clippy, running over the model code and the kernel code alike, because they are the same language. There is no second toolchain with its own version policy, its own licence server, and its own opinion about which driver you are allowed to install.

What is in 0.1.1

Twelve crates, all Apache 2.0, all on crates.io:

Crate	What it does
teeny-core	Tensors, computational graph, dtypes, nn layers, device abstraction. no_std by default.
teeny-macros	The #[kernel] attribute macro.
teeny-triton	The Triton-like kernel DSL that kernels are written against.
teeny-kernels	The kernel library — GEMM, convolution, pooling, norms, attention, losses, fused ops.
teeny-compiler	Lowers a traced graph to LLVM/MLIR object code, with a CPU path behind the ndarray feature.
teeny-cuda	The CUDA backend — driver bindings, device/runtime abstraction, AOT and JIT kernel compilation.
teeny-onnx	Parses .onnx protobuf into a teenygrad graph.
teeny-vision	Vision model definitions and datasets; MNIST and LeNet-5 today.
teeny-data	Dataset loading — download, CSV, safetensors, memory-mapped access.
teeny-quant	Weight-only post-training quantization to INT8/INT4/FP8.
teeny-cli	Ahead-of-time compilation of a model’s kernels for a given device and config.
teeny-llm	An early LLM serving application. The kernel work underneath it is not finished.

The kernel library covers what a vision model needs end to end: GEMM and matmul with backward passes, conv1d/2d/3d, the pooling family (max, average and Lp, in one, two and three dimensions), the normalisation family (batch, group, instance, layer and RMS), FlashAttention-2 generic over Float, the common losses, elementwise and reduction primitives, and a fused conv2d + batchnorm + SiLU in both tiled and GEMM formulations.

Around that: dtype-aware kernel dispatch, ahead-of-time kernel compilation with a kernel cache that defaults next to the deployed package, teenyc auto-detection through rustup so TEENYC_PATH is optional, and ONNX import including FlexAttention, LinearAttention and CausalConvWithState. teeny-quant reads and writes .safetensors following the compressed-tensors convention, and has been validated against Ultralytics YOLO checkpoints.

Every publishable crate has documented public API with #![warn(missing_docs)] enforced, CI runs -D warnings and cargo fmt --check, and the generated docs are at docs.teenygrad.org.

What works today

• The Rust → PTX pipeline, end to end: MIR through to loadable GPU code.
• Object detection inference on a Jetson Orin Nano — YOLO26, verified bounding boxes, running as the parking-garage demo on the landing page.
• Training: MNIST and LeNet-5 to 97%.
• Cross-compilation from x86 to aarch64 against CUDA 12.6.

Numbers

On a Jetson Orin Nano Super, relative inference latency, lower is better:

Runtime	Relative latency
TensorRT	1.0×
ONNX Runtime	2.8×
teenygrad	7.0×

The caveats are load-bearing. This is one model on one board — YOLO26 on a Jetson Orin Nano Super, MAXN SUPER power mode, JetPack 6.2.2. It is not a claim about GEMM throughput in general, and it is not a claim about any workload we have not measured. Reproduction steps are in the repo, and we would rather you ran them than took our word for it.

What does not work yet

The same list is on the status section of the landing page, and it stays there because a roadmap that outruns the code is worse than no roadmap:

• Detection model training compiles and runs, but is not yet validated.
• One-command build and deploy to a board is still a manual copy step.
• Transformer detection (RF-DETR) compiles but has not been run.
• Language models are blocked on kernel work in progress.
• Non-NVIDIA targets are not usable yet.

If you are evaluating teenygrad for something real today, the honest scope is: CNN-family vision inference on NVIDIA hardware, including Jetson, with training that works for small models and is unproven for large ones.

Roadmap

Next. Spacemit K3 — RISC-V with AI extensions — is under active development and due in Q4 2026. Finishing one-command deploy so that getting a build onto a board is not a manual copy. Validating detection training rather than merely running it.

After that. SPIR-V and ARM GPUs, which together are what “no vendor lock-in” has to mean in practice rather than in principle. Transformer detection (RF-DETR) actually run rather than merely compiled. The LLM kernel work that teeny-llm is waiting on.

Further out. Static activation quantization, which needs calibration through the ONNX and compiler path and so depends on work above it. Deeper sparsity support. Observability and metrics for models in the field. Pushing the no_std core down onto genuinely small devices — the microcontroller end of “every device is an AI device” is the part of the vision we have designed for and not yet delivered.

The ordering is not fixed, and it responds to who is waiting. If the board you care about is not on the hardware page, tell us what it is — that is genuinely how the queue gets sorted.

Getting started

cargo install cargo-teeny
cargo teeny build --release

For a Jetson, name the target:

cargo teeny build --target orin-nano --release

Documentation is at docs.teenygrad.org, the source is on GitHub, and the Discord is where the conversation happens. Contributions are welcome from individuals and companies alike; CONTRIBUTING.md in the repo is the place to start.

The test we hold ourselves to has not changed: if shipping a model requires a Python runtime on the device, we have not finished the job. 0.1.1 is the first version where that claim is checkable rather than aspirational. Go and check it.