It is a useful demo because it is shaped like a real deployment rather than a benchmark. Two processes, a WebSocket between them, a browser at the end, and a detection model doing work on every frame. Underneath, YOLO26-N compiled from Rust to PTX by teenygrad — no Python anywhere in the pipeline.

How it fits together:

• parking-garage-server walks a parking-lot dataset, runs YOLO26 over each frame, decides which bays are occupied, and broadcasts a snapshot over a WebSocket every two seconds. Port 3001.
• parking-garage-webapp serves the Vue page the browser loads. Port 3000.
• The page opens a WebSocket back to the server and draws each frame with its bays overlaid.

The first eight steps run everything on a workstation. The last four put it on a Jetson Orin Nano.

Everything here is against vision-rs at branch release/0.1.2.

Before you start

Check the two people most often get wrong:

nvcc --version
nvidia-smi

If nvcc is missing but nvidia-smi works, you have a driver but no toolkit. Install the toolkit first — the kernel compiler needs the headers.

Step 1 — Clone the repository

git clone https://github.com/teenygrad/vision-rs
cd vision-rs
git checkout release/0.1.2

The demo lives in demos/parking_garage/, as a member of the vision-rs workspace. You will build it from the repository root throughout — cargo teeny resolves the workspace’s path dependencies and needs the root’s Cross.toml, so it refuses to run from inside the member directory.

Step 2 — Install cargo-teeny and the teenyc toolchain

vision-rs builds on stable Rust. The GPU kernels do not — they are compiled by teenyc, a custom compiler fork, both ahead of time and via JIT at runtime. You need that binary before anything using the cuda feature will build, and cuda is what makes this demo do inference at all.

You do not have to build the compiler from source. cargo-teeny fetches a prebuilt release:

cargo install --git https://github.com/teenygrad/cargo-teeny --branch release/0.1.1
cargo teeny install-toolchain

Take the --branch seriously. Packaging this demo needs --package and --features, and those flags only exist on release/0.1.1 — cargo-teeny’s default branch does not have them yet, and step 10 will fail with an unexpected argument error if you install from it.

install-toolchain downloads the compiler and rustup toolchain links it as stable-teenyc-x86_64-unknown-linux-gnu. Verify both halves landed:

rustup toolchain list | grep teeny
rustup run stable-teenyc-x86_64-unknown-linux-gnu teenyc --version

If the first prints nothing, the link step did not happen, and every later build fails with a missing-teenyc error rather than anything obviously toolchain-shaped.

Step 3 — Configure the environment

Copy the template and point the cache paths wherever you want models to live:

cp .env.dev .env

export DATASETS_CACHE_DIR=$HOME/.cache/vision-rs/datasets
export MODELS_CACHE_DIR=$HOME/.cache/vision-rs/models

MODELS_CACHE_DIR is the one this demo requires — the server downloads YOLO26 weights into it on first run. Leave TEENYC_PATH commented out unless you built your own compiler; the rustup-linked toolchain is auto-detected.

Then, in every shell you build or run from:

source .env

Forgetting this is the single most common failure. It surfaces as MODELS_CACHE_DIR not set the moment you pass --model.

Step 4 — Get the parking lot dataset

The demo reads PKLot — a public dataset of parking lot photographs from UFPR, each image paired with an XML file marking every bay and whether it is occupied. Those XML annotations are what the demo scores itself against.

There is no downloader in the repository; fetch PKLot yourself and unpack it. What matters is the directory shape, because the server walks exactly this nesting:

<dataset_root>/
  <lot>/              # e.g. PUC, UFPR04, UFPR05
    <weather>/        # e.g. Cloudy, Rainy, Sunny
      <date>/         # e.g. 2012-09-11
        <frame>.jpg
        <frame>.xml   # same stem as the .jpg

A .jpg with no matching .xml is skipped silently, and a lot directory that yields no usable images is skipped with a warning. If the server reports no lots found under ..., you have pointed it one level too high or too low — the root is the directory that contains the lot directories.

The full dataset is around 12 GB. You do not need all of it: a single lot, or even a single date directory promoted to the right depth, is enough to see the demo work.

Step 5 — Build the demo

Both binaries live in the parking-garage workspace member, and inference is behind the cuda feature:

source .env
cargo build --release -p parking-garage --features cuda

The first build compiles the whole teenygrad stack and takes a while. You end up with two binaries in target/release/: parking-garage-server and parking-garage-webapp.

Without --features cuda the demo still builds and runs, but --model becomes an error — you get the dataset’s ground-truth annotations replayed into the browser and no inference at all. That is a reasonable way to check the plumbing before involving the GPU.

Step 6 — Start the server

source .env
./target/release/parking-garage-server /path/to/PKLot \
  --model ultralytics/yolo26n

The dataset root is positional; leave it off and it defaults to /mnt/data1/datasets/PKLot/PKLot, which is almost certainly not where yours is. --port moves it off 3001.

Three things happen on first run, in order, and the middle one takes a while:

1. The dataset is scanned — it prints one line per lot with an image count.
2. Weights are downloaded — YOLO26-N as pre-converted safetensors from Hugging Face, cached under $MODELS_CACHE_DIR. Only once.
3. Kernels are compiled — you will see compiling YOLO26N (nc=80, 640×640) — first run builds kernel cache …. Every kernel in the network is being compiled for your specific GPU. Subsequent runs read the cache and start in seconds.

Then it settles into a two-second tick, and the API is up on http://0.0.0.0:3001 with the WebSocket at /api/ws. There is a health check at /api/health if you want to confirm it from another terminal.

Step 7 — Start the webapp and open it

In a second terminal, from the repository root:

./target/release/parking-garage-webapp

It listens on 0.0.0.0:3000 and serves demos/parking_garage/ui/dist. That path is resolved relative to your working directory, so run it from the repo root or it will serve nothing — this matters again in step 12.

Pass a different listen address as the first argument if 3000 is taken:

./target/release/parking-garage-webapp 127.0.0.1:8080

Then open http://localhost:3000.

The page connects to ws://<the host you loaded the page from>:3001/api/ws. On one machine that resolves itself. If your server is elsewhere, or on a non-default port, override it with a query parameter:

http://localhost:3000/?ws=ws://192.168.1.50:3001/api/ws

A status dot in the corner tells you where you stand: amber connecting, green connected with the URL beside it, red on error. It retries on its own, so starting the webapp before the server is fine.

One caveat: the page loads Vue from a CDN, so the browser needs internet access even though nothing else does.

Step 8 — Read what it is telling you

The interesting output is in the server’s terminal, not the browser. With --model set, each tick prints a line per lot:

[  UFPR04] 2012-09-11_15_16_58.jpg  gt=22/28 inf=21/28 18.4ms

• gt — occupied bays according to the dataset’s XML annotation. Ground truth, hand-labelled.
• inf — occupied bays according to YOLO26 on this frame.
• ms — how long that inference took, wall clock.

The occupancy rule is deliberately simple: the model detects vehicles — COCO classes car, bus and truck — and a bay counts as occupied when the centre of any detected vehicle falls inside it. No tracking, no temporal smoothing, one frame at a time.

Watching gt and inf track each other frame to frame is the real content of the demo. They will not always agree, and the disagreements are the interesting part: a bay half-occluded by the vehicle in front, a motorcycle that is not one of the three vehicle classes, a car straddling a line so its centre lands in the neighbouring bay.

Worth being clear about one thing: the boxes drawn in the browser are the ground-truth annotations, not the detections. The comparison lives in the log.

Everything so far ran on a workstation with a CUDA toolkit and a kernel compiler installed. The rest puts the same demo on a Jetson Orin Nano that has neither.

Step 9 — Set up cross-compilation

Two one-time additions on the host:

cargo install cross --git https://github.com/cross-rs/cross
rustup target add aarch64-unknown-linux-gnu

Third, the aarch64 CUDA libraries for your board’s JetPack version. These are not your host’s CUDA — they are the cross-compilation variant, from NVIDIA’s CUDA Toolkit downloads for your JetPack release. The build mounts them into the cross container so the binary links against exactly the CUDA your Jetson runs. JetPack 6.2 puts them at:

/usr/local/cuda-12.6/targets/aarch64-linux

If yours differs, pass --cuda-path <path> on every build and package command.

Step 10 — Package both binaries

package cross-compiles for the board and ahead-of-time compiles the GPU kernels on the host, writing them into the bundle. That second half is the point: there is no teenyc on a Jetson, so a binary that tried to JIT its kernels there would fail outright.

--bin is repeatable, so the server and the webapp go into one bundle sharing a single kernel cache:

source .env
cargo teeny package \
  --target jetson-orin-nano \
  --package parking-garage \
  --bin parking-garage-server \
  --bin parking-garage-webapp \
  --features cuda \
  --dest ./dist/parking-garage-orin \
  --device cuda \
  --options "capability=sm_87,ptx-version=82"

Run it from the repository root, not from demos/parking_garage — that is what --package is for.

capability=sm_87 is the Orin Nano’s compute capability (Ampere). ptx-version=82 overrides teenyc’s conservative default PTX ISA floor for sm_87; keep it pinned unless your device is on a materially different CUDA version.

You get:

dist/parking-garage-orin/
  bin/parking-garage-server
  bin/parking-garage-webapp
  cache/            # AOT-compiled GPU kernels, shared by both binaries
  conf/             # provenance: target, device, options, commit, build time
  data/             # empty — you populate this

The binaries find cache/ as their sibling at runtime, so nothing needs setting on the device for kernels to resolve.

Step 11 — Deploy to the board

cargo teeny deploy \
  --package ./dist/parking-garage-orin \
  --host <user>@<orin-host> \
  --dest /home/<user>/parking-garage

It is rsync -a over SSH. Set up key-based auth first if you would rather not type a password, though the interactive prompt works. A re-run copies only what is missing, so it is safe to resume a partial transfer — pass --overwrite to force a full re-sync after rebuilding. Use --ssh "ssh -p <port>" for a non-default SSH port.

package bundles binaries and kernels, and nothing else. Two things the demo needs are therefore still on your host, and you copy them yourself:

# The web UI the webapp serves — it looks for ./ui/dist relative to its cwd
rsync -a demos/parking_garage/ui <user>@<orin-host>:/home/<user>/parking-garage/

# Enough of the dataset to tick through
rsync -a /path/to/PKLot/UFPR04 <user>@<orin-host>:/home/<user>/parking-garage/data/PKLot/

Send one lot rather than all 12 GB unless you have a reason not to. Keep the lot / weather / date nesting from step 4 intact — the server walks it the same way on the device.

Step 12 — Run it on the Jetson

ssh <user>@<orin-host>
cd /home/<user>/parking-garage
export MODELS_CACHE_DIR=$HOME/.cache/vision-rs/models
./bin/parking-garage-server ./data/PKLot --model ultralytics/yolo26n

The weights download on first run and need internet on the board. If it has none, rsync your host’s $MODELS_CACHE_DIR across instead — the layout is identical.

In a second SSH session, from the same directory, so that ui/dist resolves:

cd /home/<user>/parking-garage
./bin/parking-garage-webapp

Then, from your laptop, open http://<orin-host>:3000. The page derives its WebSocket URL from the host you loaded it from, so it finds the server on 3001 without configuration.

Worth pausing on what is not installed on that Jetson. No Rust. No CUDA toolkit. No Python. No ONNX runtime, no engine builder, no vendor inference server. One directory holding two binaries, their kernels, a web page and some JPEGs.

The latency figures in the server log will be higher than on your workstation — that is the point of measuring them there.

When it goes wrong

Where to go next

The yolo26 example in the same repository is the lower-level view of the same model: view for an inference viewer with bounding boxes, verify for mAP over a validation split, bench for throughput and latency across batch sizes, and train for training on your own data. --help on any of them is accurate.

If you want the library rather than either demo, vision_rs::detect is the entry point — an ObjectDetector/DetectorConfig pair that takes JPEG or PNG bytes and hands back labelled, NMS-filtered detections in about ten lines. The reference is at docs.teenygrad.org/api/vision-rs.

And if you get stuck, the Discord is the fastest route to an answer.

What it is: a Rust-native ML compiler and runtime. You write a model and its kernels in Rust, and teenyc lowers them through MIR, MLIR, Triton and LLVM IR to PTX. What lands on the device is a binary. There is no interpreter on the board, no ONNX round-trip at deploy time, no vendor runtime to keep matched against a driver.

This post covers where the project is going, what is in the release, what is demonstrably not finished, and — since it is the question we get most — why any of this is written in Rust.

The vision

Empowering the next generation of AI everywhere — uncompromised performance, unparalleled accessibility.

We want a world where machine learning is genuinely ubiquitous: from the smallest embedded sensor to the largest distributed cluster, every device able to run AI efficiently, safely, and without asking permission from a vendor.

Three things follow from that, and they are the standard we hold the codebase to:

• Safety meets scalability. Memory-safe, concurrent and extensible by design, so that pushing performance does not mean trading away reliability.
• Every device is an AI device. From microcontrollers to data centres, performance limited by the silicon rather than by legacy software bloat.
• Innovation is democratised. Open development, a permissive licence, and a backend interface anyone can implement for their own hardware.

The gap this addresses is a real one. The frameworks that dominate today are excellent at what they were built for — large-scale training and serving on homogeneous infrastructure you control. They are a poor fit for a camera on a production line, a drone with a power budget, or an inspection rig that has to run unattended for three years. The closest thing to a good answer has historically been TensorFlow Lite, and it is written in C.

The mission

teenygrad is a high-performance Rust ML training and inference library in the spirit of PyTorch and tinygrad. Concretely, we are building for:

• Memory safety and concurrency, without a runtime tax.
• A low memory footprint — small enough to run on the smallest devices.
• Statically typed kernels — the ML algorithms and the GPU kernels are both checked by the compiler.
• No performance compromises — hardware-accelerated wherever the hardware allows it.
• Broad hardware support — not only NVIDIA and AMD.
• An extensible backend — so a vendor or a team can bring up their own accelerator without forking the framework.
• no_std compatibility in the core training and inference path.
• Full async support, including on embedded targets.
• Multi-threaded by default.

Why Rust

This is a design decision with consequences in every direction, so it is worth arguing properly rather than asserting.

Memory safety without a garbage collector

Edge and embedded AI is still overwhelmingly C and C++. That is not an accident — those languages give you the control over layout, allocation and lifetime that the domain demands. It is also the reason a large share of the CVEs in deployed device software are memory-safety failures: use-after-free in a teardown path, an off-by-one in a buffer of image data, a data race in a producer/consumer pipeline that only shows up on a busy machine.

The usual escape from that class of bug is a managed runtime, and a managed runtime is disqualifying here. Garbage collection means non-deterministic pauses and heap headroom you have to over-provision. If your inspection camera has a 30 ms frame budget, a collector that occasionally takes 40 ms is not a performance problem; it is a correctness problem.

Rust removes the entire class of bug at compile time, with no collector, no runtime and no pauses. Send and Sync extend the same guarantee to concurrency, which matters more than it first appears: the interesting work in an inference pipeline is overlapping capture, preprocessing, kernel launches and postprocessing across cores, and that is exactly the code where data races are easiest to write and hardest to reproduce.

The device is not a workstation

teeny-core is no_std by default — std is an opt-in feature, not something you have to fight your way out of. The tensor types, the computational graph and the layer definitions are all built on that basis, because a design that assumes an allocator, a filesystem and threads is very hard to walk back later.

That matters because of what the alternative actually looks like on a device. The conventional edge path is roughly:

PyTorch → ONNX export → engine build → vendor runtime → Python dependencies → device

Five handoffs, five artifacts, and — more to the point — five version pairs you have to keep matched in the field. Every dashed line in that chain is a place a deployment breaks for reasons that have nothing to do with your model. Anyone who has debugged a mismatch between an ONNX opset, an engine built on a different minor driver version, and a Python wheel that silently pulled a new numpy knows the failure mode.

The teenygrad path is one language and one compiler:

Rust → MIR → MLIR → Triton → LLVM IR → PTX → binary

One artifact. The interpreter, the packages and the version drift stay on your workstation, where they are somebody’s problem during CI rather than at 3am on a customer site.

Portability without rewriting the model

The backend is a trait. Kernels are written generically over it, so retargeting is a change of type parameter rather than a rewrite. This is a real kernel from teeny-kernels, not a simplification:

#[kernel]
pub fn matmul_forward<T: Triton, D: Float, const BLOCK_K: i32>(
    a_ptr: T::Pointer<D>,
    b_ptr: T::Pointer<D>,
    c_ptr: T::Pointer<D>,
    M: i32,
    N: i32,
    K: i32,
) {
    let pid = T::program_id(Axis::X);
    let n = pid % N;
    let m = pid / N;
    if m >= M {
        return;
    }

    let k_offsets = T::arange(0, BLOCK_K);
    // Block-tiled loads, accumulate, store — plain Rust.
}

T is the backend, D is the dtype, and BLOCK_K is a tile size fixed at compile time. The pointers are pointers rather than tensors because that keeps the generated code close to what the GPU actually wants. Nothing in the body names CUDA. When SPIR-V lands, this kernel does not change.

Cross-compilation gets the same treatment. Targeting a Jetson from an x86 workstation is rustup target add aarch64-unknown-linux-gnu and a build flag — the same mechanism the rest of the Rust ecosystem uses, not a bespoke second build system bolted on for the model.

Static typing all the way down to the GPU

The kernel above is checked by the Rust compiler. Dtype mismatches, tile sizes that do not divide, pointer arithmetic that does not typecheck — these are build failures on your workstation, not a launch failure on a board in a cabinet you need a ladder to reach. Const generics carry tile sizes into the type system, so the shapes the compiler reasons about are the shapes the kernel runs with.

This is the part of the design that pays back most over time. The dynamic, runtime-checked approach is genuinely better for research iteration, and worse for everything that happens after the model is chosen.

No vendor toolchain lock-in

The lowering path has nothing proprietary in the middle of it. MLIR, Triton and LLVM are open, the compiler is Apache 2.0, and the backend interface is a public trait rather than a plugin API we control access to. If you are bringing up an accelerator, you implement the trait; you do not negotiate.

The practical version of this argument: your CI is cargo test. Your debugger is the one you already use. Your static analysis is the borrow checker and clippy, running over the model code and the kernel code alike, because they are the same language. There is no second toolchain with its own version policy, its own licence server, and its own opinion about which driver you are allowed to install.

What is in 0.1.1

Twelve crates, all Apache 2.0, all on crates.io:

The kernel library covers what a vision model needs end to end: GEMM and matmul with backward passes, conv1d/2d/3d, the pooling family (max, average and Lp, in one, two and three dimensions), the normalisation family (batch, group, instance, layer and RMS), FlashAttention-2 generic over Float, the common losses, elementwise and reduction primitives, and a fused conv2d + batchnorm + SiLU in both tiled and GEMM formulations.

Around that: dtype-aware kernel dispatch, ahead-of-time kernel compilation with a kernel cache that defaults next to the deployed package, teenyc auto-detection through rustup so TEENYC_PATH is optional, and ONNX import including FlexAttention, LinearAttention and CausalConvWithState. teeny-quant reads and writes .safetensors following the compressed-tensors convention, and has been validated against Ultralytics YOLO checkpoints.

Every publishable crate has documented public API with #![warn(missing_docs)] enforced, CI runs -D warnings and cargo fmt --check, and the generated docs are at docs.teenygrad.org.

What works today

• The Rust → PTX pipeline, end to end: MIR through to loadable GPU code.
• Object detection inference on a Jetson Orin Nano — YOLO26, verified bounding boxes, running as the parking-garage demo on the landing page.
• Training: MNIST and LeNet-5 to 97%.
• Cross-compilation from x86 to aarch64 against CUDA 12.6.

Numbers

On a Jetson Orin Nano Super, relative inference latency, lower is better:

The caveats are load-bearing. This is one model on one board — YOLO26 on a Jetson Orin Nano Super, MAXN SUPER power mode, JetPack 6.2.2. It is not a claim about GEMM throughput in general, and it is not a claim about any workload we have not measured. Reproduction steps are in the repo, and we would rather you ran them than took our word for it.

What does not work yet

The same list is on the status section of the landing page, and it stays there because a roadmap that outruns the code is worse than no roadmap:

• Detection model training compiles and runs, but is not yet validated.
• One-command build and deploy to a board is still a manual copy step.
• Transformer detection (RF-DETR) compiles but has not been run.
• Language models are blocked on kernel work in progress.
• Non-NVIDIA targets are not usable yet.

If you are evaluating teenygrad for something real today, the honest scope is: CNN-family vision inference on NVIDIA hardware, including Jetson, with training that works for small models and is unproven for large ones.

Roadmap

Next. Spacemit K3 — RISC-V with AI extensions — is under active development and due in Q4 2026. Finishing one-command deploy so that getting a build onto a board is not a manual copy. Validating detection training rather than merely running it.

After that. SPIR-V and ARM GPUs, which together are what “no vendor lock-in” has to mean in practice rather than in principle. Transformer detection (RF-DETR) actually run rather than merely compiled. The LLM kernel work that teeny-llm is waiting on.

Further out. Static activation quantization, which needs calibration through the ONNX and compiler path and so depends on work above it. Deeper sparsity support. Observability and metrics for models in the field. Pushing the no_std core down onto genuinely small devices — the microcontroller end of “every device is an AI device” is the part of the vision we have designed for and not yet delivered.

The ordering is not fixed, and it responds to who is waiting. If the board you care about is not on the hardware page, tell us what it is — that is genuinely how the queue gets sorted.

Getting started

cargo install cargo-teeny
cargo teeny build --release

For a Jetson, name the target:

cargo teeny build --target orin-nano --release

Documentation is at docs.teenygrad.org, the source is on GitHub, and the Discord is where the conversation happens. Contributions are welcome from individuals and companies alike; CONTRIBUTING.md in the repo is the place to start.

The test we hold ourselves to has not changed: if shipping a model requires a Python runtime on the device, we have not finished the job. 0.1.1 is the first version where that claim is checkable rather than aspirational. Go and check it.