From CUDA to Axelera

Migrating GPU Inference Pipelines to Edge AI with Voyager SDK

EDGE AI VOYAGER SDK METIS AIPU

Swipe or use arrow keys to navigate →

CUDA vs Axelera — At a Glance

NVIDIA CUDA

🖥️ Datacenter & workstation GPUs
⚡ Training + inference
🔥 150–350W per GPU
💰 $500–$40,000 per card
🔧 C++/Python + CUDA kernels
📦 TensorRT optimization
🔒 NVIDIA-only ecosystem

VS

Axelera Metis AIPU

📱 Edge devices & embedded
⚡ Inference-optimized
🔋 5–25W per card
💰 $50–$600 per card
🔧 YAML pipelines + deploy.py
📦 Auto-quantize & compile
🔓 Open-source SDK on GitHub

Performance per Watt: The Real Metric

Axelera Metis delivers datacenter-class throughput at edge power budgets

214

TOPS (single chip)

856

TOPS (4-chip PCIe)

3,200

FPS ResNet-50

834

FPS YOLOv8n

Model	Metis PCIe FPS	Accuracy Loss	Form Factor
YOLOv8n (640×640)	834	1.18%	M.2 / PCIe
YOLOv8s (640×640)	643	0.93%	M.2 / PCIe
ResNet-50 (224×224)	1,946	0.18%	M.2 / PCIe
EfficientNet-B0	1,429	1.12%	M.2 / PCIe
MobileNetV2	3,670	1.50%	M.2 / PCIe
YOLO11n (640×640)	759	0.71%	M.2 / PCIe

When to Migrate? Decision Matrix

✅ Migrate When:

✓ Deploying CV inference at the edge
✓ Scaling multi-camera analytics (24+ streams)
✓ Power budget < 25W per inference node
✓ Shipping hundreds/thousands of devices (BOM cost)
✓ Models are YOLO, ResNet, MobileNet, EfficientNet, etc.
✓ Simplifying DevOps (no CUDA driver management)
✓ Need M.2 / fanless / embedded form factor

❌ Stay on CUDA When:

✗ Training models (Axelera = inference only)
✗ Models > 16 GB (need A100/H100 VRAM)
✗ Datacenter-scale LLMs (70B+ params)
✗ Heavy custom CUDA kernels throughout
✗ Deep TensorRT/cuDNN/NCCL dependencies
✗ R&D / rapid prototyping phase
✗ Generative AI (diffusion, large transformers)

Advantages of Upgrading to Axelera

🔋

10-50× Better Perf/Watt

214 TOPS at ~15W vs 300W for datacenter GPUs. Run AI without cooling infrastructure.

💰

10× Lower BOM Cost

M.2 card at ~$150 vs $2,000+ GPU. Massive savings at volume deployment.

📐

Edge Form Factors

M.2, PCIe half-height, SBC. Fits in cameras, gateways, industrial equipment.

🧑‍💻

YAML, Not CUDA

Declare pipelines in YAML. No memory management, no kernel launches, no stream sync.

🔄

Auto-Quantization

INT8 quantization handled by compiler. No TensorRT calibrator or manual tuning needed.

🔓

Open Source SDK

Voyager SDK on GitHub. No vendor lock-in. GStreamer-based, API-compatible with standards.

The Migration Process — 8 Steps

1

Analyze
CUDA Code

→

2

Export
to ONNX

→

3

Check ONNX
Op Compat

→

4

Generate
YAML Pipeline

→

5

Configure
Model YAML

→

6

Deploy &
Compile

→

7

Benchmark
& Validate

→

8

Integrate
& Ship

Each step is detailed in the following slides

STEPS 1–2

Analyze & Export to ONNX

PyTorch Export

import torch

model.eval()
dummy = torch.randn(1, 3, 640, 640)
torch.onnx.export(
    model, dummy, "model.onnx",
    opset_version=17,  # Axelera recommended
    input_names=["images"],
    output_names=["output"]
)

Ultralytics YOLO

# One command export
yolo export model=best.pt \
     format=onnx opset=17

# Then simplify (recommended)
pip install onnxsim
onnxsim model.onnx model_sim.onnx

Verify Operators

import onnx
m = onnx.load("model.onnx")
ops = sorted({n.op_type for n in m.graph.node})
print("Operators:", ops)

STEP 3

ONNX Operator Compatibility

AIPU-accelerated operators run on hardware • Others auto-fallback to CPU

✅ Fully Supported

BatchNormalization, GlobalAveragePool, GlobalMaxPool, HardSwish, LeakyRelu, Relu, Sigmoid, Tanh

⚡ Constrained (check rules)

Add, AveragePool, Clip, Concat, Conv, ConvTranspose, Flatten, Gemm, HardSigmoid, MatMul, MaxPool, Mul, PRelu, Pad, Reshape, Resize, Selu, Slice, Softmax, Split, Squeeze, Sub, Transpose

🔄 Auto CPU Fallback

Any unsupported operator is automatically extracted into preamble.onnx or postamble.onnx and runs on the host CPU — no manual work required!

# Compiler handles this automatically:
Model ONNX
  → preamble.onnx  # CPU pre-ops
  → model.axm      # AIPU core
  → postamble.onnx # CPU post-ops

STEP 4

CUDA Preprocessing → YAML Pipeline

Mapping Table

CUDA / PyTorch	Voyager YAML
`transforms.Resize()`	`resize: {w, h}`
`transforms.CenterCrop()`	`centercrop: {w, h}`
YOLO letterbox	`letterbox: {w, h}`
`transforms.ToTensor()`	`torch-totensor:`
`transforms.Normalize()`	`normalize: {mean, std}`
`img/127.5 - 1`	`linear-scaling:`
`cv2.cvtColor(BGR2RGB)`	`convert-color:`
Custom CUDA kernel	AxOperator plugin

Example: YOLO Pipeline

pipeline:
  - detections:
      model_name: yolov8n
      input:
        type: image
      preprocess:
        - letterbox:
            height: 640
            width: 640
            scaleup: true
        - torch-totensor:
      inference:
        handle_all: true
      postprocess:
        - decodeyolo:
            conf_threshold: 0.25
            nms_iou_threshold: 0.45

STEPS 5–6

Configure & Deploy

Model Configuration

models:
  my-model:
    class: AxONNXModel
    class_path: $AXELERA_FRAMEWORK/
      ax_models/base_onnx.py
    weight_path: /path/to/model.onnx
    task_category: ObjectDetection
    input_tensor_layout: NCHW
    input_tensor_shape: [1, 3, 640, 640]
    input_color_format: RGB
    num_classes: 80
    extra_kwargs:
      compiler_config_file: yolov8s.toml

Deploy Commands

# Compile for Metis AIPU
./deploy.py my-model.yaml

# Better accuracy (more cal images)
./deploy.py my-model.yaml \
  --num-cal-images=400

# Multi-core deployment
./deploy.py my-model.yaml \
  --aipu-cores 4

Output Artifacts

build/my-model/quantized/
├── model.axm         # AIPU binary
├── preamble.onnx     # CPU pre-ops
├── postamble.onnx    # CPU post-ops
├── manifest.json     # Metadata
└── quantization_params.json

Tools, Compilers & Infrastructure

⚙️

deploy.py

One-command compilation. Quantizes FP32→INT8, compiles for AIPU, generates pipeline.

▶️

inference.py

Run inference on cameras, videos, or datasets. Benchmark FPS and measure accuracy.

🔧

axdevice

List and configure all connected Metis boards. Firmware updates, diagnostics.

📊

axrunmodel

Low-level model runner. DMA buffers, double buffering, multi-core dispatch.

🧮

TVM Compiler

Apache TVM-based ML compiler. Auto-quantization with proprietary algorithms.

🎬

GStreamer

Production pipeline framework. Hardware-accelerated video decode + AI inference.

🐍

Pipeline Builder

New Python API (v1.6). Composable operators: op.seq(), op.par(), op.foreach().

🏪

Model Zoo

90+ pre-tested models with YAML configs. Classification, detection, segmentation, pose, LLM.

STEP 7

Benchmarking & Validation

Benchmark Commands

# Live camera inference
./inference.py my-model usb:0

# FPS benchmark (production pipe)
./inference.py my-model \
  media/test.mp4 \
  --pipe=gst --no-display

# Accuracy evaluation
./inference.py my-model dataset \
  --pipe=torch-aipu --no-display

# Compare pure PyTorch vs AIPU
./inference.py my-model dataset \
  --pipe=torch --no-display

Pipeline Modes

Mode	Use Case
`--pipe=torch`	Pure CPU baseline (debug)
`--pipe=torch-aipu`	AIPU + PyTorch pre/post (accuracy)
`--pipe=gst`	Full GStreamer (production FPS)

Accuracy Targets

✓ Quantization loss < 2% mAP typical
✓ ResNet-50: only 0.18% loss
✓ YOLOv8s: only 0.93% loss
✓ Increase --num-cal-images if higher

Migrating with Amp AI Agent

Amp has a built-in cuda-to-axelera-migration skill

🔍

Step 1: Analyze

"Amp, analyze my CUDA inference code and suggest Axelera migration path"

📤

Step 2: Export

Amp writes torch.onnx.export() code, verifies operators, simplifies with onnxsim

📝

Step 3: Generate YAML

Amp generates complete Voyager YAML pipeline from your CUDA preprocessing code

🧪

Step 4: Test & Debug

Amp generates benchmark scripts, compares accuracy, troubleshoots compilation errors

# Just tell Amp what you need:
"Migrate my YOLOv8 CUDA inference pipeline
 to Axelera Voyager SDK. My model is at
 /models/yolov8n_custom.pt trained on 5 classes."

# Amp will:
# 1. Export to ONNX with correct opset
# 2. Check operator compatibility
# 3. Generate complete YAML pipeline
# 4. Write deploy & benchmark commands

Testing & QA Verification

Verification Layers

Layer	Test	Pass Criteria
ONNX Export	onnxruntime inference	Matches PyTorch output
Compilation	deploy.py exit 0	No errors
Quantization	torch-aipu vs torch	< 2% accuracy loss
Pipeline	gst end-to-end	Correct detections
Performance	FPS benchmark	Meets target FPS
Integration	App API test	Output format correct
Thermal	Sustained load	Within TDP

QA Checklist

✓ ONNX loads in onnxruntime (CPU baseline)
✓ deploy.py completes without errors
✓ Accuracy within 2% of FP32 baseline
✓ FPS meets target on Metis hardware
✓ Preprocess YAML matches training exactly
✓ Multi-stream scales linearly
✓ Thermal/power within edge budget
✓ API output matches app expectations

Troubleshooting Common Issues

Problem	Cause	Solution
Accuracy loss > 2%	Insufficient calibration	`--num-cal-images=400`, try `per_tensor_histogram`
Unsupported ONNX op	Op not on AIPU	Auto CPU fallback. Simplify with `onnxsim`
Model too large	Exceeds card memory	M.2=1GB, M.2 Max=16GB, PCIe=4-16GB
Low FPS	Wrong pipeline mode	Use `--pipe=gst`, `handle_all: false`, multi-core
Preprocessing mismatch	YAML ≠ training code	Verify mean/std, resize mode, color order exactly
Custom CUDA kernels	No direct equivalent	Reimplement as AxTransform C++ plugin
GStreamer pipeline crash	Plugin path or format	Check `GST_PLUGIN_PATH`, test with sample video first
No device found	Hardware/driver issue	Run `axdevice list`, check PCIe seating, update firmware

Hardware Options

💳

M.2 Card

214 TOPS • 1 GB DRAM
Smallest form factor. Fits any M.2 socket. Perfect for cameras, IoT gateways.

💳

M.2 Max Card

214 TOPS • up to 16 GB
LLM/VLM capable. Multi-camera, cascade models. Secure boot.

🖥️

PCIe Card (1×)

214 TOPS • Up to 3,200 FPS ResNet-50
For servers, workstations, eval kits.

🖥️

PCIe Card (4×)

856 TOPS • 12,800 FPS ResNet-50
Maximum performance. 4 Metis AIPUs on one card.

📟

Compute Board

Metis AIPU + ARM RK3588
Complete SBC. USB, HDMI, LAN, GPIO. Ready to deploy.

🚀

Europa AIPU

629 TOPS • Next-gen
Edge servers, robotics, autonomous systems. Early access.

Application Integration APIs

Choose the right abstraction level for your application

API Level	Language	Use Case	Complexity
InferenceStream	Python	Read pipeline results in your app	Low
Pipeline Builder	Python	Compose pipelines as Python expressions	Low
AxInferenceNet	C/C++	Integrate inference into C++ apps	Medium
AxRuntime	Python/C++	Manual pipeline construction	High
GStreamer Plugins	C/C++	Custom video pipeline elements	High

# Pipeline Builder API (v1.6) — Python native
from axelera.pipeline import op

pipeline = op.seq(
    op.load("yolov8n.axm"),
    op.tracker(algo="bytetrack"),
    op.nms(iou=0.45)
)
for result in pipeline.run(source="usb:0"):
    for obj in result.objects:
        print(obj.label, obj.confidence)

Migration Summary

1

Export

PyTorch → ONNX
One command

2

Deploy

YAML + deploy.py
Auto-quantize

3

Run

inference.py
Production ready

Bottom Line

If your workload is computer vision inference at the edge — Axelera delivers 10-50× better perf/watt, 10× lower cost, and dramatically simpler deployment than CUDA GPUs.