RK3588 NPU in Practice: What Can 6 TOPS Run? Deployment and Optimization Guide

The RK3588 NPU, rated at 6 TOPS (INT8), is the single most important capability separating it from ordinary application processors. But "what can 6 TOPS run" depends on the combination of model, precision and stream count. This article gives practical, field-tested criteria.

1. NPU specifications

Item	Specification
Compute	6 TOPS @ INT8 (3 NPU cores, independently schedulable)
Precision	INT4 / INT8 / INT16 / FP16
Operators	Convolution, depthwise conv, pooling, fully connected, common activations and post-processing
Toolchain	RKNN-Toolkit2 (PC conversion) + RKNPU2 SDK (on-device inference)

Note the "3 cores" design: the cores can run different models in parallel or share one model. Good scheduling substantially improves multi-task throughput.

2. What 6 TOPS can actually run

Practical reference values (1080p input, INT8 quantized, roughly 70% single-core utilization):

Model type	Typical frame rate	Notes
YOLOv5s / YOLOv8s	~30–60 FPS	The workhorse for multi-stream video analytics
YOLOv8m	~15–25 FPS	Higher accuracy, best for single-stream high-precision use
Lightweight classifiers (MobileNet class)	Hundreds of FPS	Can run many in parallel
Face detection + recognition	~20–40 FPS	Detection and recognition run in series
Pose estimation (lightweight)	~20–30 FPS	Behavior analysis projects
Small LLMs (0.5B–1.5B, INT4)	A few to ~a dozen tokens/s	On-device chat works, but not fast

In short: RK3588 comfortably handles multi-stream video analytics (e.g. 8–16 channels of 1080p detection); running LLMs is possible but should not be expected to be fast — suitable for offline or light-interaction scenarios.

3. RKNN deployment workflow

1. Model preparation: export PyTorch/ONNX models to ONNX (custom operators must be replaced or rewritten).
2. Conversion: load the ONNX model with RKNN-Toolkit2 on a PC and configure the quantization dataset (100–500 real-scene images recommended).
3. Quantization: prefer INT8; if accuracy drops too much, keep sensitive layers in FP16 (hybrid quantization).
4. On-device inference: load the .rknn model with RKNPU2 (C/C++ or Python), feeding frames from V4L2/RTSP.
5. Zero-copy: let camera data flow directly into NPU input memory (DMA-BUF) to avoid repeated copies — usually the single biggest speedup.

4. Performance optimization tips

• Don't over-size the input: 640×640 is usually the sweet spot for detection; feeding 1080p directly rarely pays off.
• Use all three cores: with multiple streams, assign channels to different NPU cores, or split one model across cores (supported by RKNN).
• Keep CPU post-processing light: NMS and similar steps on the CPU easily become the bottleneck; use NPU operators or optimized implementations.
• Calibration data must match reality: calibrating a night-surveillance model with daytime indoor images will always hurt accuracy.
• Watch shared memory bandwidth: display, video encoding and NPU all compete for DDR bandwidth — budget it centrally under heavy load.

5. Compute planning for typical scenarios

Scenario	Recommended configuration
Single 4K AI analytics	YOLOv8s @640 on one NPU core — plenty of headroom
8-channel 1080p detection	YOLOv8n/s × 8 spread across three cores; watch decoding and memory bandwidth
16+ channels	Multiple RK3588 devices or a dedicated AI accelerator; a single chip becomes the bottleneck
On-device chat / OCR assistant	~1B INT4 model plus retrieval; expect sub-second to second-level responses

6. How Hongyin Tech can help

We offer compute assessment for RK3588 AI solutions, model conversion and quantization tuning, multi-stream analytics architecture design, and complete hardware-software products. If you are unsure whether a model can run at your required stream count, send us the model and channel count — we will provide a compute assessment before committing to an architecture.