{"id":485550,"date":"2026-06-29T21:04:44","date_gmt":"2026-06-29T21:04:44","guid":{"rendered":"https:\/\/savepearlharbor.com\/?p=485550"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=485550","title":{"rendered":"5 \u0434\u0432\u0438\u0436\u043a\u043e\u0432, 1 resnet: \u0431\u0438\u0442\u0432\u0430 inference-\u0440\u0430\u043d\u0442\u0430\u0439\u043c\u043e\u0432 2026"},"content":{"rendered":"
\n
\n

TL;DR:<\/strong> \u041f\u0440\u043e\u0433\u043d\u0430\u043b ResNet-50 \u0447\u0435\u0440\u0435\u0437 PyTorch, ONNX Runtime, OpenVINO, TensorRT \u0438 TVM \u0432 FP32\/FP16\/INT8\/INT4 \u043d\u0430 CPU Ryzen 9 6900HS \u0438 GPU RTX 3070 Ti Laptop \u0432 46 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f\u0445. \u041b\u0443\u0447\u0448\u0438\u0439 CPU:<\/strong> ONNX Runtime static INT8 \u2014 15.4 ms \/ 64.8 img\/s (\u00d74.0 \u043e\u0442 torch baseline \u043f\u0440\u0438 bs=1). \u041b\u0443\u0447\u0448\u0438\u0439 GPU:<\/strong> TensorRT INT8 \u2014 1.16 ms \/ 863 img\/s (\u00d75.8). torch.compile + FP16 \u0434\u0430\u0451\u0442 \u00d72.6 \u0443\u0441\u043a\u043e\u0440\u0435\u043d\u0438\u044f \u0431\u0435\u0437 \u0441\u043c\u0435\u043d\u044b \u0434\u0432\u0438\u0436\u043a\u0430. \u041a\u043e\u0434 \u0438 \u0434\u0430\u043d\u043d\u044b\u0435: github.com\/DmitriyValetov\/resnet50-inference-benchmark<\/a>.<\/p>\n<\/blockquote>\n

\u0412\u0432\u0435\u0434\u0435\u043d\u0438\u0435<\/h3>\n

\u0415\u0441\u0442\u044c \u0442\u0430\u043a\u0430\u044f \u0440\u0443\u0431\u0440\u0438\u043a\u0430 \u2014 \u0431\u0435\u043d\u0447\u043c\u0430\u0440\u043a\u0438 \u0433\u043e\u043d\u044f\u0442\u044c. \u042d\u0442\u0430 \u0441\u0442\u0430\u0442\u044c\u044f \u043a\u0430\u043a \u0440\u0430\u0437 \u0438\u0437 \u0442\u0430\u043a\u0438\u0445. \u041f\u043e \u043c\u043e\u0442\u0438\u0432\u0430\u043c \u0440\u044f\u0434\u0430 \u043f\u0443\u0431\u043b\u0438\u043a\u0430\u0446\u0438\u0439 \u043f\u0440\u043e inference-\u0434\u0432\u0438\u0436\u043a\u0438 \u0437\u0430\u0445\u043e\u0442\u0435\u043b\u043e\u0441\u044c \u0441\u0434\u0435\u043b\u0430\u0442\u044c \u0441\u0432\u043e\u044e \u2014 \u0441 \u0431\u043e\u043b\u0435\u0435 \u043f\u043e\u0434\u0440\u043e\u0431\u043d\u044b\u043c\u0438 \u043c\u0435\u0442\u0440\u0438\u043a\u0430\u043c\u0438 \u0438 \u043e\u0442\u043a\u0440\u044b\u0442\u044b\u043c \u043a\u043e\u0434\u043e\u043c.<\/p>\n

Inference-\u0434\u0432\u0438\u0436\u043a\u0438 \u043d\u0430\u043c \u043d\u0443\u0436\u043d\u044b, \u043a\u043e\u0433\u0434\u0430 \u043c\u043e\u0434\u0435\u043b\u044c \u043e\u0431\u0443\u0447\u0435\u043d\u0430 \u0438 \u0435\u0451 \u043d\u0443\u0436\u043d\u043e \u0433\u043e\u043d\u044f\u0442\u044c \u0432 \u043f\u0440\u043e\u0434\u0435, \u043b\u0438\u0431\u043e \u043c\u0430\u0441\u0448\u0442\u0430\u0431\u043d\u043e \u0432\u0430\u043b\u0438\u0434\u0438\u0440\u043e\u0432\u0430\u0442\u044c. \u0412 \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0435\u0439 \u0442\u0430\u0431\u043b\u0438\u0446\u0435 \u043f\u0440\u0438\u0432\u0435\u0434\u0443 \u043c\u043d\u043e\u0433\u043e\u043e\u0431\u0440\u0430\u0437\u0438\u0435 \u0434\u0432\u0438\u0436\u043a\u043e\u0432 \u0438 \u0437\u043e\u043d\u0443 \u0438\u0445 \u043f\u0440\u0438\u043c\u0435\u043d\u0438\u043c\u043e\u0441\u0442\u0438:<\/p>\n

\n
\n\n\n\n\n\n\n\n\n\n\n\n\n
\n

\u0424\u0440\u0435\u0439\u043c\u0432\u043e\u0440\u043a<\/p>\n<\/th>\n

\n

\u0421\u0435\u0440\u0432\u0435\u0440\/\u041e\u0431\u043b\u0430\u043a\u043e<\/p>\n<\/th>\n

\n

Edge<\/p>\n<\/th>\n

\n

\u041f\u0440\u0438\u043c\u0435\u0447\u0430\u043d\u0438\u0435<\/p>\n<\/th>\n<\/tr>\n

\n

TensorRT<\/p>\n<\/td>\n

\n

\u2705<\/p>\n<\/td>\n

\n

\u2705<\/p>\n<\/td>\n

\n

\u0422\u043e\u043b\u044c\u043a\u043e NVIDIA GPU<\/p>\n<\/td>\n<\/tr>\n

\n

ONNX Runtime<\/p>\n<\/td>\n

\n

\u2705<\/p>\n<\/td>\n

\n

\u2705<\/p>\n<\/td>\n

\n

\u0423\u043d\u0438\u0432\u0435\u0440\u0441\u0430\u043b\u044c\u043d\u044b\u0439 \u0432\u0430\u0440\u0438\u0430\u043d\u0442<\/p>\n<\/td>\n<\/tr>\n

\n

OpenVINO<\/p>\n<\/td>\n

\n

\u2705<\/p>\n<\/td>\n

\n

\u2705<\/p>\n<\/td>\n

\n

\u0412 \u043e\u0441\u043d\u043e\u0432\u043d\u043e\u043c \u0437\u0430\u0442\u043e\u0447\u0435\u043d \u043f\u043e\u0434 intel \u044d\u043a\u043e\u0441\u0438\u0441\u0442\u0435\u043c\u0443, \u043d\u043e \u043d\u0430 rysen \u0442\u043e\u0436\u0435 \u043f\u043e\u043a\u0430\u0437\u0430\u043b \u043f\u0440\u0438\u0440\u043e\u0441\u0442 \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0441\u0442\u0438<\/p>\n<\/td>\n<\/tr>\n

\n

TVM<\/p>\n<\/td>\n

\n

\u2705<\/p>\n<\/td>\n

\n

\u2705<\/p>\n<\/td>\n

\n

\u041a\u043e\u043c\u043f\u0438\u043b\u044f\u0442\u043e\u0440, \u00b5TVM \u0434\u043b\u044f MCU edge-ai-vision<\/p>\n<\/td>\n<\/tr>\n

\n

ExecuTorch<\/p>\n<\/td>\n

\n

\u274c<\/p>\n<\/td>\n

\n

\u2705<\/p>\n<\/td>\n

\n

\u0417\u0430\u043c\u0435\u043d\u0430 PyTorch Mobile<\/p>\n<\/td>\n<\/tr>\n

\n

TFLite\/LiteRT<\/p>\n<\/td>\n

\n

\u274c<\/p>\n<\/td>\n

\n

\u2705<\/p>\n<\/td>\n

\n

\u0414\u0432\u0438\u0436\u043e\u043a \u043f\u043e\u0434 Android\/iOS\/RPi<\/p>\n<\/td>\n<\/tr>\n

\n

NCNN<\/p>\n<\/td>\n

\n

\u274c<\/p>\n<\/td>\n

\n

\u2705<\/p>\n<\/td>\n

\n

Tencent edge engine<\/p>\n<\/td>\n<\/tr>\n

\n

MNN<\/p>\n<\/td>\n

\n

\u274c<\/p>\n<\/td>\n

\n

\u2705<\/p>\n<\/td>\n

\n

Alibaba edge engine<\/p>\n<\/td>\n<\/tr>\n

\n

Paddle Lite<\/p>\n<\/td>\n

\n

\u274c<\/p>\n<\/td>\n

\n

\u2705<\/p>\n<\/td>\n

\n

Baidu edge engine<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

\u0422\u0430\u043a \u0436\u0435 \u0441\u0443\u0449\u0435\u0441\u0442\u0432\u0443\u044e\u0442 \u0440\u0430\u0437\u043b\u0438\u0447\u043d\u044b\u0435 \u043c\u0435\u0442\u043e\u0434\u044b \u043e\u043f\u0442\u0438\u043c\u0438\u0437\u0430\u0446\u0438\u0438 \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0441\u0442\u0438 (\u0432 \u043d\u0430\u0441\u0442\u043e\u044f\u0449\u0435\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u0440\u0435\u0447\u044c \u043f\u043e\u0439\u0434\u0435\u0442 \u0442\u043e\u043b\u044c\u043a\u043e \u043e \u0442\u0435\u0445, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u043d\u0435 \u043c\u0435\u043d\u044f\u044e\u0442 \u0430\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u0443 \u0441\u0435\u0442\u0438), \u043f\u043e\u0434\u0434\u0435\u0440\u0436\u0438\u0432\u0430\u0435\u043c\u044b\u0435 \u0431\u043e\u043b\u044c\u0448\u0435\u0439 \u0447\u0430\u0441\u0442\u044c\u044e \u0434\u0432\u0438\u0436\u043a\u043e\u0432, \u043f\u0443\u0441\u0442\u044c \u0438 \u0441 \u043c\u0435\u0441\u0442\u043d\u044b\u043c\u0438 \u043d\u044e\u0430\u043d\u0441\u0430\u043c\u0438, \u044d\u0442\u043e: \u0441\u043d\u0438\u0436\u0435\u043d\u0438\u0435 \u0442\u043e\u0447\u043d\u043e\u0441\u0442\u0438 \u0432\u044b\u0447\u0438\u0441\u043b\u0435\u043d\u0438\u0439 (FP32\u2192FP16\/INT8\/INT4), kernel fusion, \u0433\u0440\u0430\u0444\u043e\u0432\u044b\u0435 \u043e\u043f\u0442\u0438\u043c\u0438\u0437\u0430\u0446\u0438\u0438.<\/p>\n

\u0412 \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u044f\u043c \u043f\u043e\u0434\u0432\u0435\u0440\u0433\u043d\u0435\u0442\u0441\u044f ResNet-50 \u0432 46 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f\u0445 \u043d\u0430 5 \u0434\u0432\u0438\u0436\u043a\u0430\u0445, \u0441 \u0446\u0438\u0444\u0440\u0430\u043c\u0438, \u0442\u0430\u0431\u043b\u0438\u0446\u0430\u043c\u0438 \u0438 \u0433\u0440\u0430\u0444\u0438\u043a\u0430\u043c\u0438.<\/p>\n

\u0418\u0441\u043f\u044b\u0442\u0430\u0442\u0435\u043b\u044c\u043d\u044b\u0439 \u0441\u0442\u0435\u043d\u0434<\/h3>\n
\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n

\u041a\u043e\u043c\u043f\u043e\u043d\u0435\u043d\u0442<\/p>\n<\/th>\n

\n

\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f<\/p>\n<\/th>\n<\/tr>\n

\n

CPU<\/strong><\/p>\n<\/td>\n

\n

AMD Ryzen 9 6900HS (8C\/16T, Zen 3+)<\/p>\n<\/td>\n<\/tr>\n

\n

GPU<\/strong><\/p>\n<\/td>\n

\n

NVIDIA RTX 3070 Ti Laptop (8 GB GDDR6)<\/p>\n<\/td>\n<\/tr>\n

\n

RAM<\/strong><\/p>\n<\/td>\n

\n

32 GB DDR5<\/p>\n<\/td>\n<\/tr>\n

\n

\u041e\u043a\u0440\u0443\u0436\u0435\u043d\u0438\u0435<\/strong><\/p>\n<\/td>\n

\n

Docker, Ubuntu 22.04, CUDA 12.8<\/p>\n<\/td>\n<\/tr>\n

\n

\u041c\u043e\u0434\u0435\u043b\u044c<\/strong><\/p>\n<\/td>\n

\n

ResNet-50, \u0432\u0435\u0441\u0430 ImageNet1K_V1<\/p>\n<\/td>\n<\/tr>\n

\n

\u0414\u0430\u0442\u0430\u0441\u0435\u0442 \u0434\u043b\u044f \u044d\u0432\u0430\u043b\u0430<\/strong><\/p>\n<\/td>\n

\n

10k \u0438\u0437\u043e\u0431\u0440\u0430\u0436\u0435\u043d\u0438\u0439 (\u043f\u043e\u0434\u043c\u043d\u043e\u0436\u0435\u0441\u0442\u0432\u043e ImageNet val) — Top-1 \/ Top-5 Accuracy<\/p>\n<\/td>\n<\/tr>\n

\n

\u041a\u0430\u043a \u0441\u0447\u0438\u0442\u0430\u0435\u043c \u0441\u043a\u043e\u0440\u043e\u0441\u0442\u044c<\/strong><\/p>\n<\/td>\n

\n

Latency\/Throughput \u043f\u0440\u0438 bs=1 \u0438 bs=64<\/p>\n<\/td>\n<\/tr>\n

\n

Warmup<\/strong><\/p>\n<\/td>\n

\n

10 \u0438\u0442\u0435\u0440\u0430\u0446\u0438\u0439<\/p>\n<\/td>\n<\/tr>\n

\n

Measurement<\/strong><\/p>\n<\/td>\n

\n

50\u2013100 \u0438\u0442\u0435\u0440\u0430\u0446\u0438\u0439, \u043c\u0435\u0434\u0438\u0430\u043d\u0430<\/p>\n<\/td>\n<\/tr>\n

\n

\u0421\u0435\u0442\u043a\u0430 batch size<\/strong><\/p>\n<\/td>\n

\n

1, 8, 16, 32, 64<\/p>\n<\/td>\n<\/tr>\n

\n

Baseline<\/strong><\/p>\n<\/td>\n

\n

torch FP32 eager mode<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

\u041c\u0435\u0442\u043e\u0434\u044b<\/h3>\n

\u0412\u0441\u0435\u0433\u043e \u043f\u0440\u043e\u0433\u043d\u0430\u043d\u043e 46 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0439<\/strong> \u043f\u043e \u043f\u044f\u0442\u0438 \u0434\u0432\u0438\u0436\u043a\u0430\u043c:<\/p>\n

\n
\n\n\n\n\n\n\n\n\n\n
\n

\u0414\u0432\u0438\u0436\u043e\u043a<\/p>\n<\/th>\n

\n

\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0439<\/p>\n<\/th>\n

\n

\u0427\u0442\u043e \u0432\u043d\u0443\u0442\u0440\u0438<\/p>\n<\/th>\n<\/tr>\n

\n

PyTorch<\/strong><\/p>\n<\/td>\n

\n

19<\/p>\n<\/td>\n

\n

FP32\/FP16 CPU+CUDA, autocast, 6 \u00d7 torch.compile, 3 quant (dynamic\/static FX\/PT2E), PT2E+compile \u00d72, W4 GPU (eager + 2 compile)<\/p>\n<\/td>\n<\/tr>\n

\n

ONNX Runtime<\/strong><\/p>\n<\/td>\n

\n

8<\/p>\n<\/td>\n

\n

FP32\/FP16\/INT8 dynamic\/INT8 static \u00d7 (CPU + CUDA)<\/p>\n<\/td>\n<\/tr>\n

\n

OpenVINO<\/strong><\/p>\n<\/td>\n

\n

3<\/p>\n<\/td>\n

\n

FP32\/FP16\/INT8 \u2014 CPU<\/p>\n<\/td>\n<\/tr>\n

\n

TensorRT<\/strong><\/p>\n<\/td>\n

\n

3<\/p>\n<\/td>\n

\n

FP32\/FP16\/INT8 \u2014 CUDA<\/p>\n<\/td>\n<\/tr>\n

\n

TVM<\/strong><\/p>\n<\/td>\n

\n

13<\/p>\n<\/td>\n

\n

FP32\/FP16\/INT8 \u00d7 (CPU + CUDA), default\/MetaSchedule\/AutoTVM<\/p>\n<\/td>\n<\/tr>\n

\n

\u0412\u0441\u0435\u0433\u043e<\/strong><\/p>\n<\/td>\n

\n

46<\/strong><\/p>\n<\/td>\n

\n

\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

Torch: \u0431\u0435\u0439\u0437\u043b\u0430\u0439\u043d \u0438 \u0431\u043e\u0433\u0430\u0442\u044b\u0439 \u043d\u0430\u0431\u043e\u0440 \u043e\u043f\u0442\u0438\u043c\u0438\u0437\u0430\u0446\u0438\u0439<\/h3>\n

Torch \u2014 \u0442\u043e, \u0441 \u0447\u0435\u0433\u043e \u0432\u0441\u0435 \u043d\u0430\u0447\u0438\u043d\u0430\u044e\u0442: \u043e\u0431\u0443\u0447\u0435\u043d\u0438\u0435, \u044d\u043a\u0441\u043f\u0435\u0440\u0438\u043c\u0435\u043d\u0442\u044b. \u0427\u0430\u0449\u0435 \u0432\u0441\u0435\u0433\u043e \u043d\u0430 \u043d\u0435\u043c \u0438\u043d\u0444\u0435\u0440\u0435\u043d\u0441 \u0432 \u043f\u0440\u043e\u0434\u0435 \u043d\u0435 \u0433\u043e\u043d\u044f\u044e\u0442, \u043d\u043e, \u043a \u043c\u043e\u0435\u043c\u0443 \u0443\u0434\u0438\u0432\u043b\u0435\u043d\u0438\u044e, \u0432\u043d\u0443\u0442\u0440\u0438 \u044d\u043a\u043e\u0441\u0438\u0441\u0442\u0435\u043c\u044b PyTorch \u0435\u0441\u0442\u044c \u0431\u043e\u043b\u044c\u0448\u043e\u0439 \u0441\u043f\u0435\u043a\u0442\u0440 \u043d\u0430\u0441\u0442\u0440\u043e\u0439\u043a\u0438 \u0438\u043d\u0444\u0435\u0440\u0435\u043d\u0441\u0430.<\/p>\n

\n\u0421\u0435\u0442\u0430\u043f\u044b torch<\/summary>\n
\n
\n
\n\n\n\n\n\n\n\n\n\n\n\n\n
\n

Precision<\/p>\n<\/th>\n

\n

\u041c\u0435\u0442\u043e\u0434<\/p>\n<\/th>\n

\n

\u0423\u0441\u0442\u0440\u043e\u0439\u0441\u0442\u0432\u0430<\/p>\n<\/th>\n

\n

\u041f\u0440\u0438\u043c\u0435\u0447\u0430\u043d\u0438\u0435<\/p>\n<\/th>\n<\/tr>\n

\n

FP32<\/p>\n<\/td>\n

\n

eager<\/p>\n<\/td>\n

\n

CPU, CUDA<\/p>\n<\/td>\n

\n

baseline<\/p>\n<\/td>\n<\/tr>\n

\n

FP16<\/p>\n<\/td>\n

\n

model.half()<\/code><\/p>\n<\/td>\n

\n

CPU<\/p>\n<\/td>\n

\n

\u0432\u0445\u043e\u0434 float16<\/p>\n<\/td>\n<\/tr>\n

\n

FP16<\/p>\n<\/td>\n

\n

torch.amp.autocast<\/code><\/p>\n<\/td>\n

\n

CUDA<\/p>\n<\/td>\n

\n

\u0432\u0435\u0441\u0430 FP32, \u0433\u0434\u0435 FP16 \u2014 \u0432\u044b\u0431\u0438\u0440\u0430\u0435\u0442 PyTorch<\/p>\n<\/td>\n<\/tr>\n

\n

FP32<\/p>\n<\/td>\n

\n

torch.compile (reduce-overhead\/max-autotune)<\/p>\n<\/td>\n

\n

CPU<\/p>\n<\/td>\n

\n

Inductor<\/p>\n<\/td>\n<\/tr>\n

\n

FP16<\/p>\n<\/td>\n

\n

torch.compile (reduce-overhead\/max-autotune)<\/p>\n<\/td>\n

\n

CPU, CUDA<\/p>\n<\/td>\n

\n

Inductor<\/p>\n<\/td>\n<\/tr>\n

\n

INT8<\/p>\n<\/td>\n

\n

torchao weight-only<\/p>\n<\/td>\n

\n

CPU<\/p>\n<\/td>\n

\n

dynamic, \u0442\u043e\u043b\u044c\u043a\u043e \u0432\u0435\u0441\u0430<\/p>\n<\/td>\n<\/tr>\n

\n

INT8<\/p>\n<\/td>\n

\n

FX static PTQ (prepare \u2192 \u043a\u0430\u043b\u0438\u0431\u0440\u043e\u0432\u043a\u0430 \u2192 convert)<\/p>\n<\/td>\n

\n

CPU<\/p>\n<\/td>\n

\n

\u0432\u0435\u0441\u0430 + \u0430\u043a\u0442\u0438\u0432\u0430\u0446\u0438\u0438<\/p>\n<\/td>\n<\/tr>\n

\n

INT8<\/p>\n<\/td>\n

\n

PT2E (torch.export + x86-\u043a\u0432\u0430\u043d\u0442\u0430\u0439\u0437\u0435\u0440)<\/p>\n<\/td>\n

\n

CPU<\/p>\n<\/td>\n

\n

+ compile \u043e\u043f\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043e<\/p>\n<\/td>\n<\/tr>\n

\n

INT4 W4<\/p>\n<\/td>\n

\n

torchao, Linear \u2192 INT4, \u043c\u043e\u0434\u0435\u043b\u044c BF16<\/p>\n<\/td>\n

\n

CUDA<\/p>\n<\/td>\n

\n

weight-only, eager + compile<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<\/div>\n<\/details>\n

\n\u0422\u0430\u0431\u043b\u0438\u0446\u0430 \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442\u043e\u0432<\/summary>\n
\n
\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n

\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f<\/p>\n<\/th>\n

\n

Device<\/p>\n<\/th>\n

\n

Lat\/img bs=1 (ms)<\/p>\n<\/th>\n

\n

Thr bs=1 (img\/s)<\/p>\n<\/th>\n

\n

Lat\/img bs=64 (ms)<\/p>\n<\/th>\n

\n

Thr bs=64 (img\/s)<\/p>\n<\/th>\n

\n

Top-1 (%)<\/p>\n<\/th>\n<\/tr>\n

\n

FP32 (cpu baseline)<\/p>\n<\/td>\n

\n

CPU<\/p>\n<\/td>\n

\n

61.6<\/p>\n<\/td>\n

\n

16.2<\/p>\n<\/td>\n

\n

119.6<\/p>\n<\/td>\n

\n

8.4<\/p>\n<\/td>\n

\n

76.15<\/p>\n<\/td>\n<\/tr>\n

\n

FP32 + compile(reduce)<\/p>\n<\/td>\n

\n

CPU<\/p>\n<\/td>\n

\n

77.2<\/p>\n<\/td>\n

\n

13.0<\/p>\n<\/td>\n

\n

56.2<\/p>\n<\/td>\n

\n

17.8<\/p>\n<\/td>\n

\n

76.15<\/p>\n<\/td>\n<\/tr>\n

\n

FP16 (half)<\/p>\n<\/td>\n

\n

CPU<\/p>\n<\/td>\n

\n

2250.1<\/p>\n<\/td>\n

\n

0.4<\/p>\n<\/td>\n

\n

1230.0<\/p>\n<\/td>\n

\n

0.8<\/p>\n<\/td>\n

\n

76.15<\/p>\n<\/td>\n<\/tr>\n

\n

INT8 dynamic<\/p>\n<\/td>\n

\n

CPU<\/p>\n<\/td>\n

\n

150.7<\/p>\n<\/td>\n

\n

6.6<\/p>\n<\/td>\n

\n

120.5<\/p>\n<\/td>\n

\n

8.3<\/p>\n<\/td>\n

\n

76.09<\/p>\n<\/td>\n<\/tr>\n

\n

INT8 static FX<\/strong><\/p>\n<\/td>\n

\n

CPU<\/strong><\/p>\n<\/td>\n

\n

26.7<\/strong><\/p>\n<\/td>\n

\n

37.5<\/strong><\/p>\n<\/td>\n

\n

25.9<\/strong><\/p>\n<\/td>\n

\n

38.6<\/strong><\/p>\n<\/td>\n

\n

75.89<\/strong><\/p>\n<\/td>\n<\/tr>\n

\n

INT8 PT2E<\/p>\n<\/td>\n

\n

CPU<\/p>\n<\/td>\n

\n

158.1<\/p>\n<\/td>\n

\n

6.3<\/p>\n<\/td>\n

\n

260.4<\/p>\n<\/td>\n

\n

3.8<\/p>\n<\/td>\n

\n

76.04<\/p>\n<\/td>\n<\/tr>\n

\n

PT2E + compile(max)<\/p>\n<\/td>\n

\n

CPU<\/p>\n<\/td>\n

\n

66.0<\/p>\n<\/td>\n

\n

14.5<\/p>\n<\/td>\n

\n

59.6<\/p>\n<\/td>\n

\n

16.8<\/p>\n<\/td>\n

\n

76.04<\/p>\n<\/td>\n<\/tr>\n

\n

FP32 (gpu baseline)<\/p>\n<\/td>\n

\n

CUDA<\/p>\n<\/td>\n

\n

6.68<\/p>\n<\/td>\n

\n

149.8<\/p>\n<\/td>\n

\n

1.55<\/p>\n<\/td>\n

\n

646.4<\/p>\n<\/td>\n

\n

76.15<\/p>\n<\/td>\n<\/tr>\n

\n

FP16 autocast<\/p>\n<\/td>\n

\n

CUDA<\/p>\n<\/td>\n

\n

9.4<\/p>\n<\/td>\n

\n

105.9<\/p>\n<\/td>\n

\n

0.94<\/p>\n<\/td>\n

\n

1094.0<\/p>\n<\/td>\n

\n

76.15<\/p>\n<\/td>\n<\/tr>\n

\n

FP16 + compile(reduce)<\/strong><\/p>\n<\/td>\n

\n

CUDA<\/strong><\/p>\n<\/td>\n

\n

2.57<\/strong><\/p>\n<\/td>\n

\n

389.2<\/strong><\/p>\n<\/td>\n

\n

0.69<\/strong><\/p>\n<\/td>\n

\n

1454.1<\/strong><\/p>\n<\/td>\n

\n

76.15<\/strong><\/p>\n<\/td>\n<\/tr>\n

\n

FP16 + compile(max)<\/p>\n<\/td>\n

\n

CUDA<\/p>\n<\/td>\n

\n

3.69<\/p>\n<\/td>\n

\n

345.2<\/p>\n<\/td>\n

\n

0.71<\/p>\n<\/td>\n

\n

1412.8<\/p>\n<\/td>\n

\n

76.15<\/p>\n<\/td>\n<\/tr>\n

\n

INT4 W4 eager<\/p>\n<\/td>\n

\n

CUDA<\/p>\n<\/td>\n

\n

8.09<\/p>\n<\/td>\n

\n

123.6<\/p>\n<\/td>\n

\n

0.98<\/p>\n<\/td>\n

\n

1019.3<\/p>\n<\/td>\n

\n

76.08<\/p>\n<\/td>\n<\/tr>\n

\n

INT4 W4 + compile(reduce)<\/p>\n<\/td>\n

\n

CUDA<\/p>\n<\/td>\n

\n

3.30<\/p>\n<\/td>\n

\n

303.2<\/p>\n<\/td>\n

\n

0.78<\/p>\n<\/td>\n

\n

1290.9<\/p>\n<\/td>\n

\n

76.03<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<\/div>\n<\/details>\n

\n

Lat\/img<\/strong> = \u0437\u0430\u0434\u0435\u0440\u0436\u043a\u0430 \u043d\u0430 \u043e\u0434\u043d\u043e \u0438\u0437\u043e\u0431\u0440\u0430\u0436\u0435\u043d\u0438\u0435 (per-batch latency \u00f7 batch size). Thr<\/strong> = \u043f\u0440\u043e\u043f\u0443\u0441\u043a\u043d\u0430\u044f \u0441\u043f\u043e\u0441\u043e\u0431\u043d\u043e\u0441\u0442\u044c (img\/s). Top-1 (%)<\/strong> = \u0434\u043e\u043b\u044f \u0438\u0437\u043e\u0431\u0440\u0430\u0436\u0435\u043d\u0438\u0439, \u0434\u043b\u044f \u043a\u043e\u0442\u043e\u0440\u044b\u0445 \u043c\u043e\u0434\u0435\u043b\u044c \u043f\u0440\u0430\u0432\u0438\u043b\u044c\u043d\u043e \u043f\u0440\u0435\u0434\u0441\u043a\u0430\u0437\u0430\u043b\u0430 \u043a\u043b\u0430\u0441\u0441.<\/p>\n<\/blockquote>\n

\u041f\u043e\u0431\u0435\u0434\u0438\u0442\u0435\u043b\u0438 \u0432\u043d\u0443\u0442\u0440\u0438 Torch<\/strong><\/p>\n

\n
\n\n\n\n\n\n
\n

\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f<\/p>\n<\/th>\n

\n

Device<\/p>\n<\/th>\n

\n

bs=1 Lat\/img<\/p>\n<\/th>\n

\n

bs=1 Thr<\/p>\n<\/th>\n

\n

bs=64 Lat\/img<\/p>\n<\/th>\n

\n

bs=64 Thr<\/p>\n<\/th>\n

\n

Top-1<\/p>\n<\/th>\n<\/tr>\n

\n

INT8 static FX<\/strong><\/p>\n<\/td>\n

\n

CPU<\/strong><\/p>\n<\/td>\n

\n

26.7 ms (\u00d72.3)<\/strong><\/p>\n<\/td>\n

\n

37.5 img\/s (\u00d72.3)<\/strong><\/p>\n<\/td>\n

\n

25.9 ms (\u00d74.6)<\/strong><\/p>\n<\/td>\n

\n

38.6 img\/s (\u00d74.6)<\/strong><\/p>\n<\/td>\n

\n

75.89<\/strong><\/p>\n<\/td>\n<\/tr>\n

\n

FP16 + compile(reduce)<\/strong><\/p>\n<\/td>\n

\n

GPU<\/strong><\/p>\n<\/td>\n

\n

2.57 ms (\u00d72.6)<\/strong><\/p>\n<\/td>\n

\n

389 img\/s (\u00d72.6)<\/strong><\/p>\n<\/td>\n

\n

0.69 ms (\u00d72.2)<\/strong><\/p>\n<\/td>\n

\n

1454 img\/s (\u00d72.2)<\/strong><\/p>\n<\/td>\n

\n

76.15<\/strong><\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

\u00d7-\u044b \u043e\u0442\u043d\u043e\u0441\u0438\u0442\u0435\u043b\u044c\u043d\u043e torch FP32 eager \u043d\u0430 \u0441\u043e\u043e\u0442\u0432\u0435\u0442\u0441\u0442\u0432\u0443\u044e\u0449\u0435\u043c \u0443\u0441\u0442\u0440\u043e\u0439\u0441\u0442\u0432\u0435<\/em><\/p>\n

\"Latency<\/p>\n
Latency \u0438 Throughput vs batch size \u2014 \u0432\u0441\u0435 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 PyTorch \u043d\u0430 CPU.<\/figcaption><\/div>\n<\/figure>\n
\"Latency<\/p>\n
Latency \u0438 Throughput vs batch size \u2014 \u0432\u0441\u0435 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 PyTorch \u043d\u0430 CUDA.<\/figcaption><\/div>\n<\/figure>\n

ONNX Runtime<\/h3>\n

\u041f\u0440\u0435\u0434\u0432\u0430\u0440\u0438\u0442\u0435\u043b\u044c\u043d\u043e \u043c\u043e\u0434\u0435\u043b\u044c \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0438\u0440\u0443\u0435\u0442\u0441\u044f \u0438\u0437 torch \u0432 onnx (opset 17), \u0430 \u043f\u043e\u0442\u043e\u043c \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0441\u044f \u0438\u043d\u0444\u0435\u0440\u0435\u043d\u0441 \u0447\u0435\u0440\u0435\u0437 ONNX Runtime. FP32 \u0438 FP16 \u0433\u0440\u0430\u0444\u044b \u043d\u0430\u043f\u0440\u044f\u043c\u0443\u044e \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0438\u0440\u0443\u044e\u0442\u0441\u044f \u0438\u0437 torch, \u0430 \u043a\u0432\u0430\u043d\u0442\u043e\u0432\u0430\u043d\u043d\u044b\u0435 INT8 \u0441\u0442\u0440\u043e\u044f\u0442\u0441\u044f \u0443\u0436\u0435 \u0438\u0437 FP32-ONNX \u0441\u0440\u0435\u0434\u0441\u0442\u0432\u0430\u043c\u0438 onnxruntime.quantization<\/code>.<\/p>\n

\n\u0421\u0435\u0442\u0430\u043f\u044b onnxruntime<\/summary>\n
\n
\n
\n\n\n\n\n\n\n\n
\n

Precision<\/p>\n<\/th>\n

\n

Quant<\/p>\n<\/th>\n

\n

\u0423\u0441\u0442\u0440\u043e\u0439\u0441\u0442\u0432\u0430<\/p>\n<\/th>\n<\/tr>\n

\n

FP32<\/p>\n<\/td>\n

\n

\u2014<\/p>\n<\/td>\n

\n

CPU, CUDA<\/p>\n<\/td>\n<\/tr>\n

\n

FP16<\/p>\n<\/td>\n

\n

\u2014<\/p>\n<\/td>\n

\n

CPU, CUDA<\/p>\n<\/td>\n<\/tr>\n

\n

INT8<\/p>\n<\/td>\n

\n

dynamic (weight-only)<\/p>\n<\/td>\n

\n

CPU, CUDA<\/p>\n<\/td>\n<\/tr>\n

\n

INT8<\/p>\n<\/td>\n

\n

static (QDQ)<\/p>\n<\/td>\n

\n

CPU, CUDA<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<\/div>\n<\/details>\n

\n\u0422\u0430\u0431\u043b\u0438\u0446\u0430 \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442\u043e\u0432<\/summary>\n
\n
\n
\n\n\n\n\n\n\n\n\n\n\n\n
\n

\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f<\/p>\n<\/th>\n

\n

Device<\/p>\n<\/th>\n

\n

Lat\/img bs=1 (ms)<\/p>\n<\/th>\n

\n

Thr bs=1 (img\/s)<\/p>\n<\/th>\n

\n

Lat\/img bs=64 (ms)<\/p>\n<\/th>\n

\n

Thr bs=64 (img\/s)<\/p>\n<\/th>\n

\n

Top-1 (%)<\/p>\n<\/th>\n<\/tr>\n

\n

FP32<\/p>\n<\/td>\n

\n

CPU<\/p>\n<\/td>\n

\n

27.9<\/p>\n<\/td>\n

\n

35.9<\/p>\n<\/td>\n

\n

25.9<\/p>\n<\/td>\n

\n

38.6<\/p>\n<\/td>\n

\n

76.15<\/p>\n<\/td>\n<\/tr>\n

\n

FP16<\/p>\n<\/td>\n

\n

CPU<\/p>\n<\/td>\n

\n

64.8<\/p>\n<\/td>\n

\n

15.4<\/p>\n<\/td>\n

\n

34.3<\/p>\n<\/td>\n

\n

29.1<\/p>\n<\/td>\n

\n

76.14<\/p>\n<\/td>\n<\/tr>\n

\n

Dynamic INT8<\/p>\n<\/td>\n

\n

CPU<\/p>\n<\/td>\n

\n

26.5<\/p>\n<\/td>\n

\n

37.8<\/p>\n<\/td>\n

\n

38.5<\/p>\n<\/td>\n

\n

25.9<\/p>\n<\/td>\n

\n

75.54<\/p>\n<\/td>\n<\/tr>\n

\n

Static INT8 (QDQ)<\/strong><\/p>\n<\/td>\n

\n

CPU<\/strong><\/p>\n<\/td>\n

\n

15.4<\/strong><\/p>\n<\/td>\n

\n

64.8<\/strong><\/p>\n<\/td>\n

\n

12.0<\/strong><\/p>\n<\/td>\n

\n

83.5<\/strong><\/p>\n<\/td>\n

\n

73.78<\/strong><\/p>\n<\/td>\n<\/tr>\n

\n

FP32<\/p>\n<\/td>\n

\n

CUDA<\/p>\n<\/td>\n

\n

5.27<\/p>\n<\/td>\n

\n

189.8<\/p>\n<\/td>\n

\n

1.69<\/p>\n<\/td>\n

\n

591.9<\/p>\n<\/td>\n

\n

76.15<\/p>\n<\/td>\n<\/tr>\n

\n

FP16<\/strong><\/p>\n<\/td>\n

\n

CUDA<\/strong><\/p>\n<\/td>\n

\n

5.65<\/strong><\/p>\n<\/td>\n

\n

177.1<\/strong><\/p>\n<\/td>\n

\n

1.07<\/strong><\/p>\n<\/td>\n

\n

934.1<\/strong><\/p>\n<\/td>\n

\n

76.14<\/strong><\/p>\n<\/td>\n<\/tr>\n

\n

Static INT8<\/p>\n<\/td>\n

\n

CUDA<\/p>\n<\/td>\n

\n

9.25<\/p>\n<\/td>\n

\n

108.2<\/p>\n<\/td>\n

\n

2.27<\/p>\n<\/td>\n

\n

440.9<\/p>\n<\/td>\n

\n

75.73<\/p>\n<\/td>\n<\/tr>\n

\n

Dynamic INT8<\/p>\n<\/td>\n

\n

CUDA<\/p>\n<\/td>\n

\n

46.7<\/p>\n<\/td>\n

\n

21.4<\/p>\n<\/td>\n

\n

38.0<\/p>\n<\/td>\n

\n

26.3<\/p>\n<\/td>\n

\n

75.54<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<\/div>\n<\/details>\n

\u041f\u043e\u0431\u0435\u0434\u0438\u0442\u0435\u043b\u0438 \u0432\u043d\u0443\u0442\u0440\u0438 ONNX Runtime (\u0432\u0441\u0435 \u043c\u043d\u043e\u0436\u0438\u0442\u0435\u043b\u0438 \u2014 \u043a FP32 eager):<\/strong><\/p>\n

\n
\n\n\n\n\n\n
\n

\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f<\/p>\n<\/th>\n

\n

Device<\/p>\n<\/th>\n

\n

bs=1 Lat\/img<\/p>\n<\/th>\n

\n

bs=1 Thr<\/p>\n<\/th>\n

\n

bs=64 Lat\/img<\/p>\n<\/th>\n

\n

bs=64 Thr<\/p>\n<\/th>\n

\n

Top-1<\/p>\n<\/th>\n<\/tr>\n

\n

Static INT8 (QDQ)<\/strong><\/p>\n<\/td>\n

\n

CPU<\/strong><\/p>\n<\/td>\n

\n

15.4 ms (\u00d74.0)<\/strong><\/p>\n<\/td>\n

\n

64.8 img\/s (\u00d74.0)<\/strong><\/p>\n<\/td>\n

\n

12.0 ms (\u00d710.0)<\/strong><\/p>\n<\/td>\n

\n

83.5 img\/s (\u00d710.0)<\/strong><\/p>\n<\/td>\n

\n

73.78<\/strong><\/p>\n<\/td>\n<\/tr>\n

\n

FP16<\/strong><\/p>\n<\/td>\n

\n

GPU<\/strong><\/p>\n<\/td>\n

\n

5.65 ms (\u00d71.2)<\/strong><\/p>\n<\/td>\n

\n

177 img\/s (\u00d71.2)<\/strong><\/p>\n<\/td>\n

\n

1.07 ms (\u00d71.4)<\/strong><\/p>\n<\/td>\n

\n

934 img\/s (\u00d71.4)<\/strong><\/p>\n<\/td>\n

\n

76.14<\/strong><\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

\"Latency<\/p>\n
Latency \u0438 Throughput vs batch size \u2014 \u0432\u0441\u0435 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 ONNX Runtime \u043d\u0430 CPU.<\/figcaption><\/div>\n<\/figure>\n
\"Latency<\/p>\n
Latency \u0438 Throughput vs batch size \u2014 \u0432\u0441\u0435 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 ONNX Runtime \u043d\u0430 CUDA.<\/figcaption><\/div>\n<\/figure>\n

OpenVINO<\/h3>\n

\u0414\u0430, \u044d\u0442\u043e\u0442 \u0444\u0440\u0435\u0439\u043c\u0432\u043e\u0440\u043a \u0437\u0430\u0442\u043e\u0447\u0435\u043d \u043f\u043e\u0434 \u0440\u0430\u0431\u043e\u0442\u0443 \u0441 \u0436\u0435\u043b\u0435\u0437\u043e\u043c intel, \u043d\u043e \u044f \u0442\u0430\u043a\u0438 \u043f\u043e\u043f\u0440\u043e\u0431\u0443\u044e \u0435\u0433\u043e \u043d\u0430 rysen \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u043e\u0440\u0435. \u041c\u043e\u0434\u0435\u043b\u044c \u0442\u0430\u043a\u0436\u0435 \u0438\u0437\u043d\u0430\u0447\u0430\u043b\u044c\u043d\u043e \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0438\u0440\u0443\u0435\u0442\u0441\u044f \u0438\u0437 torch \u0432 onnx (opset 17) \u0438 \u043a\u043e\u043d\u0432\u0435\u0440\u0442\u0438\u0440\u0443\u0435\u0442\u0441\u044f \u0432 OpenVINO IR (.xml\/.bin) \u0447\u0435\u0440\u0435\u0437 openvino.convert_model<\/code>. FP16 \u2014 \u0441\u0436\u0430\u0442\u0438\u0435\u043c \u0432\u0435\u0441\u043e\u0432 \u043f\u0440\u0438 \u0441\u043e\u0445\u0440\u0430\u043d\u0435\u043d\u0438\u0438 IR (compress_to_fp16=True<\/code> \u0432 openvino.convert_model<\/code>), INT8 \u2014 \u0441\u0442\u0430\u0442\u0438\u0447\u0435\u0441\u043a\u0438\u0439 PTQ \u0441 \u043a\u0430\u043b\u0438\u0431\u0440\u043e\u0432\u043a\u043e\u0439 (FakeQuantize \u0432 \u0433\u0440\u0430\u0444\u0435, INT8 \u0432\u0435\u0441\u0430 \u0438 \u0430\u043a\u0442\u0438\u0432\u0430\u0446\u0438\u0438, \u043a\u0430\u043b\u0438\u0431\u0440\u043e\u0432\u043a\u0430 \u043d\u0430 \u0442\u0435\u0445 \u0436\u0435 \u043a\u0430\u0440\u0442\u0438\u043d\u043a\u0430\u0445, \u0447\u0442\u043e \u0434\u043b\u044f torch\/ORT).<\/p>\n

\n\u0421\u0435\u0442\u0430\u043f\u044b openvino<\/summary>\n
\n
\n
\n\n\n\n\n\n\n
\n

Precision<\/p>\n<\/th>\n

\n

Quant<\/p>\n<\/th>\n

\n

\u0423\u0441\u0442\u0440\u043e\u0439\u0441\u0442\u0432\u0430<\/p>\n<\/th>\n<\/tr>\n

\n

FP32<\/p>\n<\/td>\n

\n

\u2014<\/p>\n<\/td>\n

\n

CPU<\/p>\n<\/td>\n<\/tr>\n

\n

FP16<\/p>\n<\/td>\n

\n

\u2014<\/p>\n<\/td>\n

\n

CPU<\/p>\n<\/td>\n<\/tr>\n

\n

INT8<\/p>\n<\/td>\n

\n

static PTQ<\/p>\n<\/td>\n

\n

CPU<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<\/div>\n<\/details>\n

\n\u0422\u0430\u0431\u043b\u0438\u0446\u0430 \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442\u043e\u0432<\/summary>\n
\n
\n
\n\n\n\n\n\n\n
\n

\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f<\/p>\n<\/th>\n

\n

Lat\/img bs=1 (ms)<\/p>\n<\/th>\n

\n

Thr bs=1 (img\/s)<\/p>\n<\/th>\n

\n

Lat\/img bs=64 (ms)<\/p>\n<\/th>\n

\n

Thr bs=64 (img\/s)<\/p>\n<\/th>\n

\n

Top-1 (%)<\/p>\n<\/th>\n<\/tr>\n

\n

FP32<\/p>\n<\/td>\n

\n

39.9<\/p>\n<\/td>\n

\n

25.1<\/p>\n<\/td>\n

\n

38.9<\/p>\n<\/td>\n

\n

25.7<\/p>\n<\/td>\n

\n

76.15<\/p>\n<\/td>\n<\/tr>\n

\n

FP16<\/p>\n<\/td>\n

\n

38.4<\/p>\n<\/td>\n

\n

26.1<\/p>\n<\/td>\n

\n

36.9<\/p>\n<\/td>\n

\n

27.1<\/p>\n<\/td>\n

\n

76.15<\/p>\n<\/td>\n<\/tr>\n

\n

INT8<\/strong><\/p>\n<\/td>\n

\n

18.1<\/strong><\/p>\n<\/td>\n

\n

55.3<\/strong><\/p>\n<\/td>\n

\n

18.4<\/strong><\/p>\n<\/td>\n

\n

54.4<\/strong><\/p>\n<\/td>\n

\n

73.46<\/strong><\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<\/div>\n<\/details>\n

\u041f\u043e\u0431\u0435\u0434\u0438\u0442\u0435\u043b\u044c \u0432\u043d\u0443\u0442\u0440\u0438 OpenVINO<\/strong><\/p>\n

\n
\n\n\n\n\n
\n

\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f<\/p>\n<\/th>\n

\n

Device<\/p>\n<\/th>\n

\n

bs=1 Lat\/img<\/p>\n<\/th>\n

\n

bs=1 Thr<\/p>\n<\/th>\n

\n

bs=64 Lat\/img<\/p>\n<\/th>\n

\n

bs=64 Thr<\/p>\n<\/th>\n

\n

Top-1<\/p>\n<\/th>\n<\/tr>\n

\n

INT8<\/strong><\/p>\n<\/td>\n

\n

CPU<\/strong><\/p>\n<\/td>\n

\n

18.1 ms (\u00d73.4)<\/strong><\/p>\n<\/td>\n

\n

55.3 img\/s (\u00d73.4)<\/strong><\/p>\n<\/td>\n

\n

18.4 ms (\u00d76.5)<\/strong><\/p>\n<\/td>\n

\n

54.4 img\/s (\u00d76.5)<\/strong><\/p>\n<\/td>\n

\n

73.46<\/strong><\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

\u0412\u044b\u0432\u043e\u0434\u044b:<\/strong> \u041f\u0440\u0438 bs=1 INT8 \u0432\u0442\u0440\u043e\u0435 \u0431\u044b\u0441\u0442\u0440\u0435\u0435 baseline (18.1 ms, \u00d73.4). \u041f\u0440\u0438 bs=64 \u2014 \u00d76.5 \u0431\u044b\u0441\u0442\u0440\u0435\u0435 baseline, \u043d\u043e per-image latency \u043f\u043e\u0447\u0442\u0438 \u043d\u0435 \u043f\u0430\u0434\u0430\u0435\u0442 \u0441 \u0440\u043e\u0441\u0442\u043e\u043c batch (\u0441 18.1 \u0434\u043e 18.4 ms). \u0414\u043b\u044f \u0441\u0440\u0430\u0432\u043d\u0435\u043d\u0438\u044f, ORT static INT8 \u0441\u043d\u0438\u0436\u0430\u0435\u0442 per-image latency \u0441 15.4 \u0434\u043e 12.0 ms. \u041f\u0443\u0441\u0442\u044c OV \u0438 \u0441\u043b\u0430\u0431\u043e \u043c\u0430\u0441\u0448\u0442\u0430\u0431\u0438\u0440\u0443\u0435\u0442\u0441\u044f \u043d\u0430 \u0431\u0430\u0442\u0447\u0438 \u043e\u0442\u043d\u043e\u0441\u0438\u0442\u0435\u043b\u044c\u043d\u043e ORT,\u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442 — \u0434\u043e\u0441\u0442\u043e\u0439\u043d\u044b\u0439.<\/p>\n

\"Latency<\/p>\n
Latency \u0438 Throughput vs batch size \u2014 \u0432\u0441\u0435 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 OpenVINO \u043d\u0430 CPU.<\/figcaption><\/div>\n<\/figure>\n

TVM<\/h3>\n

\u042f \u0431\u044b\u043b \u043d\u0430\u0441\u043b\u044b\u0448\u0430\u043d, \u0447\u0442\u043e tvm \u0442\u043e\u043d\u043a\u043e \u043d\u0430\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u0435\u0442\u0441\u044f \u043f\u043e\u0434 \u0436\u0435\u043b\u0435\u0437\u043e, \u043d\u043e \u043b\u0438\u0447\u043d\u043e \u043c\u043d\u0435 \u0441\u043e\u0432\u0441\u0435\u043c \u043d\u0435 \u0443\u0434\u0430\u043b\u043e\u0441\u044c \u0432\u044b\u0436\u0430\u0442\u044c \u0438\u0437 \u043d\u0435\u0433\u043e \u044d\u0444\u0444\u0435\u043a\u0442\u0438\u0432\u043d\u043e\u0441\u0442\u044c. \u0422\u0443\u0442 \u0430\u043d\u0430\u043b\u043e\u0433\u0438\u0447\u043d\u043e: torch \u0432 onnx, \u0437\u0430\u0442\u0435\u043c \u0433\u0440\u0443\u0437\u0438\u043c \u0432 Relay \u0438 \u043a\u043e\u043c\u043f\u0438\u043b\u0438\u0440\u0443\u0435\u043c \u043f\u043e\u0434 llvm (CPU) \u0438\u043b\u0438 cuda (GPU). \u0413\u0440\u0430\u0444 \u0444\u0438\u043a\u0441\u0438\u0440\u043e\u0432\u0430\u043d \u043f\u043e batch \u2014 \u0434\u043b\u044f \u043a\u0430\u0436\u0434\u043e\u0433\u043e bs \u043e\u0442\u0434\u0435\u043b\u044c\u043d\u0430\u044f \u043a\u043e\u043c\u043f\u0438\u043b\u044f\u0446\u0438\u044f \u0438 \u0442\u044e\u043d\u0438\u043d\u0433.<\/p>\n

\n\u0421\u0435\u0442\u0430\u043f\u044b TVM<\/summary>\n
\n
\n
\n\n\n\n\n\n\n
\n

Precision<\/p>\n<\/th>\n

\n

Schedule<\/p>\n<\/th>\n

\n

\u0423\u0441\u0442\u0440\u043e\u0439\u0441\u0442\u0432\u0430<\/p>\n<\/th>\n<\/tr>\n

\n

FP32<\/p>\n<\/td>\n

\n

default, MetaSchedule, AutoTVM<\/p>\n<\/td>\n

\n

CPU, CUDA<\/p>\n<\/td>\n<\/tr>\n

\n

FP16<\/p>\n<\/td>\n

\n

default, AutoTVM<\/p>\n<\/td>\n

\n

CPU, CUDA<\/p>\n<\/td>\n<\/tr>\n

\n

INT8<\/p>\n<\/td>\n

\n

default<\/p>\n<\/td>\n

\n

CPU, CUDA<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<\/div>\n<\/details>\n

\n\u0422\u0430\u0431\u043b\u0438\u0446\u0430 \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442\u043e\u0432<\/summary>\n
\n
\n
\n\n\n\n\n\n\n\n\n\n\n\n
\n

\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f<\/p>\n<\/th>\n

\n

Device<\/p>\n<\/th>\n

\n

Lat\/img bs=1 (ms)<\/p>\n<\/th>\n

\n

Thr bs=1 (img\/s)<\/p>\n<\/th>\n

\n

Lat\/img bs=64 (ms)<\/p>\n<\/th>\n

\n

Thr bs=64 (img\/s)<\/p>\n<\/th>\n

\n

Top-1 (%)<\/p>\n<\/th>\n<\/tr>\n

\n

FP32 default<\/p>\n<\/td>\n

\n

CPU<\/p>\n<\/td>\n

\n

93.2<\/p>\n<\/td>\n

\n

10.7<\/p>\n<\/td>\n

\n

3.5<\/p>\n<\/td>\n

\n

9.7<\/p>\n<\/td>\n

\n

66.7<\/p>\n<\/td>\n<\/tr>\n

\n

FP32 MetaSchedule<\/p>\n<\/td>\n

\n

CPU<\/p>\n<\/td>\n

\n

177.4<\/p>\n<\/td>\n

\n

5.6<\/p>\n<\/td>\n

\n

3.5<\/p>\n<\/td>\n

\n

4.4<\/p>\n<\/td>\n

\n

66.7<\/p>\n<\/td>\n<\/tr>\n

\n

FP32 AutoTVM<\/p>\n<\/td>\n

\n

CPU<\/p>\n<\/td>\n

\n

196.1<\/p>\n<\/td>\n

\n

5.1<\/p>\n<\/td>\n

\n

3.2<\/p>\n<\/td>\n

\n

4.9<\/p>\n<\/td>\n

\n

66.7<\/p>\n<\/td>\n<\/tr>\n

\n

FP16 default<\/p>\n<\/td>\n

\n

CPU<\/p>\n<\/td>\n

\n

25146.2<\/p>\n<\/td>\n

\n

<0.1<\/p>\n<\/td>\n

\n

403.9<\/p>\n<\/td>\n

\n

<0.1<\/p>\n<\/td>\n

\n

\u2014<\/p>\n<\/td>\n<\/tr>\n

\n

INT8 default<\/p>\n<\/td>\n

\n

CPU<\/p>\n<\/td>\n

\n

638.8<\/p>\n<\/td>\n

\n

1.6<\/p>\n<\/td>\n

\n

9.7<\/p>\n<\/td>\n

\n

1.6<\/p>\n<\/td>\n

\n

\u2014<\/p>\n<\/td>\n<\/tr>\n

\n

FP32 default<\/strong><\/p>\n<\/td>\n

\n

CUDA<\/strong><\/p>\n<\/td>\n

\n

6.86<\/strong><\/p>\n<\/td>\n

\n

145.8<\/strong><\/p>\n<\/td>\n

\n

0.05<\/strong><\/p>\n<\/td>\n

\n

285.2<\/strong><\/p>\n<\/td>\n

\n

66.7<\/strong><\/p>\n<\/td>\n<\/tr>\n

\n

FP16 default<\/p>\n<\/td>\n

\n

CUDA<\/p>\n<\/td>\n

\n

15.8<\/p>\n<\/td>\n

\n

63.4<\/p>\n<\/td>\n

\n

0.04<\/p>\n<\/td>\n

\n

376.4<\/p>\n<\/td>\n

\n

66.7<\/p>\n<\/td>\n<\/tr>\n

\n

INT8 default<\/p>\n<\/td>\n

\n

CUDA<\/p>\n<\/td>\n

\n

206.1<\/p>\n<\/td>\n

\n

4.8<\/p>\n<\/td>\n

\n

4.7<\/p>\n<\/td>\n

\n

3.3<\/p>\n<\/td>\n

\n

\u2014<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<\/div>\n<\/details>\n

Accuracy \u043d\u0430 \u043f\u043e\u0434\u043c\u043d\u043e\u0436\u0435\u0441\u0442\u0432\u0435 \u0438\u0437 100 \u0441\u0435\u043c\u043f\u043b\u043e\u0432 (~76% \u043d\u0430 \u043f\u043e\u043b\u043d\u043e\u043c ImageNet). FP16 CPU \u0438 INT8 \u2014 eval \u043d\u0435 \u0437\u0430\u043f\u0443\u0441\u043a\u0430\u043b\u0441\u044f \u0438\u0437-\u0437\u0430 \u0443\u0436\u0430\u0441\u043d\u043e\u0439 \u043e\u0436\u0438\u0434\u0430\u0435\u043c\u043e\u0439 \u0434\u043b\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0441\u0442\u0438.<\/em><\/p>\n

\u041f\u043e\u0431\u0435\u0434\u0438\u0442\u0435\u043b\u044c \u0432\u043d\u0443\u0442\u0440\u0438 TVM (\u00d7 \u043e\u0442 torch FP32 eager):<\/strong><\/p>\n

\n
\n\n\n\n\n\n
\n

\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f<\/p>\n<\/th>\n

\n

Device<\/p>\n<\/th>\n

\n

bs=1 Lat\/img<\/p>\n<\/th>\n

\n

bs=1 Thr<\/p>\n<\/th>\n

\n

bs=64 Lat\/img<\/p>\n<\/th>\n

\n

bs=64 Thr<\/p>\n<\/th>\n

\n

Top-1<\/p>\n<\/th>\n<\/tr>\n

\n

FP32 default<\/strong><\/p>\n<\/td>\n

\n

CPU<\/strong><\/p>\n<\/td>\n

\n

93.2 ms (\u00d70.7)<\/strong><\/p>\n<\/td>\n

\n

10.7 img\/s (\u00d70.7)<\/strong><\/p>\n<\/td>\n

\n

3.5 ms (\u00d71.2)<\/strong><\/p>\n<\/td>\n

\n

9.7 img\/s (\u00d71.2)<\/strong><\/p>\n<\/td>\n

\n

66.7<\/strong><\/p>\n<\/td>\n<\/tr>\n

\n

FP32 default<\/strong><\/p>\n<\/td>\n

\n

CUDA<\/strong><\/p>\n<\/td>\n

\n

6.86 ms (\u00d71.0)<\/strong><\/p>\n<\/td>\n

\n

145.8 img\/s (\u00d71.0)<\/strong><\/p>\n<\/td>\n

\n

0.05 ms (\u00d70.4)<\/strong><\/p>\n<\/td>\n

\n

285.2 img\/s (\u00d70.4)<\/strong><\/p>\n<\/td>\n

\n

66.7<\/strong><\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

\u0412\u044b\u0432\u043e\u0434\u044b:<\/strong> \u041b\u0443\u0447\u0448\u0438\u0435 \u0441\u0435\u0442\u0430\u043f\u044b \u043f\u043e \u044d\u0444\u0444\u0435\u043a\u0442\u0438\u0432\u043d\u043e\u0441\u0442\u0438 \u043f\u043e\u0440\u044f\u0434\u043a\u0430 eagere mode torch.<\/p>\n

\"Latency<\/p>\n
Latency \u0438 Throughput vs batch size \u2014 \u0432\u0441\u0435 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 TVM \u043d\u0430 CPU.<\/figcaption><\/div>\n<\/figure>\n
\"Latency<\/p>\n
Latency \u0438 Throughput vs batch size \u2014 \u0432\u0441\u0435 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 TVM \u043d\u0430 CUDA.<\/figcaption><\/div>\n<\/figure>\n

TensorRT<\/h3>\n

\u0412\u0441\u0435 \u0442\u0430\u043a\u0436\u0435 \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0438\u0440\u0443\u0435\u043c \u043c\u043e\u0434\u0435\u043b\u044c \u0432 onnx, onnx \u043f\u0430\u0440\u0441\u0438\u043c \u0432 TensorRT, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u043f\u0435\u0440\u0435\u0441\u043e\u0431\u0438\u0440\u0430\u0435\u0442 \u0433\u0440\u0430\u0444 \u0437\u0430\u043d\u043e\u0432\u043e \u043f\u043e\u0434 \u043a\u043e\u043d\u043a\u0440\u0435\u0442\u043d\u0443\u044e GPU \u2014 \u0441 kernel fusion, \u0430\u0432\u0442\u043e\u0432\u044b\u0431\u043e\u0440\u043e\u043c \u043e\u043f\u0442\u0438\u043c\u0430\u043b\u044c\u043d\u044b\u0445 \u044f\u0434\u0435\u0440 (tactic sources extended, opt level 5) \u0438 \u0444\u0438\u043a\u0441\u0438\u0440\u043e\u0432\u0430\u043d\u043d\u044b\u043c optimization profile (min=1, max=64). \u041d\u0430 \u0432\u044b\u0445\u043e\u0434\u0435 \u2014 \u0441\u0430\u043c\u043e\u0434\u043e\u0441\u0442\u0430\u0442\u043e\u0447\u043d\u044b\u0439 engine (.engine), \u043d\u0435 \u0442\u0440\u0435\u0431\u0443\u044e\u0449\u0438\u0439 \u043d\u0438 PyTorch, \u043d\u0438 ONNX Runtime. INT8 \u2014 \u0441 entropy-\u043a\u0430\u043b\u0438\u0431\u0440\u043e\u0432\u043a\u043e\u0439 \u043d\u0430 \u0442\u0435\u0445 \u0436\u0435 \u043a\u0430\u0440\u0442\u0438\u043d\u043a\u0430\u0445 \u0438 FP16-fallback \u0434\u043b\u044f \u0441\u043b\u043e\u0451\u0432, \u043d\u0435 \u0432\u043b\u0435\u0437\u0430\u044e\u0449\u0438\u0445 \u0432 INT8.<\/p>\n

\n\u0421\u0435\u0442\u0430\u043f\u044b TensorRT<\/summary>\n
\n
\n
\n\n\n\n\n\n\n
\n

Precision<\/p>\n<\/th>\n

\n

Features<\/p>\n<\/th>\n

\n

\u0423\u0441\u0442\u0440\u043e\u0439\u0441\u0442\u0432\u0430<\/p>\n<\/th>\n<\/tr>\n

\n

FP32<\/p>\n<\/td>\n

\n

\u2014<\/p>\n<\/td>\n

\n

CUDA<\/p>\n<\/td>\n<\/tr>\n

\n

FP16<\/p>\n<\/td>\n

\n

BuilderFlag.FP16<\/p>\n<\/td>\n

\n

CUDA<\/p>\n<\/td>\n<\/tr>\n

\n

INT8<\/p>\n<\/td>\n

\n

entropy calibrator + FP16 fallback<\/p>\n<\/td>\n

\n

CUDA<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<\/div>\n<\/details>\n

\n\u0422\u0430\u0431\u043b\u0438\u0446\u0430 \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442\u043e\u0432<\/summary>\n
\n
\n
\n\n\n\n\n\n\n
\n

\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f<\/p>\n<\/th>\n

\n

Lat\/img bs=1 (ms)<\/p>\n<\/th>\n

\n

Thr bs=1 (img\/s)<\/p>\n<\/th>\n

\n

Lat\/img bs=64 (ms)<\/p>\n<\/th>\n

\n

Thr bs=64 (img\/s)<\/p>\n<\/th>\n

\n

Top-1 (%)<\/p>\n<\/th>\n<\/tr>\n

\n

FP32<\/p>\n<\/td>\n

\n

4.90<\/p>\n<\/td>\n

\n

204.2<\/p>\n<\/td>\n

\n

1.60<\/p>\n<\/td>\n

\n

625.3<\/p>\n<\/td>\n

\n

76.16<\/p>\n<\/td>\n<\/tr>\n

\n

FP16<\/p>\n<\/td>\n

\n

1.48<\/p>\n<\/td>\n

\n

675.6<\/p>\n<\/td>\n

\n

0.56<\/p>\n<\/td>\n

\n

1788.5<\/p>\n<\/td>\n

\n

76.14<\/p>\n<\/td>\n<\/tr>\n

\n

INT8 + FP16 fallback<\/strong><\/p>\n<\/td>\n

\n

1.16<\/strong><\/p>\n<\/td>\n

\n

863.3<\/strong><\/p>\n<\/td>\n

\n

0.19<\/strong><\/p>\n<\/td>\n

\n

5331.0<\/strong><\/p>\n<\/td>\n

\n

76.10<\/strong><\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<\/div>\n<\/details>\n

\u041f\u043e\u0431\u0435\u0434\u0438\u0442\u0435\u043b\u044c \u0432\u043d\u0443\u0442\u0440\u0438 TensorRT (\u00d7 \u043e\u0442 torch FP32 eager GPU):<\/strong><\/p>\n

\n
\n\n\n\n\n
\n

\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f<\/p>\n<\/th>\n

\n

Device<\/p>\n<\/th>\n

\n

bs=1 Lat\/img<\/p>\n<\/th>\n

\n

bs=1 Thr<\/p>\n<\/th>\n

\n

bs=64 Lat\/img<\/p>\n<\/th>\n

\n

bs=64 Thr<\/p>\n<\/th>\n

\n

Top-1<\/p>\n<\/th>\n<\/tr>\n

\n

INT8 + FP16 fallback<\/strong><\/p>\n<\/td>\n

\n

CUDA<\/strong><\/p>\n<\/td>\n

\n

1.16 ms (\u00d75.8)<\/strong><\/p>\n<\/td>\n

\n

863 img\/s (\u00d75.8)<\/strong><\/p>\n<\/td>\n

\n

0.19 ms (\u00d78.2)<\/strong><\/p>\n<\/td>\n

\n

5331 img\/s (\u00d78.2)<\/strong><\/p>\n<\/td>\n

\n

76.10<\/strong><\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

\u0412\u044b\u0432\u043e\u0434\u044b:<\/strong> INT8 \u2014 \u00d75.8 \u043f\u043e latency \u0438 throughput \u043f\u0440\u0438 bs=1, \u00d78.2 \u043f\u0440\u0438 bs=64. \u0410\u0431\u0441\u043e\u043b\u044e\u0442\u043d\u044b\u0439 \u0440\u0435\u043a\u043e\u0440\u0434 \u0441\u0440\u0435\u0434\u0438 \u0432\u0441\u0435\u0445 \u0434\u0432\u0438\u0436\u043a\u043e\u0432.<\/p>\n

\"Latency<\/p>\n
Latency \u0438 Throughput vs batch size \u2014 \u0432\u0441\u0435 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 TensorRT \u043d\u0430 CUDA.<\/figcaption><\/div>\n<\/figure>\n

\u0418\u0442\u043e\u0433 \u043f\u043e CPU<\/h3>\n
\n
\n\n\n\n\n\n\n\n
\n

\u0414\u0432\u0438\u0436\u043e\u043a<\/p>\n<\/th>\n

\n

\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f<\/p>\n<\/th>\n

\n

Lat\/img bs=1 (ms)<\/p>\n<\/th>\n

\n

Thr bs=1 (img\/s)<\/p>\n<\/th>\n

\n

Lat\/img bs=64 (ms)<\/p>\n<\/th>\n

\n

Thr bs=64 (img\/s)<\/p>\n<\/th>\n

\n

Top-1 (%)<\/p>\n<\/th>\n<\/tr>\n

\n

ORT<\/strong><\/p>\n<\/td>\n

\n

INT8 static QDQ<\/strong><\/p>\n<\/td>\n

\n

15.4<\/strong><\/p>\n<\/td>\n

\n

64.8<\/strong><\/p>\n<\/td>\n

\n

12.0<\/strong><\/p>\n<\/td>\n

\n

83.5<\/strong><\/p>\n<\/td>\n

\n

73.78<\/strong><\/p>\n<\/td>\n<\/tr>\n

\n

OV<\/p>\n<\/td>\n

\n

INT8<\/p>\n<\/td>\n

\n

18.1<\/p>\n<\/td>\n

\n

55.3<\/p>\n<\/td>\n

\n

18.4<\/p>\n<\/td>\n

\n

54.4<\/p>\n<\/td>\n

\n

73.46<\/p>\n<\/td>\n<\/tr>\n

\n

Torch<\/p>\n<\/td>\n

\n

INT8 static FX<\/p>\n<\/td>\n

\n

26.7<\/p>\n<\/td>\n

\n

37.5<\/p>\n<\/td>\n

\n

25.9<\/p>\n<\/td>\n

\n

38.6<\/p>\n<\/td>\n

\n

75.89<\/p>\n<\/td>\n<\/tr>\n

\n

TVM<\/p>\n<\/td>\n

\n

FP32 default<\/p>\n<\/td>\n

\n

93.2<\/p>\n<\/td>\n

\n

10.7<\/p>\n<\/td>\n

\n

3.5<\/p>\n<\/td>\n

\n

9.7<\/p>\n<\/td>\n

\n

66.7<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

\u0418\u0442\u043e\u0433 \u043f\u043e GPU<\/h3>\n
\n
\n\n\n\n\n\n\n\n
\n

\u0414\u0432\u0438\u0436\u043e\u043a<\/p>\n<\/th>\n

\n

\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f<\/p>\n<\/th>\n

\n

Lat\/img bs=1 (ms)<\/p>\n<\/th>\n

\n

Thr bs=1 (img\/s)<\/p>\n<\/th>\n

\n

Lat\/img bs=64 (ms)<\/p>\n<\/th>\n

\n

Thr bs=64 (img\/s)<\/p>\n<\/th>\n

\n

Top-1 (%)<\/p>\n<\/th>\n<\/tr>\n

\n

TRT<\/strong><\/p>\n<\/td>\n

\n

INT8<\/strong><\/p>\n<\/td>\n

\n

1.16<\/strong><\/p>\n<\/td>\n

\n

863.3<\/strong><\/p>\n<\/td>\n

\n

0.19<\/strong><\/p>\n<\/td>\n

\n

5331.0<\/strong><\/p>\n<\/td>\n

\n

76.10<\/strong><\/p>\n<\/td>\n<\/tr>\n

\n

Torch<\/p>\n<\/td>\n

\n

FP16 + compile(reduce)<\/p>\n<\/td>\n

\n

2.57<\/p>\n<\/td>\n

\n

389.2<\/p>\n<\/td>\n

\n

0.69<\/p>\n<\/td>\n

\n

1454.1<\/p>\n<\/td>\n

\n

76.15<\/p>\n<\/td>\n<\/tr>\n

\n

ORT<\/p>\n<\/td>\n

\n

FP16<\/p>\n<\/td>\n

\n

5.65<\/p>\n<\/td>\n

\n

177.1<\/p>\n<\/td>\n

\n

1.07<\/p>\n<\/td>\n

\n

934.1<\/p>\n<\/td>\n

\n

76.14<\/p>\n<\/td>\n<\/tr>\n

\n

TVM<\/p>\n<\/td>\n

\n

FP32 default<\/p>\n<\/td>\n

\n

6.86<\/p>\n<\/td>\n

\n

145.8<\/p>\n<\/td>\n

\n

0.05<\/p>\n<\/td>\n

\n

285.2<\/p>\n<\/td>\n

\n

66.7<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

\u041d\u0430 \u044d\u0442\u043e\u043c \u0432\u0441\u0451!<\/p>\n

\u041a\u043e\u0434, JSON-\u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442\u044b \u0438 Docker-\u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 \u2014 github.com\/DmitriyValetov\/resnet50-inference-benchmark<\/a>.<\/p>\n<\/div>\n

\u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 https:\/\/habr.com\/ru\/articles\/1053650\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

TL;DR: \u041f\u0440\u043e\u0433\u043d\u0430\u043b ResNet-50 \u0447\u0435\u0440\u0435\u0437 PyTorch, ONNX Runtime, OpenVINO, TensorRT \u0438 TVM \u0432 FP32\/FP16\/INT8\/INT4 \u043d\u0430 CPU Ryzen 9 6900HS \u0438 GPU RTX 3070 Ti Laptop \u0432 46 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f\u0445. \u041b\u0443\u0447\u0448\u0438\u0439 CPU: ONNX Runtime static INT8 \u2014 15.4 ms \/ 64.8 img\/s (\u00d74.0 \u043e\u0442 torch baseline \u043f\u0440\u0438 bs=1). \u041b\u0443\u0447\u0448\u0438\u0439 GPU: TensorRT INT8 \u2014 1.16 ms \/ 863 img\/s (\u00d75.8). torch.compile + FP16 \u0434\u0430\u0451\u0442 \u00d72.6 \u0443\u0441\u043a\u043e\u0440\u0435\u043d\u0438\u044f \u0431\u0435\u0437 \u0441\u043c\u0435\u043d\u044b \u0434\u0432\u0438\u0436\u043a\u0430. \u041a\u043e\u0434 \u0438 \u0434\u0430\u043d\u043d\u044b\u0435: github.com\/DmitriyValetov\/resnet50-inference-benchmark.\u0412\u0432\u0435\u0434\u0435\u043d\u0438\u0435\u0415\u0441\u0442\u044c \u0442\u0430\u043a\u0430\u044f \u0440\u0443\u0431\u0440\u0438\u043a\u0430 \u2014 \u0431\u0435\u043d\u0447\u043c\u0430\u0440\u043a\u0438 \u0433\u043e\u043d\u044f\u0442\u044c. \u042d\u0442\u0430 \u0441\u0442\u0430\u0442\u044c\u044f \u043a\u0430\u043a \u0440\u0430\u0437 \u0438\u0437 \u0442\u0430\u043a\u0438\u0445. \u041f\u043e \u043c\u043e\u0442\u0438\u0432\u0430\u043c \u0440\u044f\u0434\u0430 \u043f\u0443\u0431\u043b\u0438\u043a\u0430\u0446\u0438\u0439 \u043f\u0440\u043e inference-\u0434\u0432\u0438\u0436\u043a\u0438 \u0437\u0430\u0445\u043e\u0442\u0435\u043b\u043e\u0441\u044c \u0441\u0434\u0435\u043b\u0430\u0442\u044c \u0441\u0432\u043e\u044e \u2014 \u0441 \u0431\u043e\u043b\u0435\u0435 \u043f\u043e\u0434\u0440\u043e\u0431\u043d\u044b\u043c\u0438 \u043c\u0435\u0442\u0440\u0438\u043a\u0430\u043c\u0438 \u0438 \u043e\u0442\u043a\u0440\u044b\u0442\u044b\u043c \u043a\u043e\u0434\u043e\u043c.Inference-\u0434\u0432\u0438\u0436\u043a\u0438 \u043d\u0430\u043c \u043d\u0443\u0436\u043d\u044b, \u043a\u043e\u0433\u0434\u0430 \u043c\u043e\u0434\u0435\u043b\u044c \u043e\u0431\u0443\u0447\u0435\u043d\u0430 \u0438 \u0435\u0451 \u043d\u0443\u0436\u043d\u043e \u0433\u043e\u043d\u044f\u0442\u044c \u0432 \u043f\u0440\u043e\u0434\u0435, \u043b\u0438\u0431\u043e \u043c\u0430\u0441\u0448\u0442\u0430\u0431\u043d\u043e \u0432\u0430\u043b\u0438\u0434\u0438\u0440\u043e\u0432\u0430\u0442\u044c. \u0412 \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0435\u0439 \u0442\u0430\u0431\u043b\u0438\u0446\u0435 \u043f\u0440\u0438\u0432\u0435\u0434\u0443 \u043c\u043d\u043e\u0433\u043e\u043e\u0431\u0440\u0430\u0437\u0438\u0435 \u0434\u0432\u0438\u0436\u043a\u043e\u0432 \u0438 \u0437\u043e\u043d\u0443 \u0438\u0445 \u043f\u0440\u0438\u043c\u0435\u043d\u0438\u043c\u043e\u0441\u0442\u0438:\u0424\u0440\u0435\u0439\u043c\u0432\u043e\u0440\u043a\u0421\u0435\u0440\u0432\u0435\u0440\/\u041e\u0431\u043b\u0430\u043a\u043eEdge\u041f\u0440\u0438\u043c\u0435\u0447\u0430\u043d\u0438\u0435TensorRT\u2705\u2705\u0422\u043e\u043b\u044c\u043a\u043e NVIDIA GPUONNX Runtime\u2705\u2705\u0423\u043d\u0438\u0432\u0435\u0440\u0441\u0430\u043b\u044c\u043d\u044b\u0439 \u0432\u0430\u0440\u0438\u0430\u043d\u0442OpenVINO\u2705\u2705\u0412 \u043e\u0441\u043d\u043e\u0432\u043d\u043e\u043c \u0437\u0430\u0442\u043e\u0447\u0435\u043d \u043f\u043e\u0434 intel \u044d\u043a\u043e\u0441\u0438\u0441\u0442\u0435\u043c\u0443, \u043d\u043e \u043d\u0430 rysen \u0442\u043e\u0436\u0435 \u043f\u043e\u043a\u0430\u0437\u0430\u043b \u043f\u0440\u0438\u0440\u043e\u0441\u0442 \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0441\u0442\u0438TVM\u2705\u2705\u041a\u043e\u043c\u043f\u0438\u043b\u044f\u0442\u043e\u0440, \u00b5TVM \u0434\u043b\u044f MCU edge-ai-visionExecuTorch\u274c\u2705\u0417\u0430\u043c\u0435\u043d\u0430 PyTorch MobileTFLite\/LiteRT\u274c\u2705\u0414\u0432\u0438\u0436\u043e\u043a \u043f\u043e\u0434 Android\/iOS\/RPiNCNN\u274c\u2705Tencent edge engineMNN\u274c\u2705Alibaba edge enginePaddle Lite\u274c\u2705Baidu edge engine\u0422\u0430\u043a \u0436\u0435 \u0441\u0443\u0449\u0435\u0441\u0442\u0432\u0443\u044e\u0442 \u0440\u0430\u0437\u043b\u0438\u0447\u043d\u044b\u0435 \u043c\u0435\u0442\u043e\u0434\u044b \u043e\u043f\u0442\u0438\u043c\u0438\u0437\u0430\u0446\u0438\u0438 \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0441\u0442\u0438 (\u0432 \u043d\u0430\u0441\u0442\u043e\u044f\u0449\u0435\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u0440\u0435\u0447\u044c \u043f\u043e\u0439\u0434\u0435\u0442 \u0442\u043e\u043b\u044c\u043a\u043e \u043e \u0442\u0435\u0445, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u043d\u0435 \u043c\u0435\u043d\u044f\u044e\u0442 \u0430\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u0443 \u0441\u0435\u0442\u0438), \u043f\u043e\u0434\u0434\u0435\u0440\u0436\u0438\u0432\u0430\u0435\u043c\u044b\u0435 \u0431\u043e\u043b\u044c\u0448\u0435\u0439 \u0447\u0430\u0441\u0442\u044c\u044e \u0434\u0432\u0438\u0436\u043a\u043e\u0432, \u043f\u0443\u0441\u0442\u044c \u0438 \u0441 \u043c\u0435\u0441\u0442\u043d\u044b\u043c\u0438 \u043d\u044e\u0430\u043d\u0441\u0430\u043c\u0438, \u044d\u0442\u043e: \u0441\u043d\u0438\u0436\u0435\u043d\u0438\u0435 \u0442\u043e\u0447\u043d\u043e\u0441\u0442\u0438 \u0432\u044b\u0447\u0438\u0441\u043b\u0435\u043d\u0438\u0439 (FP32\u2192FP16\/INT8\/INT4), kernel fusion, \u0433\u0440\u0430\u0444\u043e\u0432\u044b\u0435 \u043e\u043f\u0442\u0438\u043c\u0438\u0437\u0430\u0446\u0438\u0438.\u0412 \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u0438\u0441\u043f\u044b\u0442\u0430\u043d\u0438\u044f\u043c \u043f\u043e\u0434\u0432\u0435\u0440\u0433\u043d\u0435\u0442\u0441\u044f ResNet-50 \u0432 46 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f\u0445 \u043d\u0430 5 \u0434\u0432\u0438\u0436\u043a\u0430\u0445, \u0441 \u0446\u0438\u0444\u0440\u0430\u043c\u0438, \u0442\u0430\u0431\u043b\u0438\u0446\u0430\u043c\u0438 \u0438 \u0433\u0440\u0430\u0444\u0438\u043a\u0430\u043c\u0438.\u0418\u0441\u043f\u044b\u0442\u0430\u0442\u0435\u043b\u044c\u043d\u044b\u0439 \u0441\u0442\u0435\u043d\u0434\u041a\u043e\u043c\u043f\u043e\u043d\u0435\u043d\u0442\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044fCPUAMD Ryzen 9 6900HS (8C\/16T, Zen 3+)GPUNVIDIA RTX 3070 Ti Laptop (8 GB GDDR6)RAM32 GB DDR5\u041e\u043a\u0440\u0443\u0436\u0435\u043d\u0438\u0435Docker, Ubuntu 22.04, CUDA 12.8\u041c\u043e\u0434\u0435\u043b\u044cResNet-50, \u0432\u0435\u0441\u0430 ImageNet1K_V1\u0414\u0430\u0442\u0430\u0441\u0435\u0442 \u0434\u043b\u044f \u044d\u0432\u0430\u043b\u043010k \u0438\u0437\u043e\u0431\u0440\u0430\u0436\u0435\u043d\u0438\u0439 (\u043f\u043e\u0434\u043c\u043d\u043e\u0436\u0435\u0441\u0442\u0432\u043e ImageNet val) — Top-1 \/ Top-5 Accuracy\u041a\u0430\u043a \u0441\u0447\u0438\u0442\u0430\u0435\u043c \u0441\u043a\u043e\u0440\u043e\u0441\u0442\u044cLatency\/Throughput \u043f\u0440\u0438 bs=1 \u0438 bs=64Warmup10 \u0438\u0442\u0435\u0440\u0430\u0446\u0438\u0439Measurement50\u2013100 \u0438\u0442\u0435\u0440\u0430\u0446\u0438\u0439, \u043c\u0435\u0434\u0438\u0430\u043d\u0430\u0421\u0435\u0442\u043a\u0430 batch size1, 8, 16, 32, 64Baselinetorch FP32 eager mode\u041c\u0435\u0442\u043e\u0434\u044b\u0412\u0441\u0435\u0433\u043e \u043f\u0440\u043e\u0433\u043d\u0430\u043d\u043e 46 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0439 \u043f\u043e \u043f\u044f\u0442\u0438 \u0434\u0432\u0438\u0436\u043a\u0430\u043c:\u0414\u0432\u0438\u0436\u043e\u043a\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0439\u0427\u0442\u043e \u0432\u043d\u0443\u0442\u0440\u0438PyTorch19FP32\/FP16 CPU+CUDA, autocast, 6 \u00d7 torch.compile, 3 quant (dynamic\/static FX\/PT2E), PT2E+compile \u00d72, W4 GPU (eager + 2 compile)ONNX Runtime8FP32\/FP16\/INT8 dynamic\/INT8 static \u00d7 (CPU + CUDA)OpenVINO3FP32\/FP16\/INT8 \u2014 CPUTensorRT3FP32\/FP16\/INT8 \u2014 CUDATVM13FP32\/FP16\/INT8 \u00d7 (CPU + CUDA), default\/MetaSchedule\/AutoTVM\u0412\u0441\u0435\u0433\u043e46Torch: \u0431\u0435\u0439\u0437\u043b\u0430\u0439\u043d \u0438 \u0431\u043e\u0433\u0430\u0442\u044b\u0439 \u043d\u0430\u0431\u043e\u0440 \u043e\u043f\u0442\u0438\u043c\u0438\u0437\u0430\u0446\u0438\u0439Torch \u2014 \u0442\u043e, \u0441 \u0447\u0435\u0433\u043e \u0432\u0441\u0435 \u043d\u0430\u0447\u0438\u043d\u0430\u044e\u0442: \u043e\u0431\u0443\u0447\u0435\u043d\u0438\u0435, \u044d\u043a\u0441\u043f\u0435\u0440\u0438\u043c\u0435\u043d\u0442\u044b. \u0427\u0430\u0449\u0435 \u0432\u0441\u0435\u0433\u043e \u043d\u0430 \u043d\u0435\u043c \u0438\u043d\u0444\u0435\u0440\u0435\u043d\u0441 \u0432 \u043f\u0440\u043e\u0434\u0435 \u043d\u0435 \u0433\u043e\u043d\u044f\u044e\u0442, \u043d\u043e, \u043a \u043c\u043e\u0435\u043c\u0443 \u0443\u0434\u0438\u0432\u043b\u0435\u043d\u0438\u044e, \u0432\u043d\u0443\u0442\u0440\u0438 \u044d\u043a\u043e\u0441\u0438\u0441\u0442\u0435\u043c\u044b PyTorch \u0435\u0441\u0442\u044c \u0431\u043e\u043b\u044c\u0448\u043e\u0439 \u0441\u043f\u0435\u043a\u0442\u0440 \u043d\u0430\u0441\u0442\u0440\u043e\u0439\u043a\u0438 \u0438\u043d\u0444\u0435\u0440\u0435\u043d\u0441\u0430.\u0421\u0435\u0442\u0430\u043f\u044b torchPrecision\u041c\u0435\u0442\u043e\u0434\u0423\u0441\u0442\u0440\u043e\u0439\u0441\u0442\u0432\u0430\u041f\u0440\u0438\u043c\u0435\u0447\u0430\u043d\u0438\u0435FP32eagerCPU, CUDAbaselineFP16model.half()CPU\u0432\u0445\u043e\u0434 float16FP16torch.amp.autocastCUDA\u0432\u0435\u0441\u0430 FP32, \u0433\u0434\u0435 FP16 \u2014 \u0432\u044b\u0431\u0438\u0440\u0430\u0435\u0442 PyTorchFP32torch.compile (reduce-overhead\/max-autotune)CPUInductorFP16torch.compile (reduce-overhead\/max-autotune)CPU, CUDAInductorINT8torchao weight-onlyCPUdynamic, \u0442\u043e\u043b\u044c\u043a\u043e \u0432\u0435\u0441\u0430INT8FX static PTQ (prepare \u2192 \u043a\u0430\u043b\u0438\u0431\u0440\u043e\u0432\u043a\u0430 \u2192 convert)CPU\u0432\u0435\u0441\u0430 + \u0430\u043a\u0442\u0438\u0432\u0430\u0446\u0438\u0438INT8PT2E (torch.export + x86-\u043a\u0432\u0430\u043d\u0442\u0430\u0439\u0437\u0435\u0440)CPU+ compile \u043e\u043f\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043eINT4 W4torchao, Linear \u2192 INT4, \u043c\u043e\u0434\u0435\u043b\u044c BF16CUDAweight-only, eager + compile\u0422\u0430\u0431\u043b\u0438\u0446\u0430 \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442\u043e\u0432\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044fDeviceLat\/img bs=1 (ms)Thr bs=1 (img\/s)Lat\/img bs=64 (ms)Thr bs=64 (img\/s)Top-1 (%)FP32 (cpu baseline)CPU61.616.2119.68.476.15FP32 + compile(reduce)CPU77.213.056.217.876.15FP16 (half)CPU2250.10.41230.00.876.15INT8 dynamicCPU150.76.6120.58.376.09INT8 static FXCPU26.737.525.938.675.89INT8 PT2ECPU158.16.3260.43.876.04PT2E + compile(max)CPU66.014.559.616.876.04FP32 (gpu baseline)CUDA6.68149.81.55646.476.15FP16 autocastCUDA9.4105.90.941094.076.15FP16 + compile(reduce)CUDA2.57389.20.691454.176.15FP16 + compile(max)CUDA3.69345.20.711412.876.15INT4 W4 eagerCUDA8.09123.60.981019.376.08INT4 W4 + compile(reduce)CUDA3.30303.20.781290.976.03Lat\/img = \u0437\u0430\u0434\u0435\u0440\u0436\u043a\u0430 \u043d\u0430 \u043e\u0434\u043d\u043e \u0438\u0437\u043e\u0431\u0440\u0430\u0436\u0435\u043d\u0438\u0435 (per-batch latency \u00f7 batch size). Thr = \u043f\u0440\u043e\u043f\u0443\u0441\u043a\u043d\u0430\u044f \u0441\u043f\u043e\u0441\u043e\u0431\u043d\u043e\u0441\u0442\u044c (img\/s). Top-1 (%) = \u0434\u043e\u043b\u044f \u0438\u0437\u043e\u0431\u0440\u0430\u0436\u0435\u043d\u0438\u0439, \u0434\u043b\u044f \u043a\u043e\u0442\u043e\u0440\u044b\u0445 \u043c\u043e\u0434\u0435\u043b\u044c \u043f\u0440\u0430\u0432\u0438\u043b\u044c\u043d\u043e \u043f\u0440\u0435\u0434\u0441\u043a\u0430\u0437\u0430\u043b\u0430 \u043a\u043b\u0430\u0441\u0441.\u041f\u043e\u0431\u0435\u0434\u0438\u0442\u0435\u043b\u0438 \u0432\u043d\u0443\u0442\u0440\u0438 Torch\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044fDevicebs=1 Lat\/imgbs=1 Thrbs=64 Lat\/imgbs=64 ThrTop-1INT8 static FXCPU26.7 ms (\u00d72.3)37.5 img\/s (\u00d72.3)25.9 ms (\u00d74.6)38.6 img\/s (\u00d74.6)75.89FP16 + compile(reduce)GPU2.57 ms (\u00d72.6)389 img\/s (\u00d72.6)0.69 ms (\u00d72.2)1454 img\/s (\u00d72.2)76.15\u00d7-\u044b \u043e\u0442\u043d\u043e\u0441\u0438\u0442\u0435\u043b\u044c\u043d\u043e torch FP32 eager \u043d\u0430 \u0441\u043e\u043e\u0442\u0432\u0435\u0442\u0441\u0442\u0432\u0443\u044e\u0449\u0435\u043c \u0443\u0441\u0442\u0440\u043e\u0439\u0441\u0442\u0432\u0435Latency \u0438 Throughput vs batch size \u2014 \u0432\u0441\u0435 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 PyTorch \u043d\u0430 CPU.Latency \u0438 Throughput vs batch size \u2014 \u0432\u0441\u0435 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 PyTorch \u043d\u0430 CUDA.ONNX Runtime\u041f\u0440\u0435\u0434\u0432\u0430\u0440\u0438\u0442\u0435\u043b\u044c\u043d\u043e \u043c\u043e\u0434\u0435\u043b\u044c \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0438\u0440\u0443\u0435\u0442\u0441\u044f \u0438\u0437 torch \u0432 onnx (opset 17), \u0430 \u043f\u043e\u0442\u043e\u043c \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0441\u044f \u0438\u043d\u0444\u0435\u0440\u0435\u043d\u0441 \u0447\u0435\u0440\u0435\u0437 ONNX Runtime. FP32 \u0438 FP16 \u0433\u0440\u0430\u0444\u044b \u043d\u0430\u043f\u0440\u044f\u043c\u0443\u044e \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0438\u0440\u0443\u044e\u0442\u0441\u044f \u0438\u0437 torch, \u0430 \u043a\u0432\u0430\u043d\u0442\u043e\u0432\u0430\u043d\u043d\u044b\u0435 INT8 \u0441\u0442\u0440\u043e\u044f\u0442\u0441\u044f \u0443\u0436\u0435 \u0438\u0437 FP32-ONNX \u0441\u0440\u0435\u0434\u0441\u0442\u0432\u0430\u043c\u0438 onnxruntime.quantization.\u0421\u0435\u0442\u0430\u043f\u044b onnxruntimePrecisionQuant\u0423\u0441\u0442\u0440\u043e\u0439\u0441\u0442\u0432\u0430FP32\u2014CPU, CUDAFP16\u2014CPU, CUDAINT8dynamic (weight-only)CPU, CUDAINT8static (QDQ)CPU, CUDA\u0422\u0430\u0431\u043b\u0438\u0446\u0430 \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442\u043e\u0432\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044fDeviceLat\/img bs=1 (ms)Thr bs=1 (img\/s)Lat\/img bs=64 (ms)Thr bs=64 (img\/s)Top-1 (%)FP32CPU27.935.925.938.676.15FP16CPU64.815.434.329.176.14Dynamic INT8CPU26.537.838.525.975.54Static INT8 (QDQ)CPU15.464.812.083.573.78FP32CUDA5.27189.81.69591.976.15FP16CUDA5.65177.11.07934.176.14Static INT8CUDA9.25108.22.27440.975.73Dynamic INT8CUDA46.721.438.026.375.54\u041f\u043e\u0431\u0435\u0434\u0438\u0442\u0435\u043b\u0438 \u0432\u043d\u0443\u0442\u0440\u0438 ONNX Runtime (\u0432\u0441\u0435 \u043c\u043d\u043e\u0436\u0438\u0442\u0435\u043b\u0438 \u2014 \u043a FP32 eager):\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044fDevicebs=1 Lat\/imgbs=1 Thrbs=64 Lat\/imgbs=64 ThrTop-1Static INT8 (QDQ)CPU15.4 ms (\u00d74.0)64.8 img\/s (\u00d74.0)12.0 ms (\u00d710.0)83.5 img\/s (\u00d710.0)73.78FP16GPU5.65 ms (\u00d71.2)177 img\/s (\u00d71.2)1.07 ms (\u00d71.4)934 img\/s (\u00d71.4)76.14Latency \u0438 Throughput vs batch size \u2014 \u0432\u0441\u0435 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 ONNX Runtime \u043d\u0430 CPU.Latency \u0438 Throughput vs batch size \u2014 \u0432\u0441\u0435 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 ONNX Runtime \u043d\u0430 CUDA.OpenVINO\u0414\u0430, \u044d\u0442\u043e\u0442 \u0444\u0440\u0435\u0439\u043c\u0432\u043e\u0440\u043a \u0437\u0430\u0442\u043e\u0447\u0435\u043d \u043f\u043e\u0434 \u0440\u0430\u0431\u043e\u0442\u0443 \u0441 \u0436\u0435\u043b\u0435\u0437\u043e\u043c intel, \u043d\u043e \u044f \u0442\u0430\u043a\u0438 \u043f\u043e\u043f\u0440\u043e\u0431\u0443\u044e \u0435\u0433\u043e \u043d\u0430 rysen \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u043e\u0440\u0435. \u041c\u043e\u0434\u0435\u043b\u044c \u0442\u0430\u043a\u0436\u0435 \u0438\u0437\u043d\u0430\u0447\u0430\u043b\u044c\u043d\u043e \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0438\u0440\u0443\u0435\u0442\u0441\u044f \u0438\u0437 torch \u0432 onnx (opset 17) \u0438 \u043a\u043e\u043d\u0432\u0435\u0440\u0442\u0438\u0440\u0443\u0435\u0442\u0441\u044f \u0432 OpenVINO IR (.xml\/.bin) \u0447\u0435\u0440\u0435\u0437 openvino.convert_model. FP16 \u2014 \u0441\u0436\u0430\u0442\u0438\u0435\u043c \u0432\u0435\u0441\u043e\u0432 \u043f\u0440\u0438 \u0441\u043e\u0445\u0440\u0430\u043d\u0435\u043d\u0438\u0438 IR (compress_to_fp16=True \u0432 openvino.convert_model), INT8 \u2014 \u0441\u0442\u0430\u0442\u0438\u0447\u0435\u0441\u043a\u0438\u0439 PTQ \u0441 \u043a\u0430\u043b\u0438\u0431\u0440\u043e\u0432\u043a\u043e\u0439 (FakeQuantize \u0432 \u0433\u0440\u0430\u0444\u0435, INT8 \u0432\u0435\u0441\u0430 \u0438 \u0430\u043a\u0442\u0438\u0432\u0430\u0446\u0438\u0438, \u043a\u0430\u043b\u0438\u0431\u0440\u043e\u0432\u043a\u0430 \u043d\u0430 \u0442\u0435\u0445 \u0436\u0435 \u043a\u0430\u0440\u0442\u0438\u043d\u043a\u0430\u0445, \u0447\u0442\u043e \u0434\u043b\u044f torch\/ORT).\u0421\u0435\u0442\u0430\u043f\u044b openvinoPrecisionQuant\u0423\u0441\u0442\u0440\u043e\u0439\u0441\u0442\u0432\u0430FP32\u2014CPUFP16\u2014CPUINT8static PTQCPU\u0422\u0430\u0431\u043b\u0438\u0446\u0430 \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442\u043e\u0432\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044fLat\/img bs=1 (ms)Thr bs=1 (img\/s)Lat\/img bs=64 (ms)Thr bs=64 (img\/s)Top-1 (%)FP3239.925.138.925.776.15FP1638.426.136.927.176.15INT818.155.318.454.473.46\u041f\u043e\u0431\u0435\u0434\u0438\u0442\u0435\u043b\u044c \u0432\u043d\u0443\u0442\u0440\u0438 OpenVINO\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044fDevicebs=1 Lat\/imgbs=1 Thrbs=64 Lat\/imgbs=64 ThrTop-1INT8CPU18.1 ms (\u00d73.4)55.3 img\/s (\u00d73.4)18.4 ms (\u00d76.5)54.4 img\/s (\u00d76.5)73.46\u0412\u044b\u0432\u043e\u0434\u044b: \u041f\u0440\u0438 bs=1 INT8 \u0432\u0442\u0440\u043e\u0435 \u0431\u044b\u0441\u0442\u0440\u0435\u0435 baseline (18.1 ms, \u00d73.4). \u041f\u0440\u0438 bs=64 \u2014 \u00d76.5 \u0431\u044b\u0441\u0442\u0440\u0435\u0435 baseline, \u043d\u043e per-image latency \u043f\u043e\u0447\u0442\u0438 \u043d\u0435 \u043f\u0430\u0434\u0430\u0435\u0442 \u0441 \u0440\u043e\u0441\u0442\u043e\u043c batch (\u0441 18.1 \u0434\u043e 18.4 ms). \u0414\u043b\u044f \u0441\u0440\u0430\u0432\u043d\u0435\u043d\u0438\u044f, ORT static INT8 \u0441\u043d\u0438\u0436\u0430\u0435\u0442 per-image latency \u0441 15.4 \u0434\u043e 12.0 ms. \u041f\u0443\u0441\u0442\u044c OV \u0438 \u0441\u043b\u0430\u0431\u043e \u043c\u0430\u0441\u0448\u0442\u0430\u0431\u0438\u0440\u0443\u0435\u0442\u0441\u044f \u043d\u0430 \u0431\u0430\u0442\u0447\u0438 \u043e\u0442\u043d\u043e\u0441\u0438\u0442\u0435\u043b\u044c\u043d\u043e ORT,\u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442 — \u0434\u043e\u0441\u0442\u043e\u0439\u043d\u044b\u0439.Latency \u0438 Throughput vs batch size \u2014 \u0432\u0441\u0435 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 OpenVINO \u043d\u0430 CPU.TVM\u042f \u0431\u044b\u043b \u043d\u0430\u0441\u043b\u044b\u0448\u0430\u043d, \u0447\u0442\u043e tvm \u0442\u043e\u043d\u043a\u043e \u043d\u0430\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u0435\u0442\u0441\u044f \u043f\u043e\u0434 \u0436\u0435\u043b\u0435\u0437\u043e, \u043d\u043e \u043b\u0438\u0447\u043d\u043e \u043c\u043d\u0435 \u0441\u043e\u0432\u0441\u0435\u043c \u043d\u0435 \u0443\u0434\u0430\u043b\u043e\u0441\u044c \u0432\u044b\u0436\u0430\u0442\u044c \u0438\u0437 \u043d\u0435\u0433\u043e \u044d\u0444\u0444\u0435\u043a\u0442\u0438\u0432\u043d\u043e\u0441\u0442\u044c. \u0422\u0443\u0442 \u0430\u043d\u0430\u043b\u043e\u0433\u0438\u0447\u043d\u043e: torch \u0432 onnx, \u0437\u0430\u0442\u0435\u043c \u0433\u0440\u0443\u0437\u0438\u043c \u0432 Relay \u0438 \u043a\u043e\u043c\u043f\u0438\u043b\u0438\u0440\u0443\u0435\u043c \u043f\u043e\u0434 llvm (CPU) \u0438\u043b\u0438 cuda (GPU). \u0413\u0440\u0430\u0444 \u0444\u0438\u043a\u0441\u0438\u0440\u043e\u0432\u0430\u043d \u043f\u043e batch \u2014 \u0434\u043b\u044f \u043a\u0430\u0436\u0434\u043e\u0433\u043e bs \u043e\u0442\u0434\u0435\u043b\u044c\u043d\u0430\u044f \u043a\u043e\u043c\u043f\u0438\u043b\u044f\u0446\u0438\u044f \u0438 \u0442\u044e\u043d\u0438\u043d\u0433.\u0421\u0435\u0442\u0430\u043f\u044b TVMPrecisionSchedule\u0423\u0441\u0442\u0440\u043e\u0439\u0441\u0442\u0432\u0430FP32default, MetaSchedule, AutoTVMCPU, CUDAFP16default, AutoTVMCPU, CUDAINT8defaultCPU, CUDA\u0422\u0430\u0431\u043b\u0438\u0446\u0430 \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442\u043e\u0432\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044fDeviceLat\/img bs=1 (ms)Thr bs=1 (img\/s)Lat\/img bs=64 (ms)Thr bs=64 (img\/s)Top-1 (%)FP32 defaultCPU93.210.73.59.766.7FP32 MetaScheduleCPU177.45.63.54.466.7FP32 AutoTVMCPU196.15.13.24.966.7FP16 defaultCPU25146.2<0.1403.9<0.1\u2014INT8 defaultCPU638.81.69.71.6\u2014FP32 defaultCUDA6.86145.80.05285.266.7FP16 defaultCUDA15.863.40.04376.466.7INT8 defaultCUDA206.14.84.73.3\u2014Accuracy \u043d\u0430 \u043f\u043e\u0434\u043c\u043d\u043e\u0436\u0435\u0441\u0442\u0432\u0435 \u0438\u0437 100 \u0441\u0435\u043c\u043f\u043b\u043e\u0432 (~76% \u043d\u0430 \u043f\u043e\u043b\u043d\u043e\u043c ImageNet). FP16 CPU \u0438 INT8 \u2014 eval \u043d\u0435 \u0437\u0430\u043f\u0443\u0441\u043a\u0430\u043b\u0441\u044f \u0438\u0437-\u0437\u0430 \u0443\u0436\u0430\u0441\u043d\u043e\u0439 \u043e\u0436\u0438\u0434\u0430\u0435\u043c\u043e\u0439 \u0434\u043b\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0441\u0442\u0438.\u041f\u043e\u0431\u0435\u0434\u0438\u0442\u0435\u043b\u044c \u0432\u043d\u0443\u0442\u0440\u0438 TVM (\u00d7 \u043e\u0442 torch FP32 eager):\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044fDevicebs=1 Lat\/imgbs=1 Thrbs=64 Lat\/imgbs=64 ThrTop-1FP32 defaultCPU93.2 ms (\u00d70.7)10.7 img\/s (\u00d70.7)3.5 ms (\u00d71.2)9.7 img\/s (\u00d71.2)66.7FP32 defaultCUDA6.86 ms (\u00d71.0)145.8 img\/s (\u00d71.0)0.05 ms (\u00d70.4)285.2 img\/s (\u00d70.4)66.7\u0412\u044b\u0432\u043e\u0434\u044b: \u041b\u0443\u0447\u0448\u0438\u0435 \u0441\u0435\u0442\u0430\u043f\u044b \u043f\u043e \u044d\u0444\u0444\u0435\u043a\u0442\u0438\u0432\u043d\u043e\u0441\u0442\u0438 \u043f\u043e\u0440\u044f\u0434\u043a\u0430 eagere mode torch.Latency \u0438 Throughput vs batch size \u2014 \u0432\u0441\u0435 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 TVM \u043d\u0430 CPU.Latency \u0438 Throughput vs batch size \u2014 \u0432\u0441\u0435 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 TVM \u043d\u0430 CUDA.TensorRT\u0412\u0441\u0435 \u0442\u0430\u043a\u0436\u0435 \u044d\u043a\u0441\u043f\u043e\u0440\u0442\u0438\u0440\u0443\u0435\u043c \u043c\u043e\u0434\u0435\u043b\u044c \u0432 onnx, onnx \u043f\u0430\u0440\u0441\u0438\u043c \u0432 TensorRT, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u043f\u0435\u0440\u0435\u0441\u043e\u0431\u0438\u0440\u0430\u0435\u0442 \u0433\u0440\u0430\u0444 \u0437\u0430\u043d\u043e\u0432\u043e \u043f\u043e\u0434 \u043a\u043e\u043d\u043a\u0440\u0435\u0442\u043d\u0443\u044e GPU \u2014 \u0441 kernel fusion, \u0430\u0432\u0442\u043e\u0432\u044b\u0431\u043e\u0440\u043e\u043c \u043e\u043f\u0442\u0438\u043c\u0430\u043b\u044c\u043d\u044b\u0445 \u044f\u0434\u0435\u0440 (tactic sources extended, opt level 5) \u0438 \u0444\u0438\u043a\u0441\u0438\u0440\u043e\u0432\u0430\u043d\u043d\u044b\u043c optimization profile (min=1, max=64). \u041d\u0430 \u0432\u044b\u0445\u043e\u0434\u0435 \u2014 \u0441\u0430\u043c\u043e\u0434\u043e\u0441\u0442\u0430\u0442\u043e\u0447\u043d\u044b\u0439 engine (.engine), \u043d\u0435 \u0442\u0440\u0435\u0431\u0443\u044e\u0449\u0438\u0439 \u043d\u0438 PyTorch, \u043d\u0438 ONNX Runtime. INT8 \u2014 \u0441 entropy-\u043a\u0430\u043b\u0438\u0431\u0440\u043e\u0432\u043a\u043e\u0439 \u043d\u0430 \u0442\u0435\u0445 \u0436\u0435 \u043a\u0430\u0440\u0442\u0438\u043d\u043a\u0430\u0445 \u0438 FP16-fallback \u0434\u043b\u044f \u0441\u043b\u043e\u0451\u0432, \u043d\u0435 \u0432\u043b\u0435\u0437\u0430\u044e\u0449\u0438\u0445 \u0432 INT8.\u0421\u0435\u0442\u0430\u043f\u044b TensorRTPrecisionFeatures\u0423\u0441\u0442\u0440\u043e\u0439\u0441\u0442\u0432\u0430FP32\u2014CUDAFP16BuilderFlag.FP16CUDAINT8entropy calibrator + FP16 fallbackCUDA\u0422\u0430\u0431\u043b\u0438\u0446\u0430 \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442\u043e\u0432\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044fLat\/img bs=1 (ms)Thr bs=1 (img\/s)Lat\/img bs=64 (ms)Thr bs=64 (img\/s)Top-1 (%)FP324.90204.21.60625.376.16FP161.48675.60.561788.576.14INT8 + FP16 fallback1.16863.30.195331.076.10\u041f\u043e\u0431\u0435\u0434\u0438\u0442\u0435\u043b\u044c \u0432\u043d\u0443\u0442\u0440\u0438 TensorRT (\u00d7 \u043e\u0442 torch FP32 eager GPU):\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044fDevicebs=1 Lat\/imgbs=1 Thrbs=64 Lat\/imgbs=64 ThrTop-1INT8 + FP16 fallbackCUDA1.16 ms (\u00d75.8)863 img\/s (\u00d75.8)0.19 ms (\u00d78.2)5331 img\/s (\u00d78.2)76.10\u0412\u044b\u0432\u043e\u0434\u044b: INT8 \u2014 \u00d75.8 \u043f\u043e latency \u0438 throughput \u043f\u0440\u0438 bs=1, \u00d78.2 \u043f\u0440\u0438 bs=64. \u0410\u0431\u0441\u043e\u043b\u044e\u0442\u043d\u044b\u0439 \u0440\u0435\u043a\u043e\u0440\u0434 \u0441\u0440\u0435\u0434\u0438 \u0432\u0441\u0435\u0445 \u0434\u0432\u0438\u0436\u043a\u043e\u0432.Latency \u0438 Throughput vs batch size \u2014 \u0432\u0441\u0435 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 TensorRT \u043d\u0430 CUDA.\u0418\u0442\u043e\u0433 \u043f\u043e CPU\u0414\u0432\u0438\u0436\u043e\u043a\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044fLat\/img bs=1 (ms)Thr bs=1 (img\/s)Lat\/img bs=64 (ms)Thr bs=64 (img\/s)Top-1 (%)ORTINT8 static QDQ15.464.812.083.573.78OVINT818.155.318.454.473.46TorchINT8 static FX26.737.525.938.675.89TVMFP32 default93.210.73.59.766.7\u0418\u0442\u043e\u0433 \u043f\u043e GPU\u0414\u0432\u0438\u0436\u043e\u043a\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044fLat\/img bs=1 (ms)Thr bs=1…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-485550","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/485550","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=485550"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/485550\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=485550"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=485550"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=485550"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}