Search results

  1. Andrey Ignatov

    General Questions about the AI Benchmark implementation

    Yes, K here states for thousands (10K = 10000). Target: max error not affecting the results (accuracy / visual) Per-label error: average per-label L1 loss (=mean absolute error) between the produced outputs and goldens Accuracy, digits: the number of accurately predicted digits after the...
  2. Andrey Ignatov

    IMX8M Plus AI Benchmark failed to run

    AI Benchmark v4.0.4 / Android 11
  3. Andrey Ignatov

    Custom vendor NN libraries not available?

    Well, they would, lots of things have changed since the 27th of May.
  4. Andrey Ignatov

    Inference slowdown on Google Pixel 6 Pro when camera is active

    Hi @andrewg, Sorry for the late response. Yes, the Google Tensor TPU was extensively advertised for being used for photo processing. I do not remember any option allowing to disable it while displaying the camera preview using the standard SurfaceView, though you can just try to run your...
  5. Andrey Ignatov

    Custom vendor NN libraries not available?

    A quick answer: 1) AI Benchmark V4 was based on Android Neural Networks API (NNAPI) specs and formats defined at the end of 2019. Those were quite limited but supported by almost all platforms released since that time. 2) AI Benchmark V5 is using the updated ops and models corresponding to...
  6. Andrey Ignatov

    IMX8M Plus AI Benchmark failed to run

    1. Press on the background (instead of the "Start AI Test" button) before launching the tests, and select the "Run All Tests on CPU" option. 2. Go back to the initial screen and press the volume up button three times to enter the engineering menu. 3. Run this section from the appeared menu...
  7. Andrey Ignatov

    NNAPI usage on Exynos 2100 devices

    Yes, they all support only int8 inference. For floating-point models, you can run them on Mali GPUs either with the Samsung Eden or with the TFLite GPU delegates.
  8. Andrey Ignatov

    NNAPI usage on Exynos 2100 devices

    Yes, TF Converter can drastically affect the performance as it maps the original TF ops to the corresponding TFLite layers which specifications are updated quite often (you can notice that even the standard ops like convolutions or splits have multiple revisions: V1, V2, V3, etc). Thus, it might...
  9. Andrey Ignatov

    Benchmarking the A311D / VIM3 NPU

    Hi @chro, Thanks for the info. Yes, we have some internal plans for including this delegate to one of our next releases, though do not have a concrete timeline for this yet. That looks reasonable, you can find the results of another board with VeriSilicon NPU (VideoSmart VS680) here...
  10. Andrey Ignatov

    NNAPI usage on Exynos 2100 devices

    Yes, the structure seems to be the same. The difference in the runtime is likely caused by using different TF converters: the model you extracted from the APK file was generated using the TF 2.2 nightly build, and lots of things have changed since that time, especially if you are working with...
  11. Andrey Ignatov

    NNAPI usage on Exynos 2100 devices

    Are you using exactly the same conversion script as in this post? As for the model you've shared above, its runtime will not be below 70ms as it is quite heavy. To check if it is running with acceleration, you can first run it on CPU and then with NNAPI to see if there will be a 5-10 times...
  12. Andrey Ignatov

    How to get single score of the score list?

    Hi @Bacon, > how can I get the single score? (like CPU-Q score, INT8 NNAPI, ...) You can find them by tapping on the total score or arrow on the right: > Is there a definition of score in column of ranking table? (like CPU-Q score = all int-8 tests inferenced by CPU) Yes: CPU-Q: INT8...
  13. Andrey Ignatov

    AI Benchmark Nightly 08.21 Released

    Starting from version 4.0.4, we release AI Benchmark Nightly builds for ML developers every month. Compared to the standard version, AI Benchmark Nightly: is built with the latest TensorFlow Lite nightly runtime, provides support for the full range of TensorFlow ops, contains the newest...
  14. Andrey Ignatov

    Live @ MAI Workshop - Leave your questions to the speakers in this thread!

    You can run NN models in the browser with GPU acceleration (WebGL) using the TensorFlow.js. The corresponding AI Benchmark build will also be released soon.
  15. Andrey Ignatov

    Live @ MAI Workshop - Leave your questions to the speakers in this thread!

    Yes, the majority of recorded talks will be available on our YouTube channel.
  16. Andrey Ignatov

    CVPR Mobile AI Workshop @ Live on YouTube

    We are happy to announce that the Mobile AI Workshop organized in conjunction with the CVPR 2021 conference will be streamed live on YouTube and available for everyone! During this event, you will see tutorials from all major mobile SoC vendors including MediaTek, Samsung, Qualcomm, Huawei...
  17. Andrey Ignatov

    Live @ MAI Workshop - Leave your questions to the speakers in this thread!

    Sample question format: Speaker: Andrey Ignatov Presentation Slide: 16 / N.A. Question: What are the advantages and when should TFLite delegates be used for NN inference on mobile devices?
  18. Andrey Ignatov

    Live @ MAI Workshop - Leave your questions to the speakers in this thread!

    We are happy to inform you that the Mobile AI Workshop will be streamed live on YouTube and thus available for everyone! If you have any questions to the speakers / companies delivering talks at this event, you can leave them in this thread. After each talk, we will be checking the forum and...
  19. Andrey Ignatov

    High Dynamic Range Challenge

    Hi @Densen, As one of the main goals for the participants of these challenges is to get a conference publication, this challenge will most likely be prolonged and presented at the ICCV Workshop.
  20. Andrey Ignatov

    much higher latencies with NNAPI on Snapdragon 888

    Hi @noodles, There are basically two issues: 1. The first one is that the ESPCN model contains sigmoid and tanh activations that are not well supported by TFLite quantizer and NNAPI. 2. The second issue is the same as described in this thread, the solution can be found in this post...
  21. Andrey Ignatov

    NNAPI usage on Exynos 2100 devices

    Well, that's not surprising - you are trying to get an output of size 224 x 224 x 64. This code + stride of 3 will work: x_in = tf.compat.v1.placeholder(tf.compat.v1.float32, input_shape) x_1 = create_conv_layer(x_in, 3, 64, [3, 3], name='layer_middle') x_2 = create_conv_layer(x_1, 64, 3, [3...
  22. Andrey Ignatov

    High Dynamic Range Challenge

    No, but your FP32 model will be automatically converted to FP16 during the inference on the Huawei platform.
  23. Andrey Ignatov

    Mobile AI 2021 Challenge Results and Papers Released

    The final results and solutions obtained in the Mobile AI 2021 Challenges are now available online: Learned Smartphone ISP on Mobile NPUs: https://arxiv.org/pdf/2105.07809.pdf Fast Camera Image Denoising on Mobile GPUs: https://arxiv.org/pdf/2105.08629.pdf Real-Time Quantized Image...
  24. Andrey Ignatov

    NNAPI usage on Exynos 2100 devices

    Try to define your model using tf.compat.v1 layers instead of Keras.
  25. Andrey Ignatov

    According to AI benchmark App. @ Android TV OS 10

    Hi @Han-Sang Lee, Which test has failed? As was mentioned above, you can just go to engineering menu and run it separately, then restart the benchmark and you will see the scores. Don't forget to disable the Max Initialization Time limit. Sent you the scores privately.
  26. Andrey Ignatov

    NNAPI usage on Exynos 2100 devices

    You can use TF-v1 API to get the same fully quantized model: converter = tf.compat.v1.lite.TFLiteConverter.from_session(sess, [input], [output]) converter.inference_type = tf.compat.v1.lite.constants.QUANTIZED_UINT8 input_arrays = converter.get_input_arrays() # Define the correct input stats...
  27. Andrey Ignatov

    High Dynamic Range Challenge

    Yes, that's correct.
  28. Andrey Ignatov

    NNAPI usage on Exynos 2100 devices

    You can download a standard MobileNet-V2 TFLite model that will show around 2ms on the Exynos 2100 platform using this link. Note, however, that its accuracy is not high as only a very basic quantization approach was applied. If you use TensorFlow's post-training quantization tools, the...
  29. Andrey Ignatov

    Real-Time Camera Scene Detection Challenge

    Hi @YxChen, The challenge report was sent earlier today.
  30. Andrey Ignatov

    High Dynamic Range Challenge

    Yes, this should be fine - only NHWC format is supported properly by TensorFlow Lite, thus ONNX-TensorFlow converter inserts the transpose layer. You can submit this model as long as it runs fine with the AI Benchmark using the NNAPI acceleration option.
  31. Andrey Ignatov

    According to AI benchmark App. @ Android TV OS 10

    Hi @Han-Sang Lee, Thanks for your question. Yes, you should be able to do this, we have previously benchmarked the Synaptics VS680 platform and everything was fine. Note that if some tests fail - you can try to run them separately from the engineering menu (press the volume up key three times...
  32. Andrey Ignatov

    Benchmarking the A311D / VIM3 NPU

    Hi @endian, The situation with the A311D chipsest is quite complex. First of all, there is no way to access its NPU through Android: it doesn't support Android NN API (NN HAL is missing), there are no custom TensorFlow Lite delegates for this SoC as well as any proprietary SDKs. Secondly...
  33. Andrey Ignatov

    AI Benchmark Nightly 05.21 Released

    Starting from version 4.0.4, we release AI Benchmark Nightly builds for ML developers every month. Compared to the standard version, AI Benchmark Nightly: is built with the latest TensorFlow Lite nightly runtime, provides support for the full range of TensorFlow ops, contains the newest...
  34. Andrey Ignatov

    NNAPI usage on Exynos 2100 devices

    The standard MobileNet-V2 model is used in the benchmark without any structural modifications. If you do the quantization correctly, you should be able to get the same results when running the model using AI Benchmark's PRO Mode.
  35. Andrey Ignatov

    Running AI Benchmark Nightly APK on remote device

    This documentation is private, you need to apply for it from your work email.
  36. Andrey Ignatov

    Learned Smartphone ISP Challenge

    Right now, one possible way to implement this op is to use the standard TF's max / average pooling layer and set the pooling size to be equal to the size of the corresponding feature map. It is unlikely that we will do this - all challenge participants were provided with an ability to test...
  37. Andrey Ignatov

    Higher latencies with NNAPI on Snapdragon 888

    Yes and no. NNAPI HAL is a part of Android firmware image. You cannot install these drivers separately, they are either integrated by the corresponding smartphone vendor or not. If you are working with some prototype devices or boards, it is likely that NN HAL is missing there since SDM888 is a...
  38. Andrey Ignatov

    Running AI Benchmark Nightly APK on remote device

    One can run different AI Benchmark tests separately through the command line, but running custom TFLite model through the ADB shell is unfortunately not supported in the current benchmark build. We will try to add this functionality in the next releases.
  39. Andrey Ignatov

    Real-Time Camera Scene Detection Challenge

    No, it will be sent soon.
  40. Andrey Ignatov

    NNAPI usage on Exynos 2100 devices

    @cruiser, The problem is most likely related to your model quantization codes. There are lots of tricks there, you should try to look at this script first and make sure that you are using uint8 ops only.
  41. Andrey Ignatov

    NNAPI usage on Exynos 2100 devices

    Hi @cruiser, Are you getting this number when running your model using AI Benchmark (PRO Mode -> Custom Model) or with your own scripts?
  42. Andrey Ignatov

    which backbone of deeplabv3+ you have used for bechmark testing?

    Hi @charles, MobileNet-V2 backbone is used in the DeepLabv3+ network.
  43. Andrey Ignatov

    High Dynamic Range Challenge

    Yes, the challenge is still open. The proposed task is probably not very familiar to the majority of participants, but you have still almost two months to develop an efficient solution for this problem.
  44. Andrey Ignatov

    Null classification test data file 7.jpg 0 bytes

    Thanks for the report. I guess there was some problem when downloading the pip package as we didn't see this issue on any other platform. However, please let us know if you encounter this problem again.
  45. Andrey Ignatov

    Real-Time Camera Scene Detection Challenge

    The preliminary results of the Camera Scene Detection challenge: https://docs.google.com/spreadsheets/d/e/2PACX-1vQRHXzkYAgzYkBmHpVUiQtiAeIdVQEhZScjSimUDmoQERnJkSvil78VihDbteQg2AzAMqpYoFMY0JXU/pubhtml?gid=504810025&single=true
  46. Andrey Ignatov

    Monocular Depth Estimation Challenge

    The preliminary results of the Monocular Depth Estimation challenge: https://docs.google.com/spreadsheets/d/e/2PACX-1vQRHXzkYAgzYkBmHpVUiQtiAeIdVQEhZScjSimUDmoQERnJkSvil78VihDbteQg2AzAMqpYoFMY0JXU/pubhtml?gid=1536139428&single=true
  47. Andrey Ignatov

    Real-Time Video Super-Resolution Challenge

    The preliminary results of the Video Super-Resolution challenge: https://docs.google.com/spreadsheets/d/e/2PACX-1vQYmfvsCh9rCA8dws5ySCVB3hpe7PKqPd43taAoFvgTnHO0ooPij5LAJu0U7LIf1Zh6hieTaQ4oXxgk/pubhtml?gid=261003568&single=true
  48. Andrey Ignatov

    Real-Time Image Super-Resolution Challenge

    The preliminary results of the Real-Time Image Super-Resolution challenge: https://docs.google.com/spreadsheets/d/e/2PACX-1vQ7tpWw6PtSYyx-G6q_CEUKpIg0RIZCQhY5E-PHdbSUfSO_xp-at38lSzm0HG8beWAySmXa0E3k_SiJ/pubhtml?gid=1733487232&single=true
  49. Andrey Ignatov

    Real Image Denoising Challenge

    The preliminary results of the Real Image Denoising challenge: https://docs.google.com/spreadsheets/d/e/2PACX-1vTTy6zW8Izz3PN-ynk2oec6HYglMZ1F1ANZI2XbKrnUusxUOTwuucbrnDUeYvqBrWMrENSIYc4axrXx/pubhtml?gid=1927469420&single=true
  50. Andrey Ignatov

    Learned Smartphone ISP Challenge

    The preliminary results of the Learned Smartphone ISP challenge: https://docs.google.com/spreadsheets/d/e/2PACX-1vQQ0CVgYv3D1cWNlfguHJ24oLrkoNQ_g1lkNLUsPJTLI62ZoO2GCmQt5IacaUvgXn6YOSds9SDay4l4/pubhtml?gid=1065660300&single=true
Top