Search results

Hi @bagofwater, Thank you for your suggestions. For FP16 inference, the targets are generated in FP32 mode that provides an accuracy of 7-8 digits after decimal point, so there are no issues here.

Yes, your model should have only one input layer in order to be executed successfully. The easiest workaround here would be to stack two input tensors together into a single input layer and then unstack them during inference.

Right now, it is not possible to force enable delegates when running custom models, this functionality will be added in the next benchmark version.

You need to use TensorFlow's full integer quantization: https://www.tensorflow.org/lite/performance/post_training_integer_quant You can also find a useful example showing how to obtain a fully quantized model here...

Try force enabling the Neuron delegate in the acceleration settings by pressing > 3 times on this option. The latest Neuron build should generally be compatible with Dimensity 920-based phones running Android T. It should be also accessible through Android NNAPI.

Hi @tvkamara, Yes, unfortunately the latest Android versions do not allow the app to access any files without a user picking them up explicitly. Can you please send us the error logs from the logcat? Oculus Quest 2 is using a highly customized Android version, thus we need more details...

Hi @David Kerr, Thanks! We include the majority of devices which scores can be validated to the ranking table. We also observed the results of a number of your devices in the past. Please contact me by email if you want some of them to be published - we might have a few questions regarding...

TFLite GPU delegate supports only a subset of TFLite / TF ops (mainly related to computer vision), thus it's very common that many NLP models cannot be executed with it. In your case, the problem is with the following 3 layers: GATHER, RESHAPE and SLICE. In principle, it should be possible to...

Hi @Andreas Kirmse, As far as we know, you should be able to download QNN SDK from Qualcomm Createpoint after registering there. However, it might not yet be available for all users now. Well, there is a good reason for this. NNAPI has lots of issues that were not solved by Google, and...

Hi @koka, Unfortunately, the source code of AI Benchmark for Android is not publicly available. Are your modifications related to support of new ops/layers, or to a different inference/acceleration backend?

What's New: Updated TFLite GPU, Qualcomm QNN, MediaTek Neuron and Hexagon NN delegates. Updated in-app ranking table. Various bug fixes and performance improvements. Download this release from the official website or from the Google Play store. Feel free to discuss AI Benchmark results in...

These results were obtained with the vendor's tflite delegate. NNAPI-1.1/1.3 here refers to the set of ops present in the corresponding models.

Yes, it can be integrated to AI Benchmark. Please send us an email with the information about your delegate (andrey at vision.ee.ethz.ch).

Perfect. Yes, there are a couple of changes in Android 13 preventing the lib from working correctly on a number of Dimensity-based devices. The next benchmark update will resolve these problems.

Hi @jinqimu, Thanks for the provided info. Can you please try to install and run this beta benchmark build with an updated Neuron lib and let us know if you still faces this issue?

Detailed AI Benchmark V5 results were released for over 750 Android devices: https://ai-benchmark.com/ranking https://ai-benchmark.com/ranking_detailed Historical results for previous AI Benchmark versions are still available in our archive section: v4.0.3 (year 2021)...

That's interesting, can you please specify your phone model and Android version/build? An info from the logcat for the 1st test would also be useful (if you can get it).

Try to set this option to true.

Yes, that's normal - these models are using specific NLP-based ops that are currently not fully supported by DirectML.

You can sometimes find this information in the logcat when initializing the model with TFLite/NNAPI. Or you can tell us the device model you are using. There is no easy way to do this.

Yes, AI Benchmark sections 5, 6 and 10 are testing exactly this behavior, the results of these tests can be found here.

Yes, all the AIM 2022 award certificates are available here: https://data.vision.ee.ethz.ch/cvl/aim22/AIM2022awards_certificates.pdf

Yes, all the AIM 2022 award certificates are available here: https://data.vision.ee.ethz.ch/cvl/aim22/AIM2022awards_certificates.pdf

https://ai-benchmark.com/workshops/mai/2022/#challenges

The scores reported in this table were using PSNR instead of MOS. In any case, in this challenge we have two official winners - team Antins_cs and ENERZAi, thus this is not changing anything.

Hi @yuchai84, sum of each image's Per-Label Loss / Image number Average per-pixel L1 error.

As there was no time for proofreading of this paper, it was sent directly to the publisher.

Hi @AndreaChi, Only power consumption of the acceleration unit.

No, INT8 and FP16 scores are normalized using different coefficients.

The final results were released using the same link.

Yes, you will receive them closer to the actual ECCV workshop event. We don't know yet when the publisher will release these papers, but we might also upload them to arXiv this months. It will likely be a hybrid event, more information would be published soon on the website.

Yes. An email with the final results was just sent to all challenge participants.

Hi @zhyever, An email with the final submission instructions was sent yesterday to all registered papers.

Only submissions containing a valid challenge report are evaluated during the test phase. Submissions having only a TFLite model are discarded. They, they would be measured once again in an "offline" mode.

Hi @Msss, thanks for noticing this issue. The results of your both submissions are now published in the ranking table.

We can see that you made a successful Codalab submission today at 7:19:51 PM.

You can ignore this, the name of the archive does not matter.

Yes, this is the correct input size. This code shows you how to resize the input tensor of your model.

Did you enable the experimental TFLite converter (converter.experimental_new_converter = True)? Besides that, what TF version are you using? Try to install the latest TF nightly build.

You mean your model requires the input dimensions to be multiple of 2^n? Ok, add a readme.txt file with these requirements to the Model/ folder. You can refer to Keras and TensorFlow examples - the model should be successfully converted to TFLite if you set the input tensor dimensions to...

You should upload your final submission to Codalab only. Submissions sent by email will not be considered.

TFLite supports dynamic input in general, but it seems that you have a blur_image tensor that does not support this. In this case, you can add two additional image resize tensors before and after this layer: the first one would resize the image to the size expected by the blur_image tensor...

The best score in Codalab.

Yes, you can just modify the year. Only last attempt. The total runtime per inference. You should expect to get the results at the end of the next week - checking all submissions in 1 day is not really feasible.

Please check the email that was sent to all challenge participants this Tuesday. It contains a detailed description of the final challenge submission format and data. It is very important that you follow the provided submission guideline precisely so that your submission is not disqualified...

Please check the email that was sent to all challenge participants this Tuesday. It contains a detailed description of the final challenge submission format and data. It is very important that you follow the provided submission guideline precisely so that your submission is not disqualified...

Hi @Haotian Qian, we increased the number of total submissions to 6, so you have one extra attempt.

Test submissions are used just for uploading your final results (including fact sheets and executables), and only your last submission would be considered. We need to ensure that the training method described in your fact sheet is actually real. Otherwise, you can write absolutely anything...

5 times in total. Note that test submissions are used just for uploading your final results, and only your last submission would be considered. Each participant should upload training codes that might be used for checking your submission for reproducibility. All submissions without training...

Yes, your TFLite files and your original codes for training the model.