More information about result consolidation

[emm2]

Hi all,

I've recently found this benchmark and forum. After reading the article "AI Benchmark: All About Deep Learning on Smartphones in 2019", I still have some questions about how the results were consolidated and made available through Benchmark Details page.

1- On the detailed results page for Phones, there is only a single number for each test (in ms). When the benchmark is run on multiple units of the same device or executed multiple times on the same unit, how is the consolidation to have a single value for each test? Is it an average of all results obtained for that single test on that phone?

2- On the detailed results page for SoCs, are the results a consolidation of all phones that use that SoC? Or results are based on Development Kits with that SoC built-in?

3- On IoT devices, there is no identification of RAM ammount or speed. Is that data relevant to the results obtained?

Thanks in advance for any reponse.

Best regards,

 

[Andrey Ignatov]

how is the consolidation to have a single value for each test? Is it an average of all results obtained for that single test on that phone?

Yes, average or median of the results after removing the outliers.

On the detailed results page for SoCs, are the results a consolidation of all phones that use that SoC? Or results are based on Development Kits with that SoC built-in?

For the majority of SoCs, the results are obtained based in phone measurements, but in some cases development kits are also used (e.g., when no actual devices have been released yet).

On IoT devices, there is no identification of RAM ammount or speed. Is that data relevant to the results obtained?

No, the SoC ranking is not taking into account the amount of RAM that device has.