All questions related to Learned Smartphone ISP Challenge can be asked in this thread.
The RAW images provided to us are 256x256 which is different from what's expected in the final model. So, what should be done here? One immediate thing we could do is resize the RAW images. But I wanted to know from the organizers to get more systematic directions.
Could anyone clarify as to which latency (CPU/GPU) is to be recorded in readme.txt? Do I run the TFLite model on my mobile phone or on MediaTek Dimensity 1000+ (APU)? If the latter, then what is the suggested method?
The RAW validation images we are provided with are of 256x256 resolution. So, my question is what should be the size of the output RGB images? The competition homepage mentions it should be the same as the input size of the RAW images. But then again, the TFLite model's input size is different hence I am a bit confused.
Level 0 is the right one.what would be the right level (5, 4,...,0)
After installing it on my phone (Realme Pro 2) it didn't open. Any clues?
I got the shape of the inputHi @sayak,
The provided raw 256x256px patches are used for checking PSNR / SSIM results of your model. However, when converting it to TFLite format, you should change the size of the input tensor to [1, 544, 960, 4] as is specified here.
No, here you can specify the latency obtained on your own smartphone.
Yes, the size of the outputs used for fidelity scores computation is 256x256px. As is mentioned in the challenge description, the size of the input / output tensors of the submitted TFLite model should be [1, 544, 960, 4] and [1, 1088, 1920, 3], respectively.
1. Does Input come in like the following?( as described in the sample code)
RAW_norm = RAW_combined.astype(np.float32) / (4 * 255)
RAW_norm = RAW_combined.astype(np.float32) / (16 * 255)
Do output channels 0, 1 and 2 correspond to R, G and B? Should output be float32 or uint8?
RAW_norm = RAW_combined.astype(np.float32) / (16 * 255)
bit more light on the core math to scale the pixel values when dealing with different bits of images
[0, 2^N]
(though the real interval is actually a bit smaller), therefore 10-bit images should be divided by ~2^10=1024
, and 14-bit ones - by ~2^14=16384
.output_l0 = tf.nn.tanh(conv_l0_out) * 0.58 + 0.5
would this be affected as well (the final output of PyNet)?
Hi @sayak,
The provided raw 256x256px patches are used for checking PSNR / SSIM results of your model. However, when converting it to TFLite format, you should change the size of the input tensor to [1, 544, 960, 4] as is specified here.
No, here you can specify the latency obtained on your own smartphone.
Yes, the size of the outputs used for fidelity scores computation is 256x256px. As is mentioned in the challenge description, the size of the input / output tensors of the submitted TFLite model should be [1, 544, 960, 4] and [1, 1088, 1920, 3], respectively.
Does we need to generate those PNG image based on TFlite file(if quantize to FP16 or UINT8)?
on Redmi K30 Ultra using its GPU
Hi @sayak,
The provided raw 256x256px patches are used for checking PSNR / SSIM results of your model. However, when converting it to TFLite format, you should change the size of the input tensor to [1, 544, 960, 4] as is specified here.
No, here you can specify the latency obtained on your own smartphone.
Yes, the size of the outputs used for fidelity scores computation is 256x256px. As is mentioned in the challenge description, the size of the input / output tensors of the submitted TFLite model should be [1, 544, 960, 4] and [1, 1088, 1920, 3], respectively.
Now, I just use the dummy data(random noist) to measure the latency on [1, 1088, 1920, 3] tflite file.Is the input to measure the latency dummy tensor or real raw data like provided training data?
And do you use variable tensors or only one fixed tensor to measure the latency??
We need generate the "two" tflie file(output size are [1, 256, 265, 3] and [1, 1088, 1920, 3]) base on the same ckpt file, and than use '[1, 256, 265, 3]' tflite to generate the 256x256 RGB image. Finally summit the RGB image and [1, 1088, 1920, 3] tflite file.
We can use the 256*256 RAW to generate the quantize the '[1, 256, 265, 3]' tflite file, but how to generate the quantized '[1, 1088, 1920, 3]' tflite file (544*960 data missing) ? Sorry for being such a hassle.
s the input to measure the latency dummy tensor or real raw data like provided training data?
And do you use variable tensors or only one fixed tensor to measure the latency?
Dose the "[1, 1088, 1920, 3]" tflite file just for the latency measurement ?
Thanks for your answerYes, exactly.
You can basically use any input images for quantizing the second model.
We are using real data, though this has almost no effecting on the resulting runtime values.
Yes, the size of the input tensor should be fixed and equal to [1, 544, 960, 4].
Yes.
Thanks for your reply, please let me reconfirm it.You can basically use any input images for quantizing the second model.
f you measure the latency using only one real data or measure the latency using multiple real data and then average them.
We can use the dummy data(random noise) to quantize the "[1, 1088, 1920, 3]" tflite file
If we summit the image base on fp tflite for evaluating the quality, but generate the quantized tflite for latency measurement.
Is such a result acceptable?
Could you give me a help regarding the above error?
experimental_new_converter
option to False
when generating the TFLite model. In this case, the old TOCO converter compatible with MediaTek's Neuron Delegate will be used instead of the new MLIR one.Thanks for your answerYou should try one of the following options:
1. Set theexperimental_new_converter
option toFalse
when generating the TFLite model. In this case, the old TOCO converter compatible with MediaTek's Neuron Delegate will be used instead of the new MLIR one.
2. Convert your model using TensorFlow 1.15 (you can install it in a separate python / conda environment).
What's the structure of our submission for uploading at testing phase?
Further question, Is there any restriction for tflite footprint?? (~5MB is it okay?)
Thanks. It seems that Android 11 beta version was released for the Redmi K30. So, I will upgrade Android version and then try the latest AI Benchmark app. Then will it be same environment with your evaluation server??Check the email sent to all challenge participants yesterday.
Yes, this is fine. All models are tested by MediaTek using their Neuron Delegate. It is also present in the latest AI Benchmark app, but requires Android 11 which was not yet released for the Redmi K30.
If the first option fails, please try the second one with TF 1.15.
Then will it be same environment with your evaluation server?
stuck in 'SUBMITTED' status for more than six hours
Thanks. @Andrey IgnatovCheck the email sent to all challenge participants yesterday.
Now my output range is the 0~1, i need the post-processing(rescale to 0~255) to generate PNG image.
Will the organizer provide the test scripts or reference code for RGB img. producing?Yes, you just need to add one additional layer to your model multiplying the outputs by 255.
Will the organizer provide the test scripts or reference code for RGB img. producing?
Thanks for your answer, I want to clarify my questionYour model should not save the images - its outputs will be directly used to compute the final fidelity scores. The output format is the same as of the images you submitted to Codalab before.