All questions related to Real Image Denoising Challenge can be asked in this thread.
No, the input size of the TFLite model should be constant. It will be equal to ~HD resolution, we will send the exact instructions soon when we open the runtime validation submission.Is a tflite model required to work on images with different shapes?
Is it enough to work with 3000x4000 images like the images in the given data?
How to balance the PSNR and runtime?
Which metric is the final ranking based on?
How to report the runtime values for the submission for now?
is the model quantization allowed?
"There was an error retrieving the file. Please try again later or report the issue."
What is the shape of input for the tflite model?
[1, 480, 720, 3]
, just sent an email with clarifications to all challenge participants.Thanks !The size of the input tensor should be[1, 480, 720, 3]
, just sent an email with clarifications to all challenge participants.
Does the evalution system measure the latency if I submit a tflite model?
I think the score is not reasonable.When using this score, every one tend to do faster instead of getting reasonable PSNR. Like an extreme situation:my model just has one identity block, is the score very high?The final submission score will be proportional to the fidelity scores (PSNR) and inversely proportional to the runtime. The exact scoring formula will be announced a bit later.
The score indicates that the PSNR is 1dB higher, which is equivalent to half of the running time. Obviously, if the model size is reduced to approximately 0, the output PSNR will be equal to the input PSNR, the score would be very high. A better solution should be like that, half the runtime is equivalent to an increase >1dB PSNR.I think the score is not reasonable.When using this score, every one tend to do faster instead of getting reasonable PSNR. Like an extreme situation:my model just has one identity block, is the score very high?
emmmmm... is this score right? I think it can't solve the problem i mentioned before.@haha, @chengshen,
You are right, that was actually our mistake (we were previously planning to have one additional loss component increasing the impact of fidelity scores). It's fixed now
I think it can't solve the problem i mentioned before.
~33dB
. Thus, if showing the same runtime, the score of this solution will be ~2^10 =1024
times lower compared to the score of the solutions with 38dB
PSNR. Since the runtime of the above trivial solution will anyway be around several millis, there won't be any problems with the scores.I think there should be a threshold for PSNR to make sure our model actually makes sense. For example, the score should be calculated if and only if PSNR > THRES. THRES can be dependent on some baseline such as a bilateral filter.PSNR of the trivial identity mapping solution is~33dB
. Thus, if showing the same runtime, the score of this solution will be~2^10 =1024
times lower compared to the score of the solutions with38dB
PSNR. Since the runtime of the above trivial solution will anyway be around several millis, there won't be any problems with the scores.
our scripts run the TFLite model and we will submit the output of the converted TFlite model instead of the original model
our scripts should crop the HD images first
because the input size of TFlite model is [480,720].
I think there should be a threshold for PSNR to make sure our model actually makes sense.
If it is released, some could make denoised images with a very large network and use them for training small networks.
Could the organizers update the runtime evaluation results ? The results after the Mar. 8th have not been released so far. Or could we get runtime from other way ?As was mentioned in the email sent yesterday, all runtime evaluation results will be published here:
https://docs.google.com/spreadsheet...fEmAbbZ2fWntgZS5uigJos9gawt2-0bq1pLRa/pubhtml
you will need to submit two TFLite models: processing the original resolution images and 480x720 crops
Andrey said the model with input size 480x720 is only used to calculate runtime.Hi!
I thought test dataset would be the size of 480 x 720 as you mentioned above.
But the test dataset shape is the same as validation dataset (the first dataset we used).
Could you please tell us what happened?
Dear organizer,Yes, you can use this script to produce the results using the obtained TFLite model. Since you need to submit a normal floating-point TFLite network, its results should be identical to the ones obtained with the original TF model.
No, you should process the original full-resolution images without any cropping. The above resolution is used only to validate the runtime of your model. During the final phase, you will need to submit two TFLite models: processing the original resolution images and 480x720 crops.
2. If so, how do you create such tf-lite model?
[1, None, None, 3]
before converting your model.tflite models with "static size input" work on mobile GPUs.
Thank you for response!Just add a TF / Keras input placeholder with shape[1, None, None, 3]
before converting your model.
TFLite model with dynamic input shape will only be used for checking fidelity scores.
x_ = tf.compat.v1.placeholder(tf.float32, [1, None, None, 3], name="input")
ValueError: None is only supported in the 1st dimension. Tensor 'input' has invalid shape '[1, None, None, 3]'.
Or have organizers published these results by some other means ?
please give some advices, thanks a lot.
ValueError: None is only supported in the 1st dimension. Tensor 'input' has invalid shape '[1, None, None, 3]'.
experimental_new_converter
option to True
when converting a model with None dimensions.How would you check the model for runtime and the model for fidelity are exactly same?
Is it okay to submit model for [1, 2432, 3200, 3] input instead of "model_none" for the fidelity verification
3000x4000px
, though we cannot guarantee you that the model won't be tested on images of different size.Thank you very much!! It seems working.Samsung is working on this, will hopefully add new results shortly. For now, please test the models on your own smartphone's GPU with AI Benchmark as the obtained runtime is usually proportional to the one on the Exynos Mali GPU.
As was written in the email, set theexperimental_new_converter
option toTrue
when converting a model with None dimensions.
It's not a problem to check if two TFLite models are the same except for the input dimensions.
No, the input size of the model should be None. You may assume that the resolution of the test images will be3000x4000px
, though we cannot guarantee you that the model won't be tested on images of different size.
when we convert tflite with [1, None, None, 3] input sucessfully, is it correct that the input and output of .tflite are both [1,1,1,3] checked by Netron ?Just add a TF / Keras input placeholder with shape[1, None, None, 3]
before converting your model.
TFLite model with dynamic input shape will only be used for checking fidelity scores.
can not find the runtime results of updated submissions after Mar. 8th from here:
when we convert tflite with [1, None, None, 3] input sucessfully, is it correct that the input and output of .tflite are both [1,1,1,3] checked by Netron ?
When I submitted the final result on codalab, I encountered some problems, can you give me some suggestions?
I submitted one same results by three times, but all are warned by 'Execution time limit exceeded' and marked by 'failed'. Today, I found that platform generated a submission marked by 'finished', does that mean I have sucessfully submmited one result and still remain 2 chances ?As far as I can see, all your submissions are successful now.
does that mean I have sucessfully submmited one result and still remain 2 chances
I guess the runtime measurement is extremely inaccurate. I got the 4x slower result even when I submitted the same model.The speed of our model with submission ID 833911 is 0.06s, however, it is a very big model ran on my own device with 6~7s. Please check the problem and the subsequential submissions following the ID 833911. Thanks very much.
The model was submitted before, it ran in ~2.5s.