Real-Time Video Super-Resolution Challenge

gauss

New member
Dear organizers:
There is a question that whether this formula is reasonable. According to this formula, for the same scores, the power consumption increases 0.1 Watt every time, PSNR needs to be increased by 3dB. Is the proportion of power consumption too weight?
Thanks and wait for your reply.
 

UpUpUp

New member
Acoording to the rule "To be eligible for prizes, the participants' score must improve the baseline performance provided by the challenge organizers. " in the "Terms and Conditions",the psnr must larger than 27.65 and the Energy consumption must be less than 0.49?My understanding is correct?

Yes, that is correct.


Dear organizers:

Please modify the contest rules on this page.
The psnr must larger than 27.65 and the energy consumption must be less than 0.49.


1658456307394.png
 
Last edited:

eisblume

New member
Dear organizers:
I have uploaded a quantized model, but observed that the quantized has less letancy but more energy consumption, which is not normal. I would like to know why is the reason for this case? Is it a energy measurement error or my quantized model not correct (my model is fully quantized with INT8, including input and output). My submission id is "sisyphus4869". Thanks and wait for your reply~:giggle:
 

ManYu

New member
Thanks for your clarification, but I have two questions regarding the rule:

1. Does it mean that the scoring formula in the "Learn the details" -> "evaluation" is not correct? Because the examples here should all be graded as invalid according to the above statement.


View attachment 77

2. How can I get a precise energy consumption in develop phase? ( as you said that there might be some measurement error in the automatic evaluation pipeline), and if we cannot get the precise energy score, what will this be handled in the final ranking if my energy in the validation leaderboard is under the threshold but in your manual test is slightly over the boarderline?

Thanks and wait for your reply~
1. Sorry for the mistake. We will grade your submissions based on the scoring policy for the final ranking of the competition. And to be eligible for prizes (not disqualified!), your total score should be higher than the baseline. That is, your total score should be higher than 71.399.

2. Due to the large number of submissions uploaded everyday continuously, it is hard to maintain the consistency of the device status and test every submission manually. Therefore, a slight error is inevitable. We will rank your submission based on the final manual test. As mentioned above, your submission won’t be marked as “invalid” even if your energy consumption is over 0.49.
 
Last edited:

ManYu

New member
Dear organizers:

Please modify the contest rules on this page.
The psnr must larger than 27.65 and the energy consumption must be less than 0.49.


View attachment 80
Sorry for the previous mistake. We will grade your submissions based on the scoring policy for the final ranking of the competition. And to be eligible for prizes (not disqualified!), your total score should be higher than the baseline. That is, your total score should be higher than 71.399. The hard constraint of PSNR and latency stays the same, and there are no hard constraint on the energy consumption.
 

ManYu

New member
Dear organizers:
There is a question that whether this formula is reasonable. According to this formula, for the same scores, the power consumption increases 0.1 Watt every time, PSNR needs to be increased by 3dB. Is the proportion of power consumption too weight?
Thanks and wait for your reply.
Yes, power is weighted for this challenge. As the real video SR use case is expected to be minutes+. This puts power consumption the first priority for the task. In this challenge, we try to set the PSNR/latency as the hard constraint for the task (minimum requirements for the user experience). And invites the community to design a solution that is as low power as possible.
 

tryagain

New member
according to the scoring formula, the lightweight network scores High, but we test two different configuration lightweight networks in order to choose the best configuration, but the energy consumption of both are negative(-0.03 and -0.003), so I want to konw :1、which model (-0.03 or -0.003 )is the best?suppose the PSNR are same and why?2、the energy consumption of the final manual test are all positive?3、what will you do if you get negative energy consumption?all equal to ZERO?
 
Last edited:

UpUpUp

New member
Dear organizers:

It's time to clarify the final competition rules.😀

From my understanding:

1.The hard constraint of PSNR and latency stays the same:
  • PSNR [dB]: must be larger than 27dB; otherwise, the result would not be graded.
  • Latency per frame [ms]: must be lower than 33ms; otherwise, the result would not be graded.
2.Score should be higher than 71.399
 

tryagain

New member
The test phase will begin , what should we submit to you before test phase ? can we change our model during test phase?
 

gauss

New member
Dear organizers:
The size of REDS is [1, 180, 320, 3](BHWC). However, as shown in the figure below, the size of input tensor is [1,320,180,(3*N)] (BWHC?)and the size of output tensor is [1,1280, 720, 3]. So I want to confirm if the description is correct in the figure below?

1658735027682.png
 
Last edited:

erick

New member
Dear organizers:

Q1:As you say, the input data range for inference is [0, 255]. But my model is trained by the data ranged in [0, 1], does it means that i need to write rescale op into tflite model? At the same time, do i need to add some other pre-processing op to the end of tflite model, such as rescaling(rescale model output from [0, 1] to [0, 255]) or clamp(let the range of output be limited to 0 to 255)?
Q2:If i submit a tflite model using multi-frames as input, how can I let you know which frames I'm using? Because if i using 3 frames as input, maybe my frame index is [-1, 0, 1] or [-2, -1, 0]. I think it all needs to be specified in advance.
 

ManYu

New member
The test phase will begin , what should we submit to you before test phase ? can we change our model during test phase?
Please find the submission guideline at “Participate” and follow the instructions. You have a maximum of three submission in the test phase, and we will grade your last one.

Dear organizers:
The size of REDS is [1, 180, 320, 3](BHWC). However, as shown in the figure below, the size of input tensor is [1,320,180,(3*N)] (BWHC?)and the size of output tensor is [1,1280, 720, 3]. So I want to confirm if the description is correct in the figure below?

View attachment 84
Thank you for the notice, we have fixed the typo.

Dear organizers:

Q1:As you say, the input data range for inference is [0, 255]. But my model is trained by the data ranged in [0, 1], does it means that i need to write rescale op into tflite model? At the same time, do i need to add some other pre-processing op to the end of tflite model, such as rescaling(rescale model output from [0, 1] to [0, 255]) or clamp(let the range of output be limited to 0 to 255)?
Q2:If i submit a tflite model using multi-frames as input, how can I let you know which frames I'm using? Because if i using 3 frames as input, maybe my frame index is [-1, 0, 1] or [-2, -1, 0]. I think it all needs to be specified in advance.
Q1. Yes, that is correct.
Q2. You can specified all implementation for us to correctly run your model in the fact sheet.
 

eisblume

New member
Dear organizers:

There are some issues which I think requires clarification:

(1) I have the same requirement for confirmation of the rule as another participant UpUpUp posted above, which is :
1.The hard constraint of PSNR and latency stays the same:
  • PSNR [dB]: must be larger than 27dB; otherwise, the result would not be graded.
  • Latency per frame [ms]: must be lower than 33ms; otherwise, the result would not be graded.
2.Score should be higher than 71.399
would you please give us a clear confirmation on these rules if this understanding is correct?

(2) As for the baseline of score 71.399 is calculated in validation set, does it mean that our models should perform a better score than 71.399 on val set? Or we should reach a score larger than 71.399 on test set?

(3) Considering the difference of measurement of energy consumption in develop/test phase, what would be done for the negative energy values (which already occurs in develop leaderboard) ? Or it can be confirmed that the energy values in test phase would all be positive?


Thanks for your effort in organizing this challenge, and I would appreciate it if you can reply these concerns ASAP, cause the ddl is approaching, and these rules are of importance to many participants for designing their models.


BRs
 

ManYu

New member
Dear organizers:

There are some issues which I think requires clarification:

(1) I have the same requirement for confirmation of the rule as another participant UpUpUp posted above, which is :

would you please give us a clear confirmation on these rules if this understanding is correct?

(2) As for the baseline of score 71.399 is calculated in validation set, does it mean that our models should perform a better score than 71.399 on val set? Or we should reach a score larger than 71.399 on test set?

(3) Considering the difference of measurement of energy consumption in develop/test phase, what would be done for the negative energy values (which already occurs in develop leaderboard) ? Or it can be confirmed that the energy values in test phase would all be positive?


Thanks for your effort in organizing this challenge, and I would appreciate it if you can reply these concerns ASAP, cause the ddl is approaching, and these rules are of importance to many participants for designing their models.


BRs
(1) (2)
To be marked as valid submission, your score should meet the following rule:
- The hard constraint of PSNR and latency:
  • PSNR [dB]: must be larger than 27dB; otherwise, the result would not be graded.
  • Latency per frame [ms]: must be lower than 33ms; otherwise, the result would not be graded.
- There are no constraints on power consumption.

To be eligible for prizes, your score should meet the following rule:
- Your total score should be higher than the baseline score on test set (The baseline score and your final score on test set will released after Testing phase)
- The score on val set is only for your reference.

(3.)
The negative energy values which occur in the development phase come from the inconsistency of the battery status during the competition and a linear regression prediction. The purpose of the procedure is to provide instant feedback on the power consumption to the participants. After the test phase, we will measure the power consumption of your model manually so there will be no negative value.
 

eisblume

New member
(1) (2)
To be marked as valid submission, your score should meet the following rule:
- The hard constraint of PSNR and latency:
  • PSNR [dB]: must be larger than 27dB; otherwise, the result would not be graded.
  • Latency per frame [ms]: must be lower than 33ms; otherwise, the result would not be graded.
- There are no constraints on power consumption.

To be eligible for prizes, your score should meet the following rule:
- Your total score should be higher than the baseline score on test set (The baseline score and your final score on test set will released after Testing phase)
- The score on val set is only for your reference.

(3.)
The negative energy values which occur in the development phase come from the inconsistency of the battery status during the competition and a linear regression prediction. The purpose of the procedure is to provide instant feedback on the power consumption to the participants. After the test phase, we will measure the power consumption of your model manually so there will be no negative value.
Thanks a lot for your quick reply~ It is much more clear for me~
* BTW, the "baseline score" you said on test set will be also calculated using the formula: score = alpha*PSNR+beta*(1-J) with alpha=1.66 beta=50 the same as develop phase, right?
 

ManYu

New member
Thanks a lot for your quick reply~ It is much more clear for me~
* BTW, the "baseline score" you said on test set will be also calculated using the formula: score = alpha*PSNR+beta*(1-J) with alpha=1.66 beta=50 the same as develop phase, right?
Yes, that is correct.
 

eisblume

New member
Dear organizers:
I have two more question:

(1) Is the hard threshold
PSNR [dB]: must be larger than 27dB; otherwise, the result would not be graded.
applied in val dataset or test dataset?

(2) will the output of tflite clamped to [0, 255] by you in evaluation? or should we clamp the values in the tflite model before output the results?
 
Last edited:

ManYu

New member
Dear organizers:
I have two more question:

(1) Is the hard threshold

applied in val dataset or test dataset?

(2) will the output of tflite clamped to [0, 255] by you in evaluation? or should we clamp the values in the tflite model before output the results?
(1) The rule is applied in the test set
(2) You should clamp the values in your tflite model.
 

UpUpUp

New member
(1) The rule is applied in the test set
(2) You should clamp the values in your tflite model.
Dear organizers:

If you calculate psnr using tflite model output image(cv2.imwrite can automatic clamp value),the tflite model is unnecessary to clamp value.

Can you share your evaluation script? This will be of great help to our work.
 

liuxunchenglxc

New member
After we uploaded the zip archive containing factsheet, TFLite models, checkpoint files, and all codes, the error is:

Traceback (most recent call last): File "/tmp/codalab/tmpo43EJU/run/program/evaluate.py", line 74, in <module>
print("Latency: {}".format(latency))
NameError: name 'latency' is not defined

Therefore, we want to know whether this error is caused by us. If it is caused by us, we want to know the detail about the error. If not, we sincerely hope you can repair it as soon as possible.
 

ManYu

New member
Dear organizers:

If you calculate psnr using tflite model output image(cv2.imwrite can automatic clamp value),the tflite model is unnecessary to clamp value.

Can you share your evaluation script? This will be of great help to our work.
Instead of saving the tflite output to images(cv2.imwrite), we will load your provided result images and check the consistency directly on the tflite model output.
After we uploaded the zip archive containing factsheet, TFLite models, checkpoint files, and all codes, the error is:

Traceback (most recent call last): File "/tmp/codalab/tmpo43EJU/run/program/evaluate.py", line 74, in <module>
print("Latency: {}".format(latency))
NameError: name 'latency' is not defined

Therefore, we want to know whether this error is caused by us. If it is caused by us, we want to know the detail about the error. If not, we sincerely hope you can repair it as soon as possible.
It is expected that the returned submission status is failed because the reconstructed frames and README are not included.
And you should not be able to know your score during the testing phase.
 

eisblume

New member
Instead of saving the tflite output to images(cv2.imwrite), we will load your provided result images and check the consistency directly on the tflite model output.

It is expected that the returned submission status is failed because the reconstructed frames and README are not included.
And you should not be able to know your score during the testing phase.
I have the same question about the output evaluation. Can you please provide the testing sample code (about the post-processing of output and loaded result images, and how they are fed into the calc_psnr or calc_mse function)? Because this can affect the post-processing of network output inside the tflite model (considering the min-max clamp and round to int operation etc.) which may differs the power consumption at final test.

This is important to assure the fairness of final test (if the evaluation is not affected by clamp or round, then the participants who use no clamp/round ops in tflite model can gain a few energy score compared with these who use the clamp/round (if not INT8 model) to generate the uint8 image in 0-255, which is not an expected case.), so we wish you can provide the evaluation code (only some lines about the post-process of output_tflite and output_submit is enough) for the participants to check their final models. Thanks~

BRs.
 

ManYu

New member
I have the same question about the output evaluation. Can you please provide the testing sample code (about the post-processing of output and loaded result images, and how they are fed into the calc_psnr or calc_mse function)? Because this can affect the post-processing of network output inside the tflite model (considering the min-max clamp and round to int operation etc.) which may differs the power consumption at final test.

This is important to assure the fairness of final test (if the evaluation is not affected by clamp or round, then the participants who use no clamp/round ops in tflite model can gain a few energy score compared with these who use the clamp/round (if not INT8 model) to generate the uint8 image in 0-255, which is not an expected case.), so we wish you can provide the evaluation code (only some lines about the post-process of output_tflite and output_submit is enough) for the participants to check their final models. Thanks~

BRs.
We won't apply any post-processing on your output_tflite nor output_submit.
We simply read your submit.png to numpy array and expect the value to be the same as your tflite output, which means the pixel value of your output_submit is in a range of 0-255, then the direct tflite output (the actual numpy array, we won't save your tflite output to image.png.) should also be in 0-255. The inconsistency (cosine similarity < 0.999) will lead to an invalid submission.
 

lwifi

New member
Dear organizers,

What's the meaning of the "submission entry" and "scoring entries" in the fact sheet? Is it the best score submitted to Codalab or the testing result on my local computer?

Thanks.
 

tryagain

New member
[Mobile AI & AIM 2022 Real-Time VSR Challenge (Participant_name)_(submission_entry)]. our team have write the email and set the submission_entry to 1, should we write another email?Will this affect our grades?
 

eisblume

New member
Dear organizers:
When will the preliminary results be released? on codalab overview it shows:
  • 02.08.2022 Preliminary test results release to the participants
Thanks~
 

eisblume

New member
Dear organizers:
When will the preliminary results be released? on codalab overview it shows:
  • 02.08.2022 Preliminary test results release to the participants
Thanks~
As the ddl for paper submission approaching (which is 8 Aug 2022 from the former information), when will the results be released? Will the ddl for challenge paper submission be postponed accordingly as the delay of preliminary results? Thanks and wait for your reply~
 

eisblume

New member
If you are the contact of your team and did not receive any feedback, please wait 2 more days and then please contact your challenge organizers.
Dear organizers:
As the email suggest, we have been waiting for two days and now would like to contact you here for more information about when will the results be released. Moreover, the ddl for challenge paper submission is only 2 days ahead, and we participants need to know the ranking for deciding whether to submit the paper and how the manuscipts should be organized. It will be a little tight for the schedule of paper preparation if the results can not be released in time. Wish you can provide us some information about how the evaluation procedure is going now even if the preliminary results are not finished, which will give us an expectation for the release time. Thanks~😊
 

Jarvis Irving

New member
Dear organizers, thanks for you organizing this interesting challenge. Noticed that the scoring strategy for 2022 has changed compared to 2021, will it continue to change in 2023 or remain the same as in 2022?
Thanks and wish for your reply~:)
 
Top