Mobile AI Workshop General Organization Questions

cblks

New member
The official page of the competition states that the submission deadline is March 21st. Has the competition ended now?
 

moonmina

New member
I would like to ask whether there are any plans to postpone the schedule for this MAI 2025 challenge.
As the submission deadline for the model file and factsheet is approaching, we have yet to receive updates regarding the inference speed and performance on the NPU.

I would like to know the exact deadline for MAI 2025 challenge
 

Andrey Ignatov

Administrator
Staff member
We've just sent the final submission instructions for all challenges.

The official page of the competition states that the submission deadline is March 21st. Has the competition ended now?
No, the deadline is on the 28th of March as was stated on the Codalab webpage.

As the submission deadline for the model file and factsheet is approaching, we have yet to receive updates regarding the inference speed and performance on the NPU.
As was stated in the latest instructions, you can send us up to three models this week for runtime validation on the target device.

The deadline for the current competitions cannot be extended due to hard restrictions from CVPR main organizers. However, all current challenges would transition to ICCV challenges with a new deadline set in summer.

We understand that the sRGB challenge has Final Submission Instructions for Get Data in Participate as well as for Evaluation in Learn the Details, which one of these instructions should be referred to for submissions during the Test phase?
Good point, we will fix this. In principle, both instructions tell you to send us the TFLite model, visual results (if test set provided in the challenge) and the factsheet. But we will clarify this.
 

rlghksdbs

New member
I don't understand the mail exactly.

As I understand it, people who want to submit a TFLite model and submit a paper because they need to show a report paper in order to organize a CVPR workshop, are they submitting a paper?

Also, does this mean that the actual ranking of the challenge is done by ICCV?

Or is there a challenge winner in CVPR and a challenge winner in ICCV?
 

Alex_Fall

New member
We've just sent the final submission instructions for all challenges.


No, the deadline is on the 28th of March as was stated on the Codalab webpage.


As was stated in the latest instructions, you can send us up to three models this week for runtime validation on the target device.

The deadline for the current competitions cannot be extended due to hard restrictions from CVPR main organizers. However, all current challenges would transition to ICCV challenges with a new deadline set in summer.


Good point, we will fix this. In principle, both instructions tell you to send us the TFLite model, visual results (if test set provided in the challenge) and the factsheet. But we will clarify this.
Thank you for your prompt reply, I also noticed a problem. I was about to submit information for the sRGB challenge and the system platform detected the need that I need to have 3145 png images. However, as far as I know, the sRGB track does not provide input images for testing in the current testing phase. I'm not sure if this is a bug in the sRGB challenge system platform, my attachment is two system platform logs.
 

Attachments

  • stderr.txt
    213 bytes · Views: 3
  • stdout.txt
    203 bytes · Views: 0

Andrey Ignatov

Administrator
Staff member
As I understand it, people who want to submit a TFLite model and submit a paper because they need to show a report paper in order to organize a CVPR workshop, are they submitting a paper?
These are separate things:
  • To participate in the final competition phase, you need to submit your TFLite model and a factsheet describing your solution to Codalab.
  • In addition to that, you can also submit a separate full-length paper to the CMT paper submission website. If the paper is well written, your solution is performant enough (you don't need to be the challenge winner, but the results should be clearly above some baseline model) or has some interesting ideas, then the paper will be accepted and published in CVPR Conference Workshop Proceedings.
Or is there a challenge winner in CVPR and a challenge winner in ICCV?
Yes, there will be CVPR challenge winners and ICCV challenge winners. We therefore encourage you to submit your current solution now, and then continue with its improvement till the next deadline as you can potentially win both competitions / get two publications.
 

Andrey Ignatov

Administrator
Staff member
However, as far as I know, the sRGB track does not provide input images for testing in the current testing phase. I'm not sure if this is a bug in the sRGB challenge system platform, my attachment is two system platform logs.
Yes, that's right, you can ignore this error. What we need is just a zip archive uploaded to Codalab with the requested TFLite model and a factsheet.
 

cblks

New member
In the Mobile AI 2025 Real Time Rendering Realistic Bokeh Challenge, which of the two submission guidelines should I follow?Final Submission Instructions.pngFinal test phase submission guidelines.png
 

sr_enhance

New member
Before your notification, we had already exhausted the number of submissions on Codalab. Can we increase the number of submissions by 3 times or delete our previous submissions?
 

sr_enhance

New member
Only the TFLite model, visual results (if a test set is provided in the challenge), and situation sheet are needed. Do I need to submit training code?
 

cblks

New member
Excuse me, can we use inference scripts to run the model for inference? For example, in PyTorch, our input is in NCHW format. Can we also use NCHW format to input the model for testing? If NHWC input format is mandatory, can we use inference scripts to convert it into NCHW as model input?
 

Alex_Fall

New member
Before your notification, we had already exhausted the number of submissions on Codalab. Can we increase the number of submissions by 3 times or delete our previous submissions?
Yes, I've wondered that too. Since the first submission was Failed, I kept making corrections and submitting, and have submitted 4 times now. But I guess it might be possible that only the last submission is chosen as a valid submission, hopefully.
 

Andrey Ignatov

Administrator
Staff member
Before your notification, we had already exhausted the number of submissions on Codalab. Can we increase the number of submissions by 3 times or delete our previous submissions?
We extended the number of submissions to 10 in the test phase in all challenges. Note, however, that only your last submission counts.

Only the TFLite model, visual results (if a test set is provided in the challenge), and situation sheet are needed. Do I need to submit training code?
Training code is also needed as we check some submissions for reproducibility.
 

Andrey Ignatov

Administrator
Staff member
Excuse me, can we use inference scripts to run the model for inference? For example, in PyTorch, our input is in NCHW format. Can we also use NCHW format to input the model for testing? If NHWC input format is mandatory, can we use inference scripts to convert it into NCHW as model input?

ai-edge-torch plugin automatically converts PyTorch NCHW models to TFLite NHWC models:

https://github.com/aiff22/MAI-2025-Workshop/blob/main/pytorch_to_tflite.py

If you have any issues with it, please post in this thread, we will then check this separately.
 

xiachi

New member
I would like to know whether the evaluation of the runtime component in the final score calculation for the Real-Time Image Super-Resolution Challenge is based solely on the NPU runtime, or if it incorporates a combined assessment of both CPU and NPU runtimes.
 

cblks

New member
What should I do if the log operator is not supported when converting PyTorch to TFLite? We have already resolved issues related to the if operator and dynamic sizing, but unfortunately, there seem to be so many problems along the way.
 

Andrey Ignatov

Administrator
Staff member
I would like to know whether the evaluation of the runtime component in the final score calculation for the Real-Time Image Super-Resolution Challenge is based solely on the NPU runtime, or if it incorporates a combined assessment of both CPU and NPU runtimes.
Yes, it is based solely on NPU runtime, CPU runtime is provided only for the reference.

However, if the final submission won't run on NPU, its CPU runtime will be used instead.
 

Chengzhi

New member
the submission limit on the Codalab is 300MB, but the final submission (the TFLite model + visual results) exceeds that. Is it possible to submit them separately? Thanks in advance!
 

cblks

New member
Dear Organizers, for the Real-Time Realistic Bokeh Rendering Challenge, may we submit a zip file that contains both the items specified in the figure and the rendered results on the test set?
 

Attachments

  • 1743075022815.png
    1743075022815.png
    68.6 KB · Views: 6

Andrey Ignatov

Administrator
Staff member
the submission limit on the Codalab is 300MB

If your submission exceeds 300Mbs, please upload your visual results or codes to a separate shared storage platform and provide the corresponding link in your factsheet.

may we submit a zip file that contains both the items specified in the figure and the rendered results on the test set?

Yes, you can submit everything in one zip file.
 

xiachi

New member
Dear organizers, regarding Q4 in the QA:
Q4: The results of my converted TFLite model do not correspond to the results of the original PyTorch model. What should I do?
A4: Add a note at the beginning of your factsheet about this case. We will then check your conversion codes and if the issue is indeed with the ai-edge-torch plugin, then we will use your original PyTorch model for generation of numerical and visual results.

I have some concerns about this. First, I also trained my model using PyTorch, and I manually built the TensorFlow computation graph and converted it into a quantized TFLite model. I believe there are no technical barriers in the conversion from PyTorch to TFLite. However, since the operators in PyTorch and TensorFlow are not perfectly aligned, there will inevitably be a slight accuracy loss during the conversion process (approximately 0.002dB). In your response to Q4, you mentioned using PyTorch for result generation, which I find to be an unfair comparison. If we generate results using PyTorch, what is the purpose of converting the model to TFLite? It seems to undermine the entire effort of converting to TFLite, as the goal is typically to deploy the model on edge devices where TFLite is optimized. Using the original PyTorch model for evaluation defeats the purpose of demonstrating the effectiveness of the conversion.
 

Andrey Ignatov

Administrator
Staff member
there will inevitably be a slight accuracy loss during the conversion process (approximately 0.002dB)
First of all, this is not about 0.002dB accuracy difference - this is about some ops that are completely screwed up by the ai-edge-torch converter, so that you get total corruptions instead of real output.

Secondly, this primarily refers to the challenges, where we have an unconstrained track but still require the participants to submit TFLite models. Some teams, e.g., were developing transformer models, and in this case neither TF re-implementation nor manual PyTorch-to-TF conversion is really possible.
 

xiachi

New member
First of all, this is not about 0.002dB accuracy difference - this is about some ops that are completely screwed up by the ai-edge-torch converter, so that you get total corruptions instead of real output.

Secondly, this primarily refers to the challenges, where we have an unconstrained track but still require the participants to submit TFLite models. Some teams, e.g., were developing transformer models, and in this case neither TF re-implementation nor manual PyTorch-to-TF conversion is really possible.
Alright, I understand.
 

Andrey Ignatov

Administrator
Staff member
Dear organizers,i can't download test datasets,the page shows 404 not found
You don't need to upload the results on the test images in this challenge.

have you solved these problems?i've met the same
That's another issue, we've mentioned during yesterday's Q&A session that for the denoising challenge the data would be provided later when this challenge transitions to ICCV.
 

sr_enhance

New member
When calculating PSNR for sRGB image enhancement, is it calculated at a resolution of 100x100 or at the original image resolution?
 

Jieson

New member
Is the ‘ONE DAY WORKSHOP/TUTORIAL PASS’ sufficient to cover the registration requirement for publishing a paper in the MAI Workshop?

I noticed the registration webpage states: "Each paper (Main Conference AND Workshop) MUST be registered under an AUTHOR." and "Virtual registrations will not cover a paper submission - even workshop papers." My question is: If my paper is accepted by the MAI workshop, can I use the "ONE DAY WORKSHOP/TUTORIAL PASS" for registration?
1743578146761.png
 

Andrey Ignatov

Administrator
Staff member
Dear organizers, when will final results be released ?
For the majority of challenges, the results would be sent this week.

Our paper has been conditionally accepted. How do we submit the revised paper?
You need to follow the final submission instructions. All conditionally accepted papers would be checked before sending them to the publisher to see whether all issues are fixed. If this was not done, the paper would be rejected.

Is the ‘ONE DAY WORKSHOP/TUTORIAL PASS’ sufficient to cover the registration requirement for publishing a paper in the MAI Workshop?
No, you need to select the "Author" registration option even if you have only one workshop paper. That's not really fair, but this was always the rule at the CVPR.
 

sr_enhance

New member
Dear organizer, can the status of the papers submitted on cmt3 be turned over as soon as possible so that we can register for the CVPR conference before April 30?
 

xiachi

New member
Dear Organizers, I have a few questions:
1. Are there any further actions that top-ranked participants need to take moving forward?
2. Will there be any certificates or similar materials (e.g., awards) provided to recognize the participants' achievements?
3. For those who did not submit a workshop paper, will it be necessary for them to attend the CVPR event in person?
 
Top