Learned Smartphone ISP Challenge

Alpha

New member
Hi, I have some questions.

1. In the score calculation formula for track 1, is the runtime measured in seconds? Or milliseconds?
2. For track 2, is the MOS obtained by comparing the subjective similarity with the GT, or by comparing the different submissions with each other?
For example, result 1 and GT are more similar, and result 2 and GT are not. So result 1 is 5 points and result 2 is 1 point.
For another example, result 2 is more acceptable subjectively than result 1, so result 2 is 5 points and result 1 is 1 point.
What kind of grading standard?

Looking forward to your reply.
 

Andrey Ignatov

Administrator
Staff member
Hi, the link to the validation dataset is not found!

Thanks, the link should work now.

In the score calculation formula for track 1, is the runtime measured in seconds?

The runtime is computed in milliseconds, but this does not matter as it's just a scaling constant.

For track 2, is the MOS obtained by comparing the subjective similarity with the GT, or by comparing the different submissions with each other?

With the ground truth images.
 

saikatdutta

New member
My submission is in proper format but it failed. Can you please check? Username is terra. The error log is as follows:

WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
* daemon not running; starting now at tcp:5037
* daemon started successfully
Execution time limit exceeded!
 

tuvovan

New member
Traceback (most recent call last):
File "/worker/worker.py", line 313, in run
bundles = get_bundle(root_dir, 'run', bundle_url)
File "/worker/worker.py", line 173, in get_bundle
metadata[k] = get_bundle(bundle_path, k, v)
File "/worker/worker.py", line 173, in get_bundle
metadata[k] = get_bundle(bundle_path, k, v)
File "/worker/worker.py", line 133, in get_bundle
z.extractall(bundle_path)
File "/usr/local/lib/python2.7/zipfile.py", line 1040, in extractall
self.extract(zipinfo, path, pwd)
File "/usr/local/lib/python2.7/zipfile.py", line 1028, in extract
return self._extract_member(member, path, pwd)
File "/usr/local/lib/python2.7/zipfile.py", line 1084, in _extract_member
shutil.copyfileobj(source, target)
File "/usr/local/lib/python2.7/shutil.py", line 66, in copyfileobj
fdst.write(buf)
IOError: [Errno 28] No space left on device


I got this error while submitting, under the username tuvovan. Could you please check?
 

tuvovan

New member
Screen Shot 2022-06-16 at 3.45.29 PM.png
I got this error today, I already checked and I can confirm that I have exactly 2258 png images in the folder + model.tflite + readme file.
 

Alpha

New member
Hi, when can we do a speed test on the target platform? I think the balance between runtime and PSNR is still important. Or can you tell me the specific phone model?
 

Minhyeok

New member
Hi, My submission is also failed. Could you please check? The error log is as follows:

WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Traceback (most recent call last):
File "/tmp/codalab/tmpch9OA4/run/program/evaluation.py", line 85, in
raise Exception('Expected %d .png images'%len(ref_pngs))
Exception: Expected 2258 .png images
 

Jaszheng

New member
My submission is in proper format but it failed. Can you please check? Username is terra. The error log is as follows:

WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
* daemon not running; starting now at tcp:5037
* daemon started successfully
Execution time limit exceeded!
I got the same error. Can someone provide a solution?
 

rainbow

New member
I got the following feedback:

WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
* daemon not running; starting now at tcp:5037
* daemon started successfully
Execution time limit exceeded!
 

rainbow

New member
All questions related to Learned Smartphone ISP Challenge can be asked in this thread.

I got the following feedback:

WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
* daemon not running; starting now at tcp:5037
* daemon started successfully
Execution time limit exceeded!
 

Zevin

New member
Hey, when will the target evaluation platform be open ?

Cause there is a discription about the platform in the website: "Run it remotely on the target evaluation platform (the instructions will be announced soon)."
1658645893986.png
 

gosha20777

New member
@Andrey Ignatov I am trying to create a submission
I have dot the data from validation set (MAI2021_LearnedISP_valid_raw.zip) preprocessed it using my network and put it into archive.
And I've got error
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap. Traceback (most recent call last): File "/tmp/codalab/tmpC977Vs/run/program/evaluation.py", line 95, in raise Exception('Expected %d .png images'%len(ref_pngs)) Exception: Expected 3145 .png images

What is 3145 images? There is only 2258 images in the Validation Set. https://data.vision.ee.ethz.ch/ihnatova/MAI2021_LearnedISP_valid_raw.zip
 

gosha20777

New member
@Andrey Ignatov What input shape should be in model.tflite in final submit?

If I export it using input shape [1, None, None 4] in my tf (keras) model shape it converts to
'shape': array([1, 1, 1, 4], dtype=int32), 'shape_signature': array([ 1, -1, -1, 4]

Is it right?
I am using the latest TensorFlow 2.9 and the following conversion piepline:

# convention piepline model = build_inference_model(shape=(None, None, 4), batch_size=1) # get keras model with input shape (1, None, None, 4) converter = tf.lite.TFLiteConverter.from_keras_model(model) tflite_model = converter.convert() open(save_path, "wb").write(tflite_model) ... # inference piepline interpreter = tf.lite.Interpreter( model_path=args.model_file, ) interpreter.allocate_tensors() input_details = interpreter.get_input_details() print(input_details) # get [{'name': 'serving_default_input_1:0', 'index': 0, 'shape': array([ 1, 1, 1, 4], dtype=int32), 'shape_signature': array([ 1, -1, -1, 4], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}]
 
Last edited:

gosha20777

New member
@Andrey Ignatov I am trying to create a submission
I have dot the data from validation set (MAI2021_LearnedISP_valid_raw.zip) preprocessed it using my network and put it into archive.
And I've got error
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap. Traceback (most recent call last): File "/tmp/codalab/tmpC977Vs/run/program/evaluation.py", line 95, in raise Exception('Expected %d .png images'%len(ref_pngs)) Exception: Expected 3145 .png images

What is 3145 images? There is only 2258 images in the Validation Set. https://data.vision.ee.ethz.ch/ihnatova/MAI2021_LearnedISP_valid_raw.zip
Ahahah, I have not noticed the following fact:
At the testing phase, the leaderboard will not show a specific score. Since we will download all the participants’ submissions and evaluate them on our devices, please ignore the “Failed Status” on CodaLab.
 

Jaehoon Jung

New member
  • validation data: the participants will receive the RAW images when the validation phase starts to get online feedback from the validation server; the corresponding ground truth RGB images will be released when the final phase of the challenge starts.
When will the ground truth RGB images of validation data be available?
 

Andrey Ignatov

Administrator
Staff member
When will the ground truth RGB images of validation data be available?

Please check the email that was sent to all challenge participants this Tuesday. It contains a detailed description of the final challenge submission format and data. It is very important that you follow the provided submission guideline precisely so that your submission is not disqualified.

Ahahah, I have not noticed the following fact:
(y)

'shape': array([1, 1, 1, 4], dtype=int32), 'shape_signature': array([ 1, -1, -1, 4]
Yes, that's correct, None dimensions are replaced with -1.
 

Jaszheng

New member
  • 02.08.2022 Preliminary test results release to the participants
  • 10.08.2022 Paper submission deadline for entries from the challenge (AIM workshop @ ECCV 2022)
1. When will the preliminary test results be released?
2. Is the deadline for paper submission accurate? Are there any detailed instructions about paper submission?
 

Msss

New member
Submission ID : 162072
Username : MinsuKwon
It looks like my tflite model for track 2 was reported in the result table.
Please find the email for more details about the issue!
 

gosha20777

New member
Hi @Andrey Ignatov
More than one week ago I got the email. The letter said that in a week I would receive a draft problem report and I would be asked to review the description of my solution. But I haven't got it yet. Is it ok?
 
Top