Learned Smartphone ISP Challenge

A

AIISP

New member
Hi @Andrey Ignatov

The conversion gives this kind of error with TF1.15, even I set the experimental_new_converter=True.
"ValueError: None is only supported in the 1st dimension. Tensor 'Placeholder' has invalid shape '[1, None, None, 4]'."

It could be work with TF2.3, but the tflite with [1,1,14] input shape
1616038244022.png

And when i try to run it, i got the error message:
""RuntimeError: tensorflow/lite/kernels/kernel_util.cc:404 d1 == d2 || d1 == 1 || d2 == 1 was not true.Node number 11 (MUL) failed to prepare."""

Did you have any advice for me?

By the way, the "Evaluation" section on CodaLab
1616060887111.png
Does it mean we don't need to generate the model_none.tflite ??
 
Last edited:
Andrey Ignatov

Andrey Ignatov

Administrator
Staff member
IMGClear said:
Does the submitted tflite should be produce the INT8 value for fidelity scores computeing?
Click to expand...

If you decide to submit quantized model - it should consists from INT8 ops only, thus its outputs will also be INT8.

IMGClear said:
The round processing( [0~1]=>[0~255] ) will also be included in the runtime measurement.
Click to expand...

This processing should be integrated in your model, and we will be checking its runtime - therefore the answer is yes.
 
A

AIISP

New member
Andrey Ignatov said:
Try to convert the model with TF 2.4 or TF-nightly.



No, you need to generate this file, the instructions sent by email are the main ones.
Click to expand...
Thanks~ @Andrey Ignatov
It could be work only with TF-nightly, but I have some question for you.

1. Do we need to use the same tf version to generate the "model.tflite" and "model_none.tflite" ?
(The requirement of MTK Dimensity 1000+ APU is Tensorflow 1.15.0)
2. Do we need to submit the inference_tflite.py file ??
1616082401368.png
 
I

IMGClear

New member
Andrey Ignatov said:
This processing should be integrated in your model, and we will be checking its runtime - therefore the answer is yes.
Click to expand...
So, the output range of FP model must be 0~255 and the data type could be float32 ??

Which one is your requirement ?
1. tensor * 255.0
2. floor( tensor*255.0 + 0.5)
 
Andrey Ignatov

Andrey Ignatov

Administrator
Staff member
AIISP said:
Do we need to use the same tf version to generate the "model.tflite" and "model_none.tflite" ?
Click to expand...

No.

AIISP said:
Do we need to submit the inference_tflite.py file ??
Click to expand...

This requirement was recently added by MediaTek, and its main goal is to introduce an additional sanity check for the submitted solution. Note, however, that this script will not be used by us to evaluate your solution: we will be taking only the corresponding TFLite model and running it with the standard script that compares its outputs with the ground truth and calculates the final fidelity scores.

So, the answer is - yes, better add this file to the final submission, but if it won't be there - your solution will not be disqualified.

IMGClear said:
So, the output range of FP model must be 0~255 and the data type could be float32 ??
Click to expand...

Yes.

IMGClear said:
Which one is your requirement ?
1. tensor * 255.0
2. floor( tensor*255.0 + 0.5)
Click to expand...

It's only up to you to decide on this - we will be just applying your TFLite model to test images and computing the fidelity scores based on its outputs and the ground truth images.
 
Andrey Ignatov

Andrey Ignatov

Administrator
Staff member
Idlefish said:
Does it mean we should prepare 3 model, namely model.tflite, model_none.tflite and one tflite whose input shape is [1,128,128,4] and output shape is [1, 256, 256, 3]?
Click to expand...

You need to submit three models as was stated in the email. Sorry for confusion, MHChen will modify his answer.
 
J

jaeyoon

New member
Does not the evaluation server accept global pooling?

When I submit the model with global pooling with tf.expand_dims(because there is no argument such as keepdims in tf.keras.layers.GlobalAveragePooling2D) it says it returns segmentation fault

WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
* daemon not running; starting now at tcp:5037
* daemon started successfully
timeout: invalid time interval ‘/t’
Try 'timeout --help' for more information.
WARNING: linker: Warning: "/data/local/tmp/MAI_21/benchmark_model" unused DT entry: DT_RPATH (type 0xf arg 0x951) (ignoring)
INFO: Initialized TensorFlow Lite runtime.
Segmentation fault (core dumped)
Traceback (most recent call last):
File "/tmp/codalab/tmpHlpDe1/run/program/evaluation.py", line 178, in
file = open(LOG_NAME, 'r')
IOError: [Errno 2] No such file or directory: '/tmp/codalab/tmpHlpDe1/run/output/output.csv'

Global pooling is replaced by mean operation (I looked over my model using Netron) and NNAPI seems support mean op and expand_dims

How can I use global pooling?
 
Last edited:
J

jaeyoon

New member
Andrey Ignatov said:
Just replace it with a normal pooling op and set the pool size accordingly.
Click to expand...
Actually I'm going to try it

But, when it comes to the model for variable size, I cannot assign the pool size in advance. That's why I considered tf.keras.layers.GlobalAveragePooling2D
According to the email for final submission, I should submit two tflite model, one for static input size to evaluate the runtime, the other for variable size only for fidelity

Will it be okay if I use different layers for two tflite model (of course, they make the same result)? One (for latency) will work on mobile and the other (for fidelity) won't
 
I

IMGClear

New member
Hi @Andrey Ignatov,

In quantization case, we base on the same ckpt to generate the "model.tflite" and "model_none.tflite".
But we use the "noise data" and "mediatek_raw" as representative_dataset for "model.tflite" and "model_none.tflite", respectively.
Is such a result acceptable?

thanks
 
Andrey Ignatov

Andrey Ignatov

Administrator
Staff member
IMGClear said:
But we use the "noise data" and "mediatek_raw" as representative_dataset for "model.tflite" and "model_none.tflite", respectively.
Is such a result acceptable?
Click to expand...

No, the same data should be used in both cases.

Note that post-training quantization will likely lead to a 5-7dB loss in PSNR, therefore think twice if you want to submit a quantized model.
 
M

Msss

New member
@Andrey Ignatov
Hi, we are Enerzai research team and has a question.
One of our solution used channel-wise attention with global average pooling and found that it induces a huge increase in latency. And that's why our one solution has very large latency. (We checked that removing the channel-wise attention layer (global average pooling layer) resulted in small latency)
With respect to the model footprint and architecture, it should not make such increase. We thus are sure that our implementation of global pooling - actually channel-wise attention- was not supported by NPU and executed in CPU.
But, we found that AIISP team used Global Average Pooling layer for channel wise attention with small latency.


We experienced that the latency depends ca.PNGon which tensorflow operation layer we use even though the operation result is same (like using tf.keras.layers.Conv2d or tf.compat.v1.nn.conv2d). And yet we multiplied the 1x1x4 attention result into HxWx4 input of the attention module. So we don't care the prize result but can you tell me how we can implement the global pooling layer (channel-wise attention) so that the latency is small?

Our implementation is using tf.keras.layers.AveragePooling2D((544,960)) and just multiply
Now, I think multiplying with broadcasting may be one of the cause of the large latency

And can you run the code after you let me know how to implement the Channel-wise attention with GAP to be supported by NPU and we resubmit the code and then can you report that result on the final report? (We don't mean to change the prize result of course) We spent a lot of time on the challenge
 
Last edited:
Andrey Ignatov

Andrey Ignatov

Administrator
Staff member
Msss said:
can you tell me how we can implement the global pooling layer (channel-wise attention) so that the latency is small?
Click to expand...

Right now, one possible way to implement this op is to use the standard TF's max / average pooling layer and set the pooling size to be equal to the size of the corresponding feature map.

Msss said:
we resubmit the code and then can you report that result on the final report?
Click to expand...

It is unlikely that we will do this - all challenge participants were provided with an ability to test their models on the target validation platform during two months. It is not hard to notice the above issue, and solving it was also a part of the competition. If we enter this way, then we will also have to re-evaluate many other models as well.

Note, however, that it is very likely that the next challenge on this topic will start in just a few months, so you will be able to test your upgraded solution in the following competition :)
 
You must log in or register to reply here.
Top