Mobile AI Workshop Technical and Model Conversion Questions

[Andrey Ignatov]

Any technical questions and questions related to model conversion to TFLite format can be asked in this thread.

 

[lukas]

Dear organizers, I'm trying to convert my PyTorch model into a dynamic-input TFLite model, but the conversion fails when I set the input shape to be dynamic. Interestingly, the conversion works fine with a static input shape, and I can use the resulting TFLite model for inference speed evaluation.

Have you encountered similar issues before, or is there any recommended approach or workaround for this? Also, would it be acceptable to use a static-input TFLite model for quality evaluation instead? Thanks in advance!

 

[Andrey Ignatov]

Yes, there is indeed an issue with dynamic input size for PyTorch to TFLite conversion.

Also, would it be acceptable to use a static-input TFLite model for quality evaluation instead? Thanks in advance!

We will accept this for models converted from PyTorch. However, be prepared that we might ask you to provide the same model with additional input sizes if automatic tensor resizing won't work for your model.

 

[Alex_Fall]

MAI-2025-Workshop/pytorch_to_tflite.py at main · aiff22/MAI-2025-Workshop

Contribute to aiff22/MAI-2025-Workshop development by creating an account on GitHub.

github.com

I used the method on this link to convert a pytorch model to a TFLite model with fixed input dimensions. There were no major problems with the conversion. However, deploying the resulting TFLite model to AI Benchmark running in FP16+GPU Delegate mode gave the error in the attached image. This appears to be that GPU Delegate does not support BATCH MATMUL, and as it says, I used netron to visualize my model as shown in the attached image. The success of converting Pytorch to TFLite suggests that ai-edge-torch is able to support batch matrix multiplication operators, however the problem occurs when deploying to a cell phone and I'm not sure what to do about it.

 

[Andrey Ignatov]

running in FP16+GPU Delegate mode

Does this error also occur during CPU-based inference?

GPU Delegate does not support BATCH MATMUL

One possible issue might be the dimensionality of your tensor: GPU delegate might be expecting a 4D tensor, but it received a 3D one.

I'm not sure what to do about it.

If you are still unable to solve this problem, you can send us your model by email.

 

[Alex_Fall]

Does this error also occur during CPU-based inference?


One possible issue might be the dimensionality of your tensor: GPU delegate might be expecting a 4D tensor, but it received a 3D one.


If you are still unable to solve this problem, you can send us your model by email.

Thank you for your reply. The input size of the TFLite model I got from my conversion is fixed for the sRGB Enhancement Challenge [1,3,1024,1024]. I tried inference using CPU+FP16 and it worked even though it has slower inference time. However, I noticed that the Learn the Details/Evaluation of the sRGB Enhancement Challenge has a relevant note about the choice of mode: Testing Your Model on the Target Adreno / Mali Mobile GPUs section requires that I need to choose FP16+TFLite GPU Delegate when using the AI Benchmark. So this seems to indicate that this mode doesn't support enough arithmetic operations. If the target platform must use this mode, how do I address this technical issue to get higher inference speed? Or, if the target platform allows FP16+CPU mode, will all the teams' models be compared uniformly in this mode?

 

[Andrey Ignatov]

Or, if the target platform allows FP16+CPU mode

By default, all solutions will be evaluated using FP16 + TFLite GPU Delegate mode. However, CPU backend will be used for solutions not supporting this option.

So this seems to indicate that this mode doesn't support enough arithmetic operations.

TFLite GPU delegate is the easiest mode in terms of model adaptation as it supports nearly all common ops. However, yes, there might be some ops or layers requiring small adaptations. You've already been provided with a hint as for why BATCH MATMUL might be failing in your case.

And, in principle, the general adaptation scenario looks as follows:

1. Identifying the op that is failing.
2. Removing this op from the model, converting it again, and checking that the issue is gone.
3. Trying one of the following options:
- Changing layer parameters, some non-default options are sometimes not implemented by the delegates
- Using or making an alternative PyTorch / TF layer implementation comprised of supported ops
- Replacing the layer with some other similar one

 

[Alex_Fall]

By default, all solutions will be evaluated using FP16 + TFLite GPU Delegate mode. However, CPU backend will be used for solutions not supporting this option.


TFLite GPU delegate is the easiest mode in terms of model adaptation as it supports nearly all common ops. However, yes, there might be some ops or layers requiring small adaptations. You've already been provided with a hint as for why BATCH MATMUL might be failing in your case.

And, in principle, the general adaptation scenario looks as follows:

1. Identifying the op that is failing.
2. Removing this op from the model, converting it again, and checking that the issue is gone.
3. Trying one of the following options:
- Changing layer parameters, some non-default options are sometimes not implemented by the delegates
- Using or making an alternative PyTorch / TF layer implementation comprised of supported ops
- Replacing the layer with some other similar one

Thank you very much for your reply, I will consider your approach!

 

[cblks]

Dear organizers, I have a question: after converting my PyTorch model to TFLite, the format did not automatically switch from NCHW to NHWC. I'm stuck at this step — do you have any suggestions on how to handle this?

 

[Andrey Ignatov]

the format did not automatically switch from NCHW to NHWC

Are you using the ai_edge_torch plugin for model conversion like in this tutorial?

 

[cblks]

Are you using the ai_edge_torch plugin for model conversion like in this tutorial?

Dear organizers,
When testing our converted model on mobile, we encountered the error shown in the image. We're not sure what this error means. While checking the model, we suspect it might be related to the input channels. Could you please help us understand this error message?Thanks in advance for your support!

 

[Andrey Ignatov]

While checking the model, we suspect it might be related to the input channels.

It's not an issue with the input channels - it's a problem with a bilinear resize layer. The issue is self-explanatory: half_pixel_center and align_corner options cannot be used at the same time. Try changing the parameters of the resize layer in your model.

 

[cblks]

It's not an issue with the input channels - it's a problem with a bilinear resize layer. The issue is self-explanatory: half_pixel_center and align_corner options cannot be used at the same time. Try changing the parameters of the resize layer in your model.

Thank you very much for your guidance！

 

[cblks]

Yes, there is indeed an issue with dynamic input size for PyTorch to TFLite conversion.


We will accept this for models converted from PyTorch. However, be prepared that we might ask you to provide the same model with additional input sizes if automatic tensor resizing won't work for your model.

Dear organizers,

It seems that TFLite models may not officially support dynamic input shapes.
Would it be possible for you to specify an additional fixed input size that we can use to evaluate the model quality?

Alternatively, we could implement this in the inference script by resizing the input images to a fixed size before passing them into the model, and then stretching the outputs back to the original dimensions after inference.
However, this approach may lead to a certain degree of accuracy loss.

Thank you for your understanding and support.

Best regards,

 

[Andrey Ignatov]

Could you please confirm whether my understanding is correct?
Because even after adding that line, my exported model still has a fixed input size.

The correct one is:

dim_2 = 8 * torch.export.Dim("height", min=8, max=256)
dim_3 = 8 * torch.export.Dim("width", min=8, max=256)

edge_model = ai_edge_torch.convert(model.eval(), sample_input, dynamic_shapes=({2:dim_2, 3:dim_3},))

However, this option is broken in the latest PyTorch releases. Therefore, we've allowed submission of TFLite models with a static size when they are converted from PyTorch.

 

[cblks]

The correct one is:

dim_2 = 8 * torch.export.Dim("height", min=8, max=256)
dim_3 = 8 * torch.export.Dim("width", min=8, max=256)

edge_model = ai_edge_torch.convert(model.eval(), sample_input, dynamic_shapes=({2:dim_2, 3:dim_3},))

However, this option is broken in the latest PyTorch releases. Therefore, we've allowed submission of TFLite models with a static size when they are converted from PyTorch.

Thank you very much for your guidance！