Support for FasterViT

[awsaf49]

This model is recently released by Nvidia which is both accurate and fast comparing sota models like ConvNeXt, MaxViT, and SwinV2. I think it would be great to have this model here.

Paper: https://arxiv.org/abs/2306.06189
Code: https://github.com/NVlabs/FasterViT

[leondgarse]

Nice, they have it released. Just took a quick skim, seems a bit complicated...

[awsaf49]

Also, one interesting aspect is that they're incorporating a significant number of Conv blocks, which deviates from the typical ViT series models. I haven't had the chance to thoroughly examine the paper yet, but I'm genuinely excited to find the secrets behind its good speed and accuracy.

[leondgarse]

Almost there, just uploading weights and some document works, keras_cv_attention_models/fastervit.

[awsaf49]

Thanks. Love to see your T4 Inference benchmark on this model......

[leondgarse]

The inference speed should be easy, but I met some problem. That the results vary so much from ~590 qps - ~710 qps, depending on if using 2D / 3D / 4D input data for onnx Linear. Still need some time.

[leondgarse]

Results are updated with almost the best qps I can achieve.

Config	1st	2nd
torch original	615.078 qps	635.427 qps
kecam original	482.952 qps	490.919 qps
+ attention 3D inputs	623.892 qps	620.316 qps
+ hat 3D inputs	663.72 qps	667.523 qps
+ MLP 2D inputs	662.036 qps	653.943 qps
+ qkv reshape together	663.134 qps	655.864 qps

from keras_cv_attention_models import plot_func
plot_series = ["efficientnetv2", 'fastervit', 'DINOv2', 'maxvit', 'SwinTransformerV2', 'convnextv2', 'flexivit', 'hiera']
plot_func.plot_model_summary(plot_series, x_label='inference_qps', model_table="model_summary.csv", allow_extras=['mae_in1k_ft1k'])