Channel/Spatial/Element wise Attention Modules

[innat]

In the timm package, it provides some soft attention modules to building network blocks and I think it's a good fit here, for example:

• Squeeze-and-Excitation (SE): https://arxiv.org/abs/1709.01507 Cited by 10411 | Issue
• CBAM: https://arxiv.org/abs/1807.06521 Cited by 3530
• Residual Attention https://arxiv.org/abs/1704.06904 Cited by 2177
• Dual Attention https://arxiv.org/abs/1809.02983 Cited by 1937
• Selective Kernel (SK) https://arxiv.org/abs/1903.06586 Cited by 667
• GCNet https://arxiv.org/abs/1904.11492 Cited by 598
• SAN https://ieeexplore.ieee.org/document/8954252 Cited by 566
• ECA-Net https://arxiv.org/abs/1910.03151 Cited by 471
• BAM https://arxiv.org/abs/1807.06514 Cited by 365
• A2-Nets https://arxiv.org/abs/1810.11579 Cited by 278
• Bottleneck Transformer https://arxiv.org/abs/2101.11605 Cited by 147
• Lambda Layer: https://arxiv.org/abs/2102.08602 Cited by 67
• SGENet https://arxiv.org/abs/1905.09646 Cited by 51
• HaloNet https://arxiv.org/abs/2103.12731 Cited by 51
• CoAtNet https://arxiv.org/abs/2106.04803 Cited by 35 ( < 50 current)

and many others.

[bhack]

I have not extensively checked about these in the model gardens components, but just picking the First One in the list:

https://github.com/tensorflow/models/blob/master/official/vision/beta/projects/yolo/modeling/layers/nn_blocks.py#L1170

[LukeWood]

Currently we are prioritizing components that are required to achieve state of the art results on specific tasks. I.e., imagenet1k classification, COCO object detection, etc. Any chance I could get some guidance as to where these components excel?

Thanks

[innat]

@LukeWood
The above list isn't meant to be the replacement for the current priority list. The point is whether these cv components are fit in keras-cv or not.

KerasCV is a repository of modular building blocks (layers, ... .

It can be considered as a to-do list, an interested contributor can get references from here in the coming days. Let me know if I miss something. Also, I think a few components should be already on the current priority list, for example, Squeeze-and-Excitation or CBAM.

[bhack]

II think that in this specific field you need to always find a balance between popularity, state of the art, sedimentation and the available human and computing resources you have at a specific point on the time dimension in your community.
So you need to define a policy with an equilibrium over all these tensions especially if you want try to growth your code contributor community and not just your user base.

E.g. we are still investing resources for resnext-rs and waiting for the citation/popularity threshold for the next STOA:
https://arxiv.org/abs/2201.03545

Often an high popularity component/model is needed to be maintained just cause it is a recurent baseline for new academic work.

Another addition dimension/tension Is the computing resources required by a model and its own components.

[old-school-kid]

@innat @LukeWood
I have a two part question about this?

1. Are we adding modules like Spatial Attention Module and Channel Attention Module or should we be adding the whole model (example CBAM as whole) as an application with parameters to change configuration of the model?
2. As @bhack pointed out we cannot solely depend on state of the art and we have to take into account the popularity and its "promosing"prospects. Soif you could throw some light on that area too.

Could you post these separately so people could be assigned if interested and maybe some of the model could be prioritized.

[old-school-kid]

Ah, also can SWIN transformer be added to the list?

[innat]

@old-school-kid
about (1). I would prefer these as modules.
about (2). I agree with those points. But it's not clear to me how to define promising things here. And what role tfa.layers can serve here.

For Swin-Transformer, it's been asked, #15545

[innat]

@leondgarse
I've found that you maintained an amazing repo, HERE. It would be great if you consider some contribution. Thanks.

[bhack]

leondgarse/keras_cv_attention_models#45 😉

[leondgarse]

@innat Yes, I would like to contribute, and we have a talk here Keras-cv #45. Speaking model architecture modules, in my practice, components like halo_attention and bottleneck_transformer usually need to be rewritten for other new models, as most of them have some detail different. Personally, I'm using most of them as functions, not layers, so prefer port with the whole model. I also support we would better only hold most common models here, and yes, I would like to port them here if decided. :)

[innat]

@leondgarse yes, that would be great. I think currently, the keras team is focusing more on preprocessing and metric layer on its first releases. By this time, you can start working on modules/layers.

Also, note that most probably soon, all the keras.application would be live in keras-cv, SOURCE. A discussion was made, HERE; regarding the possible architecture for keras-cv. I also shared some of the architecture that I believe should be available from the official API, listed HERE.

[leondgarse]

I think we at least should get ResNet available first, as it's mostly used as a baseline, and set a baseline for how models should be organized:

• Shall we consider channel_first format? If so, I would recommend some genarel functions like my currently using batchnorm / lay_norm / conv2d.
• How the detection / segmentation header be added to a baseline recognition model? We may need a general method. I'm currently using a function get_pyramide_feture_layers extracting feature layers by layer name, then add a detection header.
• Where some general modules should go?
• How the preprocess should be added to model? My method is set a preprocess_input function for each model like preprocess_input.
• Set other standad like general model parameters input_shape / num_classes / weights, also a standard for layer names.

[innat]

@LukeWood mentioning to put some feedback here.

[innat]

@old-school-kid In the first post, I've mentioned many attention module with citation count. You can pick one with highest citation layer.

• https://github.com/xmu-xiaoma666/External-Attention-pytorch
• https://github.com/ccfco-Ivan/External-Attention-tensorflow

Also @leondgarse is maintaining amazing library

• https://github.com/leondgarse/keras_cv_attention_models