I would like to highlight an interesting research contribution by Prof. Mourad Zaied, introducing SaluNet, a framework that explores an alternative perspective on normalization in modern deep learning architectures.
The central idea is particularly interesting: instead of relying on batch-wise or activation statistics, can learnable transformations provide an effective alternative to conventional normalization layers?
The proposed framework introduces three key components:
-
SALU: A learnable elementwise transformation designed to replace conventional normalization layers.
-
SWALU and GALU: Learnable activation alternatives inspired by Swish and GELU.
-
SaluNet-C and SaluNet-T: Extensions of this approach to Convolutional Neural Networks and Vision Transformers.
The reported experiments on CIFAR-100 and ImageNet-1K explore the framework’s performance under different batch-size configurations.
What I find particularly interesting is the broader research direction: rethinking fundamental architectural choices in deep learning rather than simply modifying existing components.
Resources:
Paper: LinkedIn
Code, configurations and pretrained models: LinkedIn
Original announcement:
I would be interested in hearing the community’s thoughts:
Could learnable geometric transformations offer a promising direction for reducing the dependence of deep learning architectures on conventional normalization techniques?
#DeepLearning #ComputerVision 
Transformers #CNN Research