This is an automated archive made by the Lemmit Bot.

The original was posted on /r/machinelearning by /u/Sad-Razzmatazz-5188 on 2025-01-18 10:05:15+00:00.

This is a half joke, and the core concepts are quite easy, but I’m sure the community will cite lots of evidence to both support and dismiss the claim that softmax sucks, and actually make it into a serious and interesting discussion.

What is softmax? It’s the operation of applying an element-wise exponential function, and normalizing by the sum of activations. What does it do intuitively? One point is that outputs sum to 1. Another is that the the relatively larger outputs become more relatively larger wrt the smaller ones: big and small activations are teared apart.

One problem is you never get zero outputs if inputs are finite (e.g. without masking you can’t attribute 0 attention to some elements). The one that makes me go crazy is that for most of applications, magnitudes and ratios of magnitudes are meaningful, but in softmax they are not: softmax cares for differences. Take softmax([0.1, 0.9]) and softmax([1,9]), or softmax([1000.1,1000.9]). Which do you think are equal? In what applications that is the more natural way to go?

Numerical instabilities, strange gradients, embedding norms are all things affected by such simple cores. Of course in the meantime softmax is one of the workhorses of deep learning, it does quite a job.

Is someone else such a hater? Is someone keen to redeem softmax in my eyes?