Encyclopedia

Smooth maximum

Mathematical approximation

Wikipedia contributorsAdapted for Fathomly · View authors

2 min read
Source updated October 21, 2025

In mathematics, a smooth maximum of an indexed family x₁, ..., x_n of numbers is a smooth approximation to the maximum function $\max(x_{1},\ldots ,x_{n}),$ meaning a parametric family of functions $m_{\alpha }(x_{1},\ldots ,x_{n})$ such that for every ⁠ $\alpha$ ⁠, the function ⁠ $m_{\alpha }$ ⁠ is smooth, and the family converges to the maximum function ⁠ $m_{\alpha }\to \max$ ⁠ as ⁠ $\alpha \to \infty$ ⁠. The concept of smooth minimum is similarly defined. In many cases, a single family approximates both: maximum as the parameter goes to positive infinity, minimum as the parameter goes to negative infinity; in symbols, ⁠ $m_{\alpha }\to \max$ ⁠ as ⁠ $\alpha \to \infty$ ⁠ and ⁠ $m_{\alpha }\to \min$ ⁠ as ⁠ $\alpha \to -\infty$ ⁠. The term can also be used loosely for a specific smooth function that behaves similarly to a maximum, without necessarily being part of a parametrized family.

01Examples

Boltzmann operator

For large positive values of the parameter $\alpha >0$ , the following formulation is a smooth, differentiable approximation of the maximum function. For negative values of the parameter that are large in absolute value, it approximates the minimum.

${\mathcal {S}}_{\alpha }(x_{1},\ldots ,x_{n})={\frac {\sum _{i=1}^{n}x_{i}e^{\alpha x_{i}}}{\sum _{i=1}^{n}e^{\alpha x_{i}}}}$

${\mathcal {S}}_{\alpha }$ has the following properties:

${\mathcal {S}}_{\alpha }\to \max$ as $\alpha \to \infty$
${\mathcal {S}}_{0}$ is the arithmetic mean of its inputs
${\mathcal {S}}_{\alpha }\to \min$ as $\alpha \to -\infty$

The gradient of ${\mathcal {S}}_{\alpha }$ is closely related to softmax and is given by

$\nabla _{x_{i}}{\mathcal {S}}_{\alpha }(x_{1},\ldots ,x_{n})={\frac {e^{\alpha x_{i}}}{\sum _{j=1}^{n}e^{\alpha x_{j}}}}[1+\alpha (x_{i}-{\mathcal {S}}_{\alpha }(x_{1},\ldots ,x_{n}))].$

This makes the softmax function useful for optimization techniques that use gradient descent.

This operator is sometimes called the Boltzmann operator, after the Boltzmann distribution.

LogSumExp

Another smooth maximum is LogSumExp:

$\mathrm {LSE} _{\alpha }(x_{1},\ldots ,x_{n})={\frac {1}{\alpha }}\log \sum _{i=1}^{n}\exp \alpha x_{i}$

This can also be normalized if the $x_{i}$ are all non-negative, yielding a function with domain $[0,\infty )^{n}$ and range $[0,\infty )$ :

$g(x_{1},\ldots ,x_{n})=\log \left(\sum _{i=1}^{n}\exp x_{i}-(n-1)\right)$

The $(n-1)$ term corrects for the fact that $\exp(0)=1$ by canceling out all but one zero exponential, and $\log 1=0$ if all $x_{i}$ are zero.

Mellowmax

The mellowmax operator is defined as follows:

$\mathrm {mm} _{\alpha }(x)={\frac {1}{\alpha }}\log {\frac {1}{n}}\sum _{i=1}^{n}\exp \alpha x_{i}$

It is a non-expansive operator. As $\alpha \to \infty$ , it acts like a maximum. As $\alpha \to 0$ , it acts like an arithmetic mean. As $\alpha \to -\infty$ , it acts like a minimum. This operator can be viewed as a particular instantiation of the quasi-arithmetic mean. It can also be derived from information theoretical principles as a way of regularizing policies with a cost function defined by KL divergence. The operator has previously been utilized in other areas, such as power engineering.

Connection between LogSumExp and Mellowmax

LogSumExp and Mellowmax are the same function differing by a constant ${\frac {\log {n}}{\alpha }}$ . LogSumExp is always larger than the true max, differing at most from the true max by ${\frac {\log {n}}{\alpha }}$ in the case where all n arguments are equal and being exactly equal to the true max when all but one argument is $-\infty$ . Similarly, Mellowmax is always less than the true max, differing at most from the true max by ${\frac {\log {n}}{\alpha }}$ in the case where all but one argument is $-\infty$ and being exactly equal to the true max when all n arguments are equal.

p-Norm

Another smooth maximum is the p-norm:

$\|(x_{1},\ldots ,x_{n})\|_{p}=\left(\sum _{i=1}^{n}|x_{i}|^{p}\right)^{\frac {1}{p}}$

which converges to $\|(x_{1},\ldots ,x_{n})\|_{\infty }=\max _{1\leq i\leq n}|x_{i}|$ as $p\to \infty$ .

An advantage of the p-norm is that it is a norm. As such it is scale invariant (homogeneous): $\|(\lambda x_{1},\ldots ,\lambda x_{n})\|_{p}=|\lambda |\cdot \|(x_{1},\ldots ,x_{n})\|_{p}$ , and it satisfies the triangle inequality.

Smooth maximum unit

The following binary operator is called the Smooth Maximum Unit (SMU):

${\begin{aligned}\textstyle \max _{\varepsilon }(a,b)&={\frac {a+b+|a-b|_{\varepsilon }}{2}}\\&={\frac {a+b+{\sqrt {(a-b)^{2}+\varepsilon }}}{2}}\end{aligned}}$

where $\varepsilon \geq 0$ is a parameter. As $\varepsilon \to 0$ , $|\cdot |_{\varepsilon }\to |\cdot |$ and thus $\textstyle \max _{\varepsilon }\to \max$ .

Smoothmax of (−x, x) versus x for various parameter values. Very smooth for =0.5, and more sharp for =8.

Watch videos about Smooth maximumExplainers and documentaries on YouTube (opens in a new tab)

Sources and credits

This article is adapted from the Wikipedia article “Smooth maximum”, written by its contributors and licensed under CC BY-SA 4.0. Fathomly has changed the layout, removed citation markers, navigation and maintenance notices, and adjusted punctuation. This adapted version is shared under the same license. For references, see the original article.

Images, from Wikimedia Commons:

Smoothmax.png by 1&only, CC BY-SA 4.0

Fathomly is not affiliated with or endorsed by the Wikimedia Foundation. Spotted a problem? Tell us.

Smooth maximum

01Examples

Boltzmann operator

LogSumExp

Mellowmax

Connection between LogSumExp and Mellowmax

p-Norm

Smooth maximum unit

Sources and credits

Related topics

LogSumExp

Softmax function

Generalized mean