Reference articles on history, science, culture and more
Surprise meSurprise meSubscribe
Encyclopedia

Proximal gradient method

Form of projection

Wikipedia contributorsAdapted for Fathomly · View authors
1 min read
Source updated June 21, 2025
Image credit is listed at the end of this article.

Proximal gradient methods are a generalized form of projection used to solve non-differentiable convex optimization problems.

Many interesting problems can be formulated as convex optimization problems of the form

\min _{\mathbf {x} \in \mathbb {R} ^{d}}\sum _{i=1}^{n}f_{i}(\mathbf {x} )

where f_{i}:\mathbb {R} ^{d}\rightarrow \mathbb {R} ,\ i=1,\dots ,n are possibly non-differentiable convex functions. The lack of differentiability rules out conventional smooth optimization techniques like the steepest descent method and the conjugate gradient method, but proximal gradient methods can be used instead.

Proximal gradient methods starts by a splitting step, in which the functions f_{1},...,f_{n} are used individually so as to yield an easily implementable algorithm. They are called proximal because each non-differentiable function among f_{1},...,f_{n} is involved via its proximity operator. Iterative shrinkage thresholding algorithm, projected Landweber, projected gradient, alternating projections, alternating-direction method of multipliers, alternating split Bregman are special instances of proximal algorithms.

For the theory of proximal gradient methods from the perspective of and with applications to statistical learning theory, see proximal gradient methods for learning.

01Projection onto convex sets (POCS)

One of the widely used convex optimization algorithms is projections onto convex sets (POCS). This algorithm is employed to recover/synthesize a signal satisfying simultaneously several convex constraints. Let f_{i} be the indicator function of non-empty closed convex set C_{i} modeling a constraint. This reduces to convex feasibility problem, which require us to find a solution such that it lies in the intersection of all convex sets C_{i}. In POCS method each set C_{i} is incorporated by its projection operator P_{C_{i}}. So in each iteration x is updated as

x_{k+1}=P_{C_{1}}P_{C_{2}}\cdots P_{C_{n}}x_{k}

However beyond such problems projection operators are not appropriate and more general operators are required to tackle them. Among the various generalizations of the notion of a convex projection operator that exist, proximal operators are best suited for other purposes.

02Examples

Special instances of Proximal Gradient Methods are

Watch videos about Proximal gradient methodExplainers and documentaries on YouTube (opens in a new tab)

Sources and credits

This article is adapted from the Wikipedia article “Proximal gradient method”, written by its contributors and licensed under CC BY-SA 4.0. Fathomly has changed the layout, removed citation markers, navigation and maintenance notices, and adjusted punctuation. This adapted version is shared under the same license. For references, see the original article.

Images, from Wikimedia Commons:

Fathomly is not affiliated with or endorsed by the Wikimedia Foundation. Spotted a problem? Tell us.

Continue exploring

Related topics

Proximal operator

In mathematical optimization, the proximal operator is an operator associated with a proper, lower semi-continuous convex function f {\displaystyle f} from a Hilbert space X {\displaystyle {\mathcal {X}}} to [ − ∞ , + ∞ ] {\displaystyle [-\infty ,+\infty ]} , and is defined by: prox f ⁡ = arg ⁡ min x ∈ X ( f ( x ) + 1 2 ‖ x − v ‖ X 2 ) . {\displaystyle \operatorname {prox} _{f}(v)=\arg \min _{x\in {\mathcal {X}}}\left(f(x)+{\frac {1}{2}}\|x-v\|_{\mathcal {X}}^{2}\right).} For any function in this class, the minimizer of the right-hand side above is unique, hence making the proximal operator well-defined.

Proximal gradient methods for learning

Proximal gradient methods for learning is an area of research in optimization and statistical learning theory which studies algorithms for a general class of convex regularization problems where the regularization penalty may not be differentiable. One such example is ℓ 1 {\displaystyle \ell _{1}} regularization (also known as Lasso) of the form min w ∈ R d 1 n ∑ i = 1 n ( y i − ⟨ w , x i ⟩ ) 2 + λ ‖ w ‖ 1 , where x i ∈ R d and y i ∈ R .

Frank-Wolfe algorithm

The Frank-Wolfe algorithm is an iterative first-order optimization algorithm for constrained convex optimization. Also known as the conditional gradient method, reduced gradient algorithm and the convex combination algorithm, the method was originally proposed by Marguerite Frank and Philip Wolfe in 1956.