Welcome to macaw!¶
macaw is a colorful long-tailed package for beautiful Majorization-Minimization applied to Machine Learning
Majorization-minimization is all about inequalities such as
- Jensen’s Inequality:
\[\phi\left(\mathbb{E}\left[X\right]\right) \leq \mathbb{E}\left[\phi\left(X\right)\right],\]
for convex \(\phi\).
- Cauchy-Schwarz Inequality:
\[\left(\mathbb{E}\left[XY\right]\right)^2 \leq \mathbb{E}\left[X^2\right]\mathbb{E}\left[Y^2\right]\]
- Arithmetic and Geometric Means:
\[\mathbb{E}\left[\left(X + Y\right)^2\right] \leq 4\mathbb{E}\left[XY\right]\]
Those inequalities are applied to complicated objective functions in order to find upper bounds (majorize), which are subsequently minimized. Iterating this procedure has been proven to be powerful tool for optimization problems that arise often in signal processing and machine learning ;)
See, for instance, Majorization-Minimization Algorithms in Signal Processing, Communications, and Machine Learning by Y. Sun, P. Babu, and D. P. Palomar.