• Separate objective function (Type I)

$$\text{minimize} f(x) + g(x)$$ rewrite: $$\text{minimize} f(x) + g(z) \text{, subject to } x - z = 0$$

• Constrained Convex optimization (Type II)

$$\text{minimize} f(x) \text{, subject to } x \in C \\ $$

rewrite:

$$\text{minimize} f(x) + g(z) \text{, subject to } x - z = 0$$

where $g(z)$ is indicator function

$$g(z) = 1_C(z) = \begin{cases} 1, & \text{ if } x \in C\\ 0, & \text{otherwise} \end{cases}$$

• loss function + regularization

$$\text{minimize } \: l(x) + \lambda||x||_1$$ Rewrite: $$\text{minimize } \: l(x) + \lambda||z||_1 \text{ , subject to } x - z = 0$$

• Sparse Linear System

$$\text{minimize } \: ||x||_1 \text{, subject to } Ax - b = 0 \\ $$ rewrite: $$\text{minimize} \: ||x||_1 + 1_C(z) \text{, subject to } x - z = 0 \\ \text{ where } C = \{x \in R^n | Ax = b \}$$ Another Examples: Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM, AAAI 2017

• Variational inequality

Given a convex set $\Omega \in R^n$ and a convex function $F: R^n \rightarrow R$, the variational inequality is to find a $x^\ast$ $$(x - x^\ast)^TF(x^\ast) \geq 0, \forall x \in \Omega$$

• Convex optimization

For diffierential convex problem $min \left\{ \Theta(x) | x \in \Omega \right\}$, if $x^\ast$ is its solution, then from the point of $x^\ast$, all loss decreasing set: $S_d(x^\ast) = \{s \in R^n | s^T\delta \Theta(x^\ast) < 0 \}$ and all possible update set $S_f(x^\ast) = \{ s \in R^n | s = x - x^\ast, x \in \Omega \}$ have no overlap: $$(x - x^\ast)^T\delta \Theta(x^\ast) \geq 0, \forall x \in \Omega$$ If $\Theta(x)$ is second-order derivable, the Hessian matrix $\delta^2 \Theta(x)$ is symmetric. In $VI(\Omega, F)$, if $F$ is derivable, we don't require the Jacobian maxtrix $\delta F(x)$ to be symmetric.

Able to Convergence

• 2-Block ADMM (equation constraint as an example)

$$\text{minimize } f_1(x) + f_2(z) \text{ , subject to } Ax + Bz = c$$

where $f_1(x)$ : $R^{n_1} \rightarrow R$, $f_2(z)$ : $R^{n_2} \rightarrow R$, $c \in R^m$. If we solve with ADMM, we have

$$x^{k+1} := \underset{x}{\text{argmin }} (f_1(x) + (\rho/2)||Ax + Bz^k - c - u^k ||_2^2) \\ z^{k+1} := \underset{z}{\text{argmin }} (f_2(z) + (\rho/2)||Ax^{k+1} + Bz - c - u^k ||_2^2) \\ u^{k+1} := u^k - \rho(Ax^{k+1} + Bz^{k+1} - c)$$

we denote $v = (z, u)$, $w = (x, z, u)$. Our goal is to prove with iteration time $k$ increases, $v$ goes to $v^\ast$, $w$ goes to $w^\ast$

Able to Convergence

• VI

$$\text{minimize } f_1(x) + f_2(z) \text{ , subject to } Ax + Bz = c$$

based on the mix-max of lagrange function, for the solution $(x^\ast, z^\ast, u^\ast)$ of the problem, we have $$\begin{cases} f_1(x) - f_1(x^\ast) + (x - x^\ast)^T (-A^Tu^\ast) \geq 0 \\ f_2(z) - f_2(z^\ast) + (z - z^\ast)^T (-B^Tu^\ast) \geq 0 \\ (u - u^\ast)^T(Ax^\ast + Bz^\ast - c) \geq 0 \\ \end{cases}$$ by defining $\lambda = \begin{pmatrix} x \\ z \end{pmatrix}$, $w = \begin{pmatrix} x \\ z \\ u \end{pmatrix}$, $G(w) = \begin{pmatrix} -A^Tu \\ -B^Tu \\ Ax + Bz - c \end{pmatrix}$ , and $F(\lambda) = f_1(x) + f_2(z)$ origin task is actually the problem VI$(\Omega, G, F)$: $$F(\lambda) - F(\lambda^\ast) + (w - w^\ast)^TG(w^\ast) \geq 0, \forall w \in \Omega$$

Able to Convergence

• 2-Block ADMM

$$f_1(x) - f_1(x^{k+1}) + (x - x^{k+1})^T (-A^Tu^k + \rho A^T(Ax^{k+1} + Bz^k -c)) \geq 0$$ holds for any $x \in R^{n_1}$, similarly, $$f_2(z) - f_2(z^{k+1}) + (z - z^{k+1})^T (-B^Tu^k + \rho B^T(Ax^{k+1} + Bz^{k+1} - c)) \geq 0$$ holds for any $z \in R^{n_2}$. Then substituing $u^{k}$ in the two inequaility, we get $$f_1(x) - f_1(x^{k+1}) + (x - x^{k+1})^T (-A^Tu^k + \rho A^T(Ax^{k+1} + Bz^k -c)) = \\ f_1(x) - f_1(x^{k+1}) + (x - x^{k+1})^T (-A^T(u^{k+1} + \rho(Ax^{k+1} + Bz^{k+1} - c)) + \rho A^T(Ax^{k+1} + Bz^k -c)) = \\ f_1(x) - f_1(x^{k+1}) + (x - x^{k+1})^T (-A^Tu^{k+1} + \rho A^TB(z^k -z^{k+1})) \geq 0$$ holds for any $x \in R^{n_1}$ and $$f_2(z) - f_2(z^{k+1}) + (z - z^{k+1})^T (-B^Tu^{k+1}) \geq 0$$ holds for any $z \in R^{n_2}$.

Able to Convergence

• 2-Block ADMM

Next, we add the two inequaility together and get: $$f_1(x) - f_1(x^{k+1}) + (x - x^{k+1})^T (-A^Tu^{k+1} + \rho A^TB(z^k -z^{k+1})) +\\ f_2(z) - f_2(z^{k+1}) + (z - z^{k+1})^T (-B^Tu^k + \rho B^T(Ax^{k+1} + Bz^{k+1} -c)) \geq 0$$ Denote $\lambda = (x, z)$ and $F(\lambda) = f_1(x) + f_2(z)$, above inequaility can be rewriten in a more compact form, $$F(\lambda) - F(\lambda^{k+1}) + \begin{pmatrix} x - x^{k+1} \\ z - z^{k+1} \end{pmatrix}^T \left\{ \begin{pmatrix} -A^Tu^{k+1} \\ -B^Tu^{k+1} \end{pmatrix} + \rho \begin{pmatrix} A^T \\ B^T \end{pmatrix} B(z^k - z^{k+1}) + \begin{pmatrix} 0 & 0 \\ 0 & \rho B^TB \end{pmatrix} \begin{pmatrix} x^{k+1} - x^k \\ z^{k+1} - z^k \end{pmatrix} \right\} \geq 0$$ for $\forall \lambda \in (R^{n_1}, R^{n_2})$ holds. We addtionally add the scaled dual variable $u$ in the inequaility to get $$F(\lambda) - F(\lambda^{k+1}) + \begin{pmatrix} x - x^{k+1} \\ z - z^{k+1} \\ u - u^{k+1} \end{pmatrix}^T \left\{ \begin{pmatrix} -A^Tu^{k+1} \\ -B^Tu^{k+1} \\ Ax^{k+1} + Bz^{k+1} - c \end{pmatrix} + \rho \begin{pmatrix} A^T \\ B^T \\ 0 \end{pmatrix} B(z^k - z^{k+1}) + \begin{pmatrix} 0 & 0 \\ 0 & \rho B^TB \\ 0 & 1/\rho I_m \end{pmatrix} \begin{pmatrix} x^{k+1} - x^k \\ z^{k+1} - z^k \end{pmatrix} \right\} \geq 0$$

Able to Convergence

• 2-Block ADMM

Last inequaility holds for all $w = (x, z, u)$, thus we can set $w = w^\ast$ and get $$F(\lambda^\ast) - F(\lambda^{k+1}) + (w^\ast - w ^{k+1})^T \left\{ \begin{pmatrix} -A^Tu^{k+1} \\ -B^Tu^{k+1} \\ Ax^{k+1} + Bz^{k+1} - c \end{pmatrix} + \rho \begin{pmatrix} A^T \\ B^T \\ 0 \end{pmatrix} B(z^k - z^{k+1}) + \begin{pmatrix} 0 & 0 \\ 0 & \rho B^TB \\ 0 & 1/\rho I_m \end{pmatrix} \begin{pmatrix} x^{k+1} - x^k \\ z^{k+1} - z^k \end{pmatrix} \right\} \geq 0$$ For simplicity, we rewrite it to $$F(\lambda^\ast) - F(\lambda^{k+1}) + (w^\ast - w ^{k+1})^T \left\{ G(w^{k+1} + P(z^k, z^{k+1}) + H_0(v^{k+1} - v^k) \right\} \geq 0$$ Then move several term form left to right, we get: $$(w^\ast - w ^{k+1})^TH_0(v^{k+1} - v^k) \geq (w ^{k+1} - w^\ast)^TP(z^k, z^{k+1}) + F(\lambda^{k+1}) - F(\lambda^\ast) + (w ^{k+1} - w^\ast)^TG(w^{k+1})$$ As the first row of $H_0$ are all zero, we can rewrite it to ($H$ represents the last two rows of $H_0$): $$(v^{k+1} - v^\ast)H(v^k - v^{k+1}) \geq (w ^{k+1} - w^\ast)^TP(z^k, z^{k+1}) + \\ F(\lambda^{k+1}) - F(\lambda^\ast) + (w ^{k+1} - w^\ast)^TG(w^{k+1})$$ On the other hand, for the second part, we have: $$\left\{F(\lambda^{k+1}) - F(\lambda^\ast) \right\} + (w ^{k+1} - w^\ast)^TG(w^{k+1}) \geq \left\{F(\lambda^{k+1}) - F(\lambda^\ast) \right\} + (w ^{k+1} - w^\ast)^TG(w^\ast) \geq 0$$ The former inquaility holds as F is monotone and the later inquaility holds as it is the definition of VI.

Able to Convergence

• 2-Block ADMM

It is clear that $$(v^{k+1} - v^\ast)H(v^k - v^{k+1}) \geq (w ^{k+1} - w^\ast)^TP(z^k, z^{k+1})$$ For the right side, we have $$\begin{align} & (w ^{k+1} - w^\ast)^TP(z^k, z^{k+1}) = \\ & (w ^{k+1} - w^\ast)^T \rho \begin{pmatrix} A^T \\ B^T \\ 0 \end{pmatrix} B(z^k - z^{k+1}) = \\ & (B(z^k - z^{k+1}))^T \rho (A, B, 0)(w ^{k+1} - w^\ast) = \\ & (B(z^k - z^{k+1}))^T \rho (Ax^{k+1} + Bz^{k+1}) - (Ax^\ast + Bz^\ast) = \\ & (B(z^k - z^{k+1}))^T \rho (Ax^{k+1} + Bz^{k+1} - c) = \\ & (B(z^k - z^{k+1}))^T (u^k - u^{k+1}) = \\ & (u^k - u^{k+1})^TB(z^k - z^{k+1}) \end{align}$$

Able to Convergence

• 2-Block ADMM

Recall that $$f_2(z) - f_2(z^{k+1}) + (z - z^{k+1})^T (-B^Tu^{k+1}) \geq 0$$ holds for any $z \in R^{n_2}$. It can also be known that $$f_2(z) - f_2(z^{k}) + (z - z^{k})^T (-B^Tu^{k}) \geq 0$$ holds for any $z \in R^{n_2}$. Set the $z$ to be $z^{k}$ in the former inequation and $z^{k+1}$ in the latter inequation, and add the two inequations, we get: $$(z^k - z^{k+1})^T (-B^Tu^{k+1}) + (z^{k+1} - z^{k})^T (-B^Tu^{k}) = (u^k - u^{k+1})^TB(z^k - z^{k+1}) \geq 0$$ It ensures $$(w ^{k+1} - w^\ast)^TP(z^k, z^{k+1}) = (u^k - u^{k+1})^TB(z^k - z^{k+1}) \geq 0$$ Overall, we have $$(v^{k+1} - v^\ast)H(v^k - v^{k+1}) \geq (w ^{k+1} - w^\ast)^TP(z^k, z^{k+1}) \geq 0$$

Able to Convergence

• 2-Block ADMM

Having the $$(v^{k+1} - v^\ast)H(v^k - v^{k+1}) \geq 0$$ It is easy to prove that $$(v^{k} - v^\ast)^TH(v^k - v^{\ast}) = ||(v^{k} - v^\ast)||_H^2 = \\ ||((v^{k+1} - v^\ast) + (v^{k} - v^{k+1}))||_H^2 = \\ ||((v^{k+1} - v^\ast)||_H^2 + 2 (v^{k+1} - v^\ast)H(v^k - v^{k+1}) + ||(v^{k} - v^{k+1}))||_H^2 \geq ||((v^{k+1} - v^\ast)||_H^2 + ||(v^{k} - v^{k+1}))||_H^2$$ namely, $$||((v^{k+1} - v^\ast)||_H^2 \leq ||(v^{k} - v^\ast)||_H^2 - ||(v^{k} - v^{k+1}))||_H^2$$ the sequence is monotonically deceasing (in the word, it can convergence). It further can be proved to convergene at rate $(1/t)$