Convergence Rate Analysis of Modified BiG-SAM for Solving Bi-Level Optimization Problems Based on S-FISTA

In this paper, we consider a more general bi-level optimization problem, where the inner objective function is consisted of three convex functions, involving a smooth and two non-smooth functions. The outer objective function is a classical strongly convex function which may not be smooth. Motivated by the smoothing approaches, we modify the classical bi-level gradient sequential averaging method to solve the bi-level optimization problem. Under some mild conditions, we obtain the convergence rate of the generated sequence, and then based on the analysis framework of S-FISTA, we show the global convergence rate of the proposed algorithm.

• Bi-Level Optimization
• Convex Problems
• First-Order Methods
• Proximal Gradient Method
• Sequential Averaging Method
• Moreau Envelope

1. Introduction

In this paper, we mainly consider the bi-level optimization problems, which is derived from the Stackelberg game in game theorey. Bi-level optimization problems is a special kind of optimization problem that involves two levels, called outer level and inner level. This structure means that the goals and constraints of the outer level problem depend on the solution of the inner problem. Bi-level optimization problems have a wide range of applications in many fields, including economics, engineering design, transportation planning, machine learning and so on.

Recall that the classical bi-level optimization problem, where the outer and inner objective functions are convex. The outer objective function is a constrained minimization problem, it is

$\underset{x\in X^{\text{*}}}{\text{min}}\omega \left(x\right),$ (OP)

where $X^{\text{*}}$ is the set of minimizers of the inner objective function, which is a composite convex minimization problem, as follows,

$\underset{x\in {ℝ}^n}{\text{min}}\left\{\Phi \left(x\right):=f\left(x\right)+g\left(x\right)\right\}.$ (P1)

In this case, $\omega$ is a strong convex and differentiable function, $f$ is a convex and continuously differentiable function and $g$ is an extended real-valued function on ${ℝ}^n$. Here, $g$ maybe is a nonsmooth function. Problem (OP)-(P1) is called simple bi-level optimization in [1] , which is opposed to the more general version of the problem, see in [2] .

Note that, both inner problem (P1)and outer problem (OP) are classical convex optimization problems, which can be solved in different cases, by projected gradient, proximal gradient algorithm, forward-backward splitting algorithm, and so on. However, if we combine problem (OP) and (P1) together, it is difficult to handle.

For problem (OP)-(P1), we can solve it directly or indirectly. In general, we can transform the bi-level optimization problems into a simple optimization problem structure. Then, it can be solved indirectly and easily. The common method for solving the classical bi-level optimization problem is Tikhonov regularization [3] , it is for some $\theta >0$, solving the following regularized problem:

$\underset{x\in {ℝ}^n}{\text{min}}\left\{{\Phi }_{\theta }\left(x\right):=\text{Φ}\left(x\right)+\theta \omega \left(x\right)\right\}.$ (1.1)

Problem (OP) and (P1) can be traced back to the work of Managsarian and Meyer [4] in the process of developing efficient algorithms for large scale linear programs. They proposed a modification of the Tikhonov regularization technique [3] , the underlying idea is called finite-perturbation property. It is that finding a parameter ${\theta }^{\text{*}}$ (Tikhonov perturbation parameter) such that for all $\theta \in \left[0,{\theta }^{\text{*}}\right]$,

$\text{arg}\underset{x\in X^{\text{*}}}{\text{min}}\omega \left(x\right)=\text{arg}\underset{x\in {ℝ}^n}{\text{min}}{\Phi }_{\theta }\left(x\right).$

This property is proven by Managsarian initially, when the inner level problem is a linear program. Then, it was extended by Ferris et al. [5] , where the inner objective problem is general convex optimization problem.

In [5] , they considered the case that $g$ is an indicator function of a closed convex set $C$, and under some restrictions, they demonstrated that the optimal solution of problem (1.1) is the optimal solution of problem (OP), when there exists a small enough ${\theta }^{\text{*}}>0$. In practice, the value of ${\theta }^{\text{*}}$ is unknown, it means that solving problem (1.1) should depend on a sequence of regularizing parameters $\left\{{\theta }_n\right\}$, where ${\theta }_n\to 0$ as $n\to +\infty$. Solodov [6] showed that for ${\displaystyle {\sum }_{n=1}^{\infty }{\theta }_n}=\infty$, there is no need to find the optimal solution of problem (1.1) with indicator ${\theta }_n$. He proposed an explicit and more tractable proximal point method for the bi-level optimization problem (OP)-(P1), which is opposite to the algorithm proposed by Cabot [7] , where the approximation scheme is only implicit thus making the method of Cabot [7] not easy to implement. Based on the proximal point algorithm, some researchers developed various proximal point algorithms to solve the problems under different types of framework, see [8] [9] .

On the other hand, we can solve the bi-level problem (OP)-(P1) by a direct approach, called hybrid steepest descent method [10] , where the sequence converges to the optimal solution according to ${\displaystyle {\sum }_{n=1}^{\infty }{\theta }_n}=\infty$ and ${\theta }_n\to 0\left(n\to \infty \right)$. Then, the hybrid steepest descent method was further extended by Neto et al. for solving a more general outer objective function.

Recently, Beck et al. [11] proposed a new direct first order method, which is called Minimal Norm Gradient, it can solve problem (OP). The author proved that in terms of the inner objective function, the algorithm has $\mathcal{O}\left(1/\sqrt{n}\right)$ convergence rate result. However, for some choice of outer objective function $\omega$, the computation of this method is so expensive to get the optimal solution. Motivated by the minimal norm gradient method, Sabach [12] suggested a first order method, called BiG-SAM, to solve the bilevel optimization problem, which is based on existing viscosity approximation methods [13] . According to the convergence analysis of the BiG-SAM, they get $\mathcal{O}\left(1/n\right)$ global convergence rate of in the light of the inner level function. In addition, Yekini et al. combined inertial technique with Big-SAM and proposed an inertial BiG-SAM algorithm, more details see in [14] .

In this paper, we consider a more general composite convex function as the inner objective function of the bi-level optimization problem. It is,

$\underset{x\in {ℝ}^n}{\text{min}}\left\{H\left(x\right):=f\left(x\right)+h\left(x\right)+g\left(x\right)\right\},$ (P2)

where $f$ is a continuously differentiable function with $L_f$-Lipschitz continuous gradient, $h$ is a real-valued and convex and $g$ is an extended valued function. It is rich enough to cover many interesting generic optimization models by appropriate choices of $\left(f,h,g\right)$. For more details about the assumption of the functions we will give in the following Section 2. Let $x_{op}^{\text{*}}$ is the unique optimal solution of problem (OP).

This paper is organized as follows. In Section 2, we use smooth technique partially smooth inner level problem (P2), construct the inner objective function (Q) and give some useful lemmas for the convergence rate analysis. In Section 3, we introduce a new BiG-SAM algorithm for solving the bi-level optimization problem with outer level (OP). In Section 4, we investigate the convergence rate of BiG-SAM for non-smooth version of bi-level optimization problem.

2. Motivation and Construction

In this section, we will present the motivation and the process of our algorithm design, as well as the useful lemma. Recall the bi-level optimization problem, where the outer level is problem (OP), the inner level is

$\underset{x\in {ℝ}^n}{\text{min}}\left\{H\left(x\right):=f\left(x\right)+h\left(x\right)+g\left(x\right)\right\},$ (P2)

where $f$, $h$ and $g$ are satisfy the following Assumption.

Assumption I:

i) $f:{ℝ}^n\to ℝ$ is a convex and continuous differentiable function, it has a Lipschitz continuous gradient with constant $L_f$, i.e.,

$\Vert \nabla f\left(x\right)-\nabla f\left(y\right)\Vert \le L_f\Vert x-y\Vert ,\forall x,y\in {ℝ}^n$

ii) $h:{ℝ}^n\to ℝ$ is a $\left(\alpha ,\beta \right)$-smoothable function, $\left(\alpha ,\beta >0\right)$. It is that for any $\mu >0$, $h_{\mu }$ denotes a $\frac{1}{\mu }$-smooth approximation of $h$ with parameters $\left(\alpha ,\beta \right)$.

iii) $g:{ℝ}^n\to \left(-\infty ,\infty \right]$ is a proper, lower semicontinuous and convex function.

iv) $H$ has bounded level sets. Specifically, for any $\delta >0$, there exists $R_{\delta }>0$ such that

$\Vert x\Vert \le R_{\delta }\text{for}\text{any}x\text{satisfying}H\left(x\right)\le \delta .$

v) Let $X_{P2}^{\text{*}}$ be the optimal set of problem (P2), and it is nonempty. Set $H_{\text{opt}}$ as the optimal value of the problem (P2).

Definition 2.1. [15] A convex function $h:{ℝ}^n\to ℝ$ is called $\left(\alpha ,\beta \right)$-smoothable, $\left(\alpha ,\beta >0\right)$ if for any $\mu >0$, there exists a convex differentiable function $h:{ℝ}^n\to ℝ$ such that the following holds:

a) $h_{\mu }\left(x\right)\le h\left(x\right)\le h_{\mu }\left(x\right)+\beta \mu$ for all $x\in {ℝ}^n$.

b) $h_{\mu }$ is $\frac{\alpha }{\mu }$-smooth.

The function $h_{\mu }$ is called a $\frac{1}{\mu }$-smooth approximation of $h$ with parameters $\left(\alpha ,\beta \right)$.

According to the definition of $h$, combined with the Definition 2.1, we can smooth $h$ as a $1/\mu$-smooth function $h_{\mu }$. Then, problem (P2) becomes into

$\underset{x\in {ℝ}^n}{\text{min}}\left\{H_{\mu }\left(x\right):=\underset{F_{\mu }\left(x\right)}{\underset{︸}{f\left(x\right)+h_{\mu }\left(x\right)}}+g\left(x\right)\right\}.$

For convenience, we write the above composite minimization problem as the following form,

$\underset{x\in {ℝ}^n}{\text{min}}\left\{F_{\mu }\left(x\right)+g\left(x\right)\right\}.$ (Q)

Remark 2.1. Here, let $X^{*}$ be the optimal solution set of problem (Q), which is non-empty and $X^{*}\subseteq X_{P2}^{*}$. When $\mu$ is small enough, the optimal solution set $X^{*}$ is equal to $X_{P_2}^{*}$. This implies that when $\mu$ is small enough, the optimal solution of $\omega$ over $X^{*}$ is equivalent to the optimal solution of $\omega$ over $X_{P2}^{*}$, i.e.,

$\underset{x\in X^{\text{*}}}{\text{min}}\omega \left(x\right)=\underset{x\in X_{P2}^{\text{*}}}{\text{min}}\omega \left(x\right).$

Observe that problem (P2) is a non-smooth composite function, involving two non-smooth functions. A common methodology for solving non-smooth optimization problems is to replace the original problem by a sequence of approximating smooth problems, and then using direct and classical methods [16] to solve. The main idea is to transform the nondifferentiable problem into a smooth problem, there are many different smoothing approaches to various classes of non-smooth optimization problem, see in [17] - [19] . Motivated by the work of Beck et al. [15] , we consider to partially smooth the inner objective function (P2) and transform it into a classical structure of convex optimization problem. The motivation of this approach is twofold. Firstly, according to the design and algorithmic analysis of the related schemes, it comes from the classical composite optimization problem formula, like (P1) where $f$ is smooth and $g$ is nonsmooth, can be solved by gradient-based algorithms, [20] [21] . Second, in many applications [22] [23] , one of the non-smooth terms in (Q), plays a key role in describing a desirable property of the decision variable $x$. If we smooth all non-smooth functions in (P2), it will destroy the property of $x$.

Since $h_{\mu }$ is $\frac{\alpha }{\mu }$-smooth, it has $\frac{\alpha }{\mu }$-Lipschitz continuous gradient $\nabla h_u$. Due to $F_{\mu }=f+h_{\mu }$ and $f$ is also have $L_f$-Lipschitz continuous gradient, it implies the $F_{\mu }$ is a continuous differentiable convex function, the Lipschitz constant of the gradient is equal to $L_f+\frac{\alpha }{\mu }$. Thus, problem (Q) can be solved by the classical proximal gradient (PG) method or proximal forward-backward method, the iteration is as follow:

$x^{n+1}={\text{prox}}_{\lambda g}\left(x^n-\lambda \nabla F_{\mu }\left(x^n\right)\right),n\in \mathbb{N},$ (2.1)

where the stepsize is $\lambda =1/\left(L_f+\frac{\alpha }{\mu }\right)$. Since $g$ is a proper, lower semicontinuous and convex function, ${\text{prox}}_{\lambda g}$ is called Moreau Proximal Mapping, which is defined as follow:

${\text{prox}}_{\lambda g}\left(x\right):=\underset{u\in {ℝ}^n}{\arg \min }\left\{g\left(u\right)+\frac{1}{2\lambda }{\Vert u-x\Vert }^2\right\}.$ (2.2)

In addition, the PG method (1) can be regarded as a fixed-point algorithm, it can be formulated as

$T_{\lambda }\left(x\right):={\text{prox}}_{\lambda g}\left(x-\lambda \nabla F_{\mu }\left(x\right)\right),$ (2.3)

it is called the prox-grad mapping (proximal-gradient mapping). Denote $\text{Fix}\left(T_{\lambda }\right):=\left\{x\in {ℝ}^n|T_{\lambda }\left(x\right)=x\right\}$, it is the fixed point set of $T_{\lambda }$. From [24] and [22] , we have the following two crucial properties.

Lemma 2.1. [12]

i) The prox-grad mapping $T_{\lambda }$ is nonexpansive for all $\lambda \in \left(0,1/\left(L_f+\alpha /\mu \right)\right]$, that is,

$T_{\lambda }\left(x\right)-T_{\lambda }\left(y\right)\le \Vert x-y\Vert ,\forall x,y\in {ℝ}^n$ (2.4)

ii) Fixed points of the prox-grad mapping $T_{\lambda }$ are optimal solutions of problem (Q) and the reverse is also true, i.e.,

$x\in X^{\text{*}}\iff x=T_{\lambda }\left(x\right)={\text{prox}}_{\lambda g}\left(x-\lambda \nabla F_{\mu }\left(x\right)\right)$ (2.5)

Therefore, we have that $\text{Fix}\left(T_{\lambda }\right)=X^{\text{*}}$ for all $\lambda >0$.

Now, we give a key proposition, which is a significant result in convergence rate analysis. Indeed, we consider the following quadratic approximation of $H_{\mu }\left(x\right):=f\left(x\right)+h_{\mu }\left(x\right)+g\left(x\right)$ at $y$, it is:

$Q_{\lambda }\left(x,y\right):=f\left(y\right)+h_{\mu }\left(y\right)+\langle x-y,\nabla f\left(y\right)+\nabla h_{\mu }\left(y\right)\rangle +\frac{1}{2\lambda }{\Vert x-y\Vert }^2+g\left(x\right),$

which admits a unique minimizer

$p_{\lambda }\left(y\right):=\arg \min \left\{Q_{\lambda }\left(x,y\right):x\in {ℝ}^n\right\}.$

It implies that we have,

$p_{\lambda }\left(y\right)=\underset{x\in {ℝ}^n}{\arg \min }\left\{g\left(x\right)+\frac{1}{2\lambda }x-{\left(y-\lambda \nabla f\left(y\right)-\lambda \nabla h_{\mu }\left(y\right)\right)}^2\right\}.$

From the characterize of the optimality of $p_{\lambda }(\cdot )$, we have the following lemma.

Lemma 2.2. For any $y\in {ℝ}^n$, one has $z=p_{\lambda }\left(y\right)$ if and only if there exists $\gamma \left(y\right)\in \partial g\left(z\right)$, the subdifferential of $g(\cdot )$, such that

$\nabla f\left(y\right)+\nabla h_{\mu }\left(y\right)+\frac{1}{\lambda }\left(z-y\right)+\gamma \left(y\right)=0.$ (2.6)

Then we have the following proposition.

Proposition 2.1. Suppose that Assumption I holds true. Let $y\in {ℝ}^n$ and denote $p_{\lambda }\left(y\right)=T_{\lambda }\left(y\right)$, such that

$H_{\mu }\left(p_{\lambda }\left(y\right)\right)\le Q_{\lambda }\left(p_{\lambda }\left(y\right),y\right).$ (2.7)

Then, for any $\lambda \le 1/\left(L_f+\alpha /\mu \right)$ and $x\in {ℝ}^n$, we have

$H_{\mu }\left(x\right)-H_{\mu }\left(p_{\lambda }\left(y\right)\right)\ge \frac{1}{2\lambda }{\Vert p_{\lambda }\left(y\right)-y\Vert }^2+\frac{1}{\lambda }\langle y-x,p_{\lambda }\left(y\right)-y\rangle .$ (2.8)

Proof. From (2.7), we have,

$H_{\mu }\left(x\right)-H_{\mu }\left(p_{\lambda }\left(y\right)\right)\ge H_{\mu }\left(x\right)-Q_{\lambda }\left(p_{\lambda }\left(y\right),y\right).$ (2.9)

Since $f,h$, and $g$ are convex, it implies

$f\left(x\right)\ge f\left(y\right)+\langle x-y,\nabla f\left(y\right)\rangle ,$

$h_{\mu }\left(x\right)\ge h_{\mu }\left(y\right)+\langle x-y,\nabla h_{\mu }\left(y\right)\rangle ,$

$g\left(x\right)\ge g\left(p_{\lambda }\left(y\right)\right)+\langle x-p_{\lambda }\left(y\right),\gamma \left(y\right)\rangle ,$

where the $\gamma \left(y\right)$ is defined from lemma 2.2. Now, Summing the above inequalities together, we have

$\begin{array}{c}{H}_{\mu }\left(x\right)\ge f\left(y\right)+h_{\mu }\left(y\right)+\langle x-y,\nabla f\left(y\right)+\nabla h_{\mu }\left(y\right)\rangle +g\left(p_{\lambda }\left(y\right)\right)\\ +\langle x-p_{\lambda }\left(y\right),\gamma \left(y\right)\rangle .\end{array}$ (2.10)

On the other hand, from the definition of $Q_{\lambda }\left(x,y\right)$, let $x:=p_{\lambda }\left(y\right)$, we have

$\begin{array}{c}{Q}_{\lambda }\left(p_{\lambda }\left(y\right),y\right)=f\left(y\right)+h_{\mu }\left(y\right)+\langle p_{\lambda }\left(y\right)-y,\nabla f\left(y\right)+\nabla h_{\mu }\left(y\right)\rangle \\ +\frac{1}{2\lambda }{\Vert p_{\lambda }\left(y\right)-y\Vert }^2+g\left(p_{\lambda }\left(y\right)\right).\end{array}$ (2.11)

Now, combine (9) with (10) and (11), it follows that

$\begin{array}{c}{H}_{\mu }\left(x\right)-H_{\mu }\left(p_{\lambda }\left(y\right)\right)\ge H_{\mu }\left(x\right)-Q_{\lambda }\left(p_{\lambda }\left(y\right),y\right)\\ \ge \langle x-p_{\lambda }\left(y\right),\nabla f\left(y\right)+\nabla h_{\mu }\left(y\right)+\gamma \left(y\right)\rangle -\frac{1}{2\lambda }{\Vert p_{\lambda }\left(y\right)-y\Vert }^2\\ =-\frac{1}{2\lambda }{\Vert p_{\lambda }\left(y\right)-y\Vert }^2+\langle x-p_{\lambda }\left(y\right),-\frac{1}{\lambda }\left(p_{\lambda }\left(y\right)-y\right)\rangle \\ =-\frac{1}{2\lambda }{\Vert p_{\lambda }\left(y\right)-y\Vert }^2+\frac{1}{\lambda }\langle p_{\lambda }\left(y\right)-y+y-x,p_{\lambda }\left(y\right)-y\rangle \\ =-\frac{1}{2\lambda }{\Vert p_{\lambda }\left(y\right)-y\Vert }^2+\frac{1}{\lambda }{\Vert p_{\lambda }\left(y\right)-y\Vert }^2+\frac{1}{\lambda }\langle y-x,p_{\lambda }\left(y\right)-y\rangle \\ =\frac{1}{2\lambda }{\Vert p_{\lambda }\left(y\right)-y\Vert }^2+\frac{1}{\lambda }\langle y-x,p_{\lambda }\left(y\right)-y\rangle ,\end{array}$

where the first equality is getting from we used (6). Thus, we complete the proof.

□

Now, we turn to discuss the details of outer level problem (OP). Recall the formulation of (OP), it is a convex constraint optimization, where $X^{\text{*}}$ is the optimal solution set of problem (Q). In general, we suppose that outer objective function (OP) satisfies the following assumptions.

Assumption II.

i) $\omega :{ℝ}^n\to ℝ$ is $\sigma$-strongly convex, $\sigma >0$.

ii) $\omega$ is a continuously differentiable function such that $\nabla \omega$ is Lipschitz continuous with constant $L_{\omega }>0$.

Due to $\omega$ is differential, we can use the gradient descent method solving the outer level problem (OP). Nevertheless, not all the outer function $\omega$ satisfies Assumption II (ii), that is, $\omega$ is nonsmooth. So, we assume $\omega$ satisfies the following property.

Assumption III: $\omega :{ℝ}^n\to ℝ$ is strong convex with parameter $\sigma >0$ and ${\ell }_{\omega }$-Lipschitz continuous.

In this case, we can depend on the Moreau envelop of $\omega$ and solve the outer level problem, which is denoted by $M_{s\omega }$, the formula is as follow:

$M_{s\omega }\left(x\right)=\underset{u\in {ℝ}^n}{\text{min}}\left\{\omega \left(u\right)+\frac{1}{2s}{\Vert u-x\Vert }^2\right\}.$ (2.12)

It is well-known that $M_{s\omega }$ is continuously differentiable on ${ℝ}^n$ with an $1/s$-Lipschitz continuous gradient, which is given by

$\nabla M_{s\omega }\left(x\right)=\frac{1}{s}\left(x-{\text{prox}}_{s\omega }\left(x\right)\right).$ (2.13)

Additionaly, the Moreau envelope has another useful property, that is:

Lemma 2.3. [12] Let $\omega :{ℝ}^n\to \left(-\infty ,\infty \right]$ be a strongly convex function with strong convexity parameter $\sigma$ and let $s>0$. Then, the Moreau envelope $M_{s\omega }$ is strongly convex with parameter $\sigma /\left(1+s\sigma \right)$.

Definition 2.2. [12] A mapping $S:{ℝ}^n\to {ℝ}^n$ is said to be $\eta$-contraction if there exists $\eta <1$ such that

$\Vert S\left(x\right)-S\left(y\right)\Vert \le \eta \Vert x-y\Vert ,\forall x,y\in {ℝ}^n.$

When $\omega$ satisfies Assumption II, the following result is crucial for our derivations.

Lemma 2.4. [12] Suppose that Assumption II holds and let $I$ is a identity operator. Then, the mapping $S_s=I-s\nabla \omega$ is a contractive operator, for all $s\le 2/\left(L_{\omega }+\sigma \right)$, that is,

$\Vert x-s\nabla \omega \left(x\right)-\left(y-s\nabla \omega \left(y\right)\right)\Vert \le \sqrt{1-\frac{2s\sigma L_{\omega }}{\sigma +L_{\omega }}}\Vert x-y\Vert ,\forall x,y\in {ℝ}^n.$ (2.14)

3. BiG-SAM Algorithm for Smooth Bi-Level Optimization

In this section, we will introduce a new BiG-SAM algorithm for solving bi-level optimization. Firstly, we similarly construct a general framwork for the bi-level problem, consisting of inner problem (Q).

3.1. The General Framework

Motivated by Sabach et al. [12] , our approach is also to use the Sequential Averaging Method (SAM), in which we can handle the fixed point problem, proposed in [13] . Right now, we will analyse how to use it for solving the bi-level optimization problems, which is made up of problem (OP) and (Q). The sequence $\left\{x^n\right\}$ generated by SAM algorithm, converges to a solution of the fixed-point problem [13] . The iteration is

$x^n={\alpha }_{n}S\left(x^{n-1}\right)+\left(1-{\alpha }_n\right)T\left(x^{n-1}\right),$

where ${\left\{{\alpha }_n\right\}}_{n\in \mathbb{N}}$ is a carefully chosen sequence in $\left(0,1\right]$.

The above algorithm, designed in [13] , is to find a fixed-point of a nonexpansive operator $T$, i.e. $x^{\text{*}}\in \text{Fix}\left(T\right)$. This point also satisfies a variational inequality:

$\langle x^{\text{*}}-S\left(x^{\text{*}}\right),x-x^{\text{*}}\rangle \ge 0,\forall x\in \text{Fix}\left(T\right),$ (3.1)

where $S$ is a contraction mapping. Here, it means that $x^{\text{*}}$ is the “better” fixed-point in $\text{Fix}\left(T\right)$. Where $\left\{{\alpha }_n\right\}$ satisfies the following assumption.

Assumption IV. Let ${\left\{{\alpha }_n\right\}}_{n\in \mathbb{N}}$ be a sequence of real numbers in $\left(0,1\right]$ which satisfies ${\lim }_{n\to \infty }{\alpha }_n=0$, ${\displaystyle {\sum }_{n=1}^{\infty }{\alpha }_n}=\infty$ and ${\lim }_{n\to \infty }{\alpha }_{n+1}/{\alpha }_n=1$.

It should be noted that Assumption IV holds true for several choices of sequences ${\left\{{\alpha }_n\right\}}_{n\in \mathbb{N}}$ which include, for example, ${\alpha }_n=\alpha /n,n\in \mathbb{N}$ for any choice of $\alpha \in \left(0,1\right]$.

The following lemma summarizes the key results on SAM, as established in ( [13] , Theorem 3.2), which serve as the foundation for this paper.

Lemma 3.1. [12] Assume that $S:{ℝ}^n\to {ℝ}^n$ is a $\eta$-contraction and that $T:{ℝ}^n\to {ℝ}^n$ is nonexpansive mapping, for which $\text{Fix}\left(T\right)\ne \varnothing$. Let ${\left\{x^n\right\}}_{n\in \mathbb{N}}$ be the sequence generated by SAM. If Assumption IV holds true, then the following assertions are valid.

i) The sequence ${\left\{x^n\right\}}_{n\in \mathbb{N}}$ is bounded, in particular, for any $\tilde{x}\in \text{Fix}\left(T\right)$ we have, for all $n\in \mathbb{N}$, that

$\Vert x^n-\tilde{x}\Vert \le C_{\tilde{x}}:=\text{max}\left\{\Vert x^0-\tilde{x}\Vert ,\frac{1}{1-\eta }\Vert \left(I-S\right)\tilde{x}\Vert \right\}.$ (3.2)

Moreover, for all $n\in \mathbb{N}$, we also have that

$\Vert T\left(x^n\right)-\tilde{x}\Vert \le C_{\tilde{x}}\text{and}\Vert S\left(x^n\right)-S\left(\tilde{x}\right)\Vert \le \eta C_{\tilde{x}}.$

ii) The sequence ${\left\{x^n\right\}}_{n\in \mathbb{N}}$ converges to some $x^{\text{*}}\in \text{Fix}\left(T\right)$.

iii) The limit point $x^{\text{*}}$ of ${\left\{x^n\right\}}_{n\in \mathbb{N}}$, which the existence is ensured by (ii), satisfies the following variational inequality

$\langle x^{\text{*}}-S\left(x^{\text{*}}\right),x-x^{\text{*}}\rangle \ge 0,\forall x\in \text{Fix}\left(T\right).$ (3.3)

3.2. SAM for Smooth Bi-Level Optimization Problem

From the Section 2, we know that the inner level optimization problem (P2) can be smoothed as problem (Q), it has a same structure of the inner level optimization problem in [12] . Inspired by the works of [12] , we can match the outer problem (OP) and the inner problem (Q) with mapping $S$ and $T$, respectively. Here, we know that,

i) The mapping $T$ and its fixed-point set $\text{Fix}\left(T\right)$ are related to problem (Q) with the composite function $H_{\mu }=F_{\mu }+g$ and the optimal solution set $X^{\text{*}}$.

ii) The mapping $S$ is related to problem (OP) and the outer objective function $\omega$.

Thus, we set $T$ as the prox-grad mapping defined in (2.3), that is, for some $\lambda \in \left(0,1/\left(L_f+\alpha /\mu \right)\right]$ we have

$T\left(x\right):=T_{\lambda }\left(x\right)={\text{prox}}_{\lambda g}\left(x-\lambda \nabla F_{\mu }\left(x\right)\right).$ (3.4)

According to Lemma 2.1 and based on the Assumption I, it implies that $T$ is nonexpansive and $\text{Fix}\left(T\right)=X^{\text{*}}$. Then, from Lemma 2.4 and Assumption II, we can construct the $\eta$-contraction mapping $S$ as follow:

$S\left(x\right):=x-s\nabla \omega \left(x\right),$ (3.5)

where $s\in \left(0,2/\left(\sigma +L_{\omega }\right)\right]$, and the contraction parameter is $\eta ={\left(1-2s{L}_{\omega }\sigma /\left(L_{\omega }+\sigma \right)\right)}^{1/2}$.

Similarly, we use the Sequential Averaging Method (SAM) to design a new BiG-SAM algorithm for solving the bi-level optimization problems(Q) and (OP). The iteration is as follow.

New Bi-level Gradient SAM (BiG-SAM)

Input: $\lambda \in \left(0,1/\left(L_f+\alpha /\mu \right)\right],s\in \left(0,2/\left(L_{\omega }+\sigma \right)\right]$, and ${\left\{{\alpha }_n\right\}}_{n\in \mathbb{N}}$ satisfying Assumption IV.

Initialization: $x^0\in {ℝ}^n$.

General Step ( $n=1,2,\cdots$):

$y^n={\text{prox}}_{\lambda g}\left(x^{n-1}-\lambda \nabla F_{\mu }\left(x^{n-1}\right)\right),$ (3.6) $z^n=x^{n-1}-s\nabla \omega \left(x^{n-1}\right),$ (3.7) $x^n={\alpha }_n{z}^n+\left(1-{\alpha }_n\right)y^n.$ (3.8)

Due to the new BiG-SAM algorithm is similar to the works in [12] , we can get the similar convergence result.

Lemma 3.2. Let ${\left\{x^n\right\}}_{n\in \mathbb{N}}$ be a sequence generated by the new BiG-SAM. Suppose that Assumptions I, II and IV hold true. Then, the sequence ${\left\{x^n\right\}}_{n\in \mathbb{N}}$ converges to $x^{*}\in X^{*}$ which satisfies

$\langle \nabla \omega \left(x^{\text{*}}\right),x-x^{\text{*}}\rangle \ge 0,\forall x\in X^{\text{*}},$ (3.9)

and therefore, $x^{\text{*}}=x_{op}^{\text{*}}$ is the optimal solution of problem (OP).

The proof of lemma 3.2 is similar to the proof Proposition 5 in [12] .

3.3. The Global Convergence Rate of BiG-SAM

In this section, we first prove a technical result on the convergence rate of the gap between successive SAM iterations for fixed-point problems, as described in Section 3.1. Then, we use this to derive our main result: a convergence rate for BiG-SAM in terms of the values of the inner objective function.

We first present a technical lemma which will assist us in the rate of convergence proof.

Lemma 3.3. [12] Let $M>0$. Assume that ${\left\{a_n\right\}}_{n\in \mathbb{N}}$ is a sequence of nonnegative real numbers which satisfy $a_1\le M$ and

$a_{n+1}\le \left(1-\gamma b_{n+1}\right)a_n+\left(b_n-b_{n+1}\right)c_n,n\ge 1,$

where $\gamma \in \left(0,1\right]$, ${\left\{b_n\right\}}_{n\in \mathbb{N}}$ is a sequence which is defined by $b_n=\text{min}\left\{2/\left(\gamma n\right),1\right\}$, and ${\left\{c_n\right\}}_{n\in \mathbb{N}}$ is a sequence of real numbers such that $c_n\le M<\infty$. Then, the sequence ${\left\{a_n\right\}}_{n\in \mathbb{N}}$ satisfies

$a_n\le \frac{MJ}{\gamma n},n\ge 1,$

where $J=\lfloor 2/\gamma \rfloor$.

For simplicity, we denote $y^n=T\left(x^{n-1}\right)$ and $z^n=S\left(x^{n-1}\right)$ for any $n\in \mathbb{N}$. The convergence analysis rate is divided into two parts, which ultimately lead to the main conclusions of Theorem 3.1 and Theorem 3.2. Lemma 3.4 provides useful inequalities, while Proposition 3.1 shows that by choosing an appropriate sequence ${\left\{{\alpha }_n\right\}}_{n\in \mathbb{N}}$, the distance between successive elements of ${\left\{x^n\right\}}_{n\in \mathbb{N}}$ is bounded by $\mathcal{O}\left(1/n\right)$, and the sequence converges to a fixed-point of $T$ at the same rate.

Lemma 3.4. [12] Assume that $S:{ℝ}^n\to {ℝ}^n$ is a $\eta$-contraction and that $T:{ℝ}^n\to {ℝ}^n$ is nonexpansive mapping, for which $\text{Fix}\left(T\right)\ne \varnothing$. Let ${\left\{x^n\right\}}_{n\in \mathbb{N}},{\left\{y^n\right\}}_{n\in \mathbb{N}}$ and ${\left\{z^n\right\}}_{n\in \mathbb{N}}$ be sequences generated by SAM. Then, for any $n\ge 1$ and any $\tilde{x}\in \text{Fix}\left(T\right)$, defining $\tilde{z}=S\left(\tilde{x}\right)$ the following inequalities hold true

$\Vert y^{n+1}-y^n\Vert \le \Vert x^n-x^{n-1}\Vert ,$ (3.10)

$\Vert z^{n+1}-z^n\Vert \le \eta \Vert x^n-x^{n-1}\Vert ,$ (3.11)

$\Vert y^n-\tilde{x}\Vert \le \Vert x^{n-1}-\tilde{x}\Vert ,$ (3.12)

$\Vert z^n-\tilde{z}\Vert \le \eta \Vert x^{n-1}-\tilde{x}\Vert .$ (3.13)

Now, we prove the convergence rate of the sequence ${\left\{\Vert x^n-x^{n-1}\Vert \right\}}_{n\in \mathbb{N}}$, where ${\left\{x^n\right\}}_{n\in \mathbb{N}}$ is generated by SAM and the averaging parameters ${\alpha }_n,n\in \mathbb{N}$, are chosen as follows.

${\alpha }_n=\text{min}\left\{\frac{2\gamma }{n\left(1-\eta \right)},1\right\},n\ge 1,$ (3.14)

where $\gamma \in \left(0,1\right]$. For simplicity, we prove our results under the assumption that $\gamma =1$. It is important to note that all the following results remain valid for any $\gamma$ chosen from the interval $\left(0,1\right]$.

Proposition 3.1. Let ${\left\{x^n\right\}}_{n\in \mathbb{N}},{\left\{y^n\right\}}_{n\in \mathbb{N}}$ and ${\left\{z^n\right\}}_{n\in \mathbb{N}}$ be sequences generated by SAM where ${\left\{{\alpha }_n\right\}}_{n\in \mathbb{N}}$ is defined by (3.14). Then, for any $\tilde{x}\in \text{Fix}\left(T\right)$, the two sequences ${\left\{\Vert x^n-x^{n-1}\Vert \right\}}_{n\in \mathbb{N}}$ and ${\left\{\Vert y^n-x^{n-1}\Vert \right\}}_{n\in \mathbb{N}}$ converge to $0$, and the rates of convergence are given by

$\Vert x^n-x^{n-1}\Vert \le \frac{2C_{\tilde{x}}J}{\left(1-\eta \right)n},n\ge 1,$ (3.15)

and

$\Vert y^n-x^{n-1}\Vert \le \frac{2C_{\tilde{x}}\left(J+2\right)}{\left(1-\eta \right)n},n\ge 1,$ (3.16)

where $C_{\tilde{x}}$ is defined in (3.2), and $J=\lfloor 2/\left(1-\eta \right)\rfloor$.

Proof. From the definitions of $x^n$ and $x^{n+1}$, we directly obtain:

$\begin{array}{c}\Vert x^{n+1}-x^n\Vert =\Vert \left(1-{\alpha }_{n+1}\right)y^{n+1}+{\alpha }_{n+1}{z}^{n+1}-\left(\left(1-{\alpha }_n\right)y^n+{\alpha }_n{z}^n\right)\Vert \\ =\Vert \left(1-{\alpha }_{n+1}\right)\left(y^{n+1}-y^n\right)+{\alpha }_{n+1}\left(z^{n+1}-z^n\right)+\left({\alpha }_n-{\alpha }_{n+1}\right)\left(y^n-z^n\right)\Vert \\ \le \left(1-{\alpha }_{n+1}\right)\Vert y^{n+1}-y^n\Vert +{\alpha }_{n+1}\Vert z^{n+1}-z^n\Vert +\left({\alpha }_n-{\alpha }_{n+1}\right)\Vert y^n-z^n\Vert \\ \le \left(1-{\alpha }_{n+1}\right)\Vert x^n-x^{n-1}\Vert +{\alpha }_{n+1}\eta \Vert x^n-x^{n-1}\Vert +\left({\alpha }_n-{\alpha }_{n+1}\right)\Vert y^n-z^n\Vert \\ =\left(1-{\alpha }_{n+1}\left(1-\eta \right)\right)\Vert x^n-x^{n-1}\Vert +\left({\alpha }_n-{\alpha }_{n+1}\right)\Vert y^n-z^n\Vert ,\end{array}$ (3.17)

where the second inequality follows from (3.10) and (3.11). Now, let $\tilde{x}\in \text{Fix}\left(T\right)$ and let $\tilde{z}=S\left(\tilde{x}\right)$, then

$\begin{array}{c}\Vert y^n-z^n\Vert =\Vert y^n-\tilde{x}+\tilde{x}-\tilde{z}+\tilde{z}-z^n\Vert \\ \le \Vert y^n-\tilde{x}\Vert +\Vert \tilde{x}-\tilde{z}\Vert +\Vert \tilde{z}-z^n\Vert \\ \le \Vert x^{n-1}-\tilde{x}\Vert +\Vert \left(I-S\right)\tilde{x}\Vert +\eta \Vert x^{n-1}-\tilde{x}\Vert \\ \le C_{\tilde{x}}+\left(1-\eta \right)C_{\tilde{x}}+\eta C_{\tilde{x}}\\ =2C_{\tilde{x}},\end{array}$ (3.18)

where the second inequality follows from (3.12) and (3.13), as well as the definition of $\tilde{z}$, and the last inequality follows from Lemma 3.1(i). Additionally, we have

$\begin{array}{c}\Vert x^1-x^0\Vert =\Vert x^1-\tilde{x}+\tilde{x}-x^0\Vert \\ \le \Vert x^1-\tilde{x}\Vert +\Vert x^0-\tilde{x}\Vert \\ \le 2C_{\tilde{x}},\end{array}$ (3.19)

where the second inequality follows from Lemma 3.1(i). Let $a_n:=\Vert x^n-x^{n-1}\Vert$, $b_n:={\alpha }_n,\gamma :=1-\eta$ and $c_n:=\Vert y^n-z^n\Vert$. Then, it means that (3.17) is equal to

$\begin{array}{c}{a}_{n+1}=\Vert x^{n+1}-x^n\Vert \le \left(1-{\alpha }_{n+1}\left(1-\eta \right)\right)\Vert x^n-x^{n-1}\Vert +\left({\alpha }_n-{\alpha }_{n+1}\right)\Vert y^n-z^n\Vert \\ =\left(1-b_{n+1}\gamma \right)a_n+\left(b_n-b_{n+1}\right)c_n.\end{array}$

Note that, $c_n:=\Vert y^n-z^n\Vert$ and combine it with (3.18), we know that $c_n\le 2C_{\tilde{x}}$. Now, set $M:=2C_{\tilde{x}}$. According to Lemma 3.3, we know that

$a_n\le \frac{MJ}{\gamma n}=\frac{2C_{\tilde{x}}J}{\left(1-\eta \right)n},$

it means that we have (3.15). Then the convergence rate for ${\left\{\Vert y^n-x^{n-1}\Vert \right\}}_{n\in \mathbb{N}}$ is derived from the following arguments. Recall that $x^n=\alpha z^n+\left(1-\alpha \right)y^n$, then

$\begin{array}{c}\Vert y^n-x^{n-1}\Vert =\Vert y^n-x^n+x^n-x^{n-1}\Vert \\ \le \Vert y^n-x^n\Vert +\Vert x^n-x^{n-1}\Vert \\ ={\alpha }_n\Vert y^n-z^n\Vert +\Vert x^n-x^{n-1}\Vert \\ \le \frac{2}{\left(1-\eta \right)n}2C_{\tilde{x}}+\frac{2C_{\tilde{x}}J}{\left(1-\eta \right)n}\\ =\frac{2C_{\tilde{x}}\left(J+2\right)}{\left(1-\eta \right)n},\end{array}$

where the second inequality is due to the previous result as well as (3.18).

□

It is no hard to see from (3.16) that the sequence generated by BiG-SAM algorithm converges to an optimal solution of the inner problem (Q) with $\mathcal{O}\left(1/n\right)$. In the following, we will discuss the convergence an important result that is the convergence of ${\left\{H_{\mu }\left(y^n\right)\right\}}_{n\in \mathbb{N}}$. Why not discuss the convergence of ${\left\{H_{\mu }\left(x^n\right)\right\}}_{n\in \mathbb{N}}$ directly? Because of the domain of the function $H_{\mu }$ may not be feasible for $\left\{H_{\mu }\left(x^n\right)\right\}$. However, due to $\Vert y^n-x^{n-1}\Vert \to 0$ as $n\to \infty$ and $H_{\mu }$ is lower semicontinuous, we know that proving convergence of ${\left\{H_{\mu }\left(y^n\right)\right\}}_{n\in \mathbb{N}}$ to the optimal value also means the convergence of ${\left\{H_{\mu }\left(x^n\right)\right\}}_{n\in \mathbb{N}}$ to the same value. The global convergence rate result is as follow.

Theorem 3.1. Let ${\left\{x^n\right\}}_{n\in \mathbb{N}},{\left\{y^n\right\}}_{n\in \mathbb{N}}$ and ${\left\{z^n\right\}}_{n\in \mathbb{N}}$ be sequences generated by BiG-SAM. Let ${\left\{{\alpha }_n\right\}}_{n\in \mathbb{N}}$ be defined by (3.14). Then, for all $\lambda \le 1/\left(L_f+\alpha /\mu \right)$ and $n\in \mathbb{N}$, we have that

$H_{\mu }\left(y^n\right)-H_{\mu }\left(x_{op}^{\text{*}}\right)\le \frac{2C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{\left(n+1\right)\left(1-\eta \right)\lambda },$

where $C_{x_{op}^{\text{*}}}=\text{max}\left\{\Vert x^0-\tilde{x}\Vert ,\frac{1}{1-\eta }\Vert \left(I-S\right)\tilde{x}\Vert \right\}$, and $J=\lfloor 2/\left(1-\eta \right)\rfloor$.

Proof. From Proposition 2.1 we have, for any step-size $\lambda \le 1/\left(L_f+\alpha /\mu \right)$, that the following inequality that holds true,

$H_{\mu }\left(y^{n+1}\right)-H_{\mu }\left(x_{op}^{\text{*}}\right)\le \frac{1}{\lambda }\langle x^n-y^{n+1},x^n-x_{op}^{\text{*}}\rangle -\frac{1}{2\lambda }{\Vert x^n-y^{n+1}\Vert }^2.$ (3.20)

For $x_{op}^{\text{*}}\in X^{\text{*}}=\text{Fix}\left(T_{\lambda }\right)$, from Lemma 3.1(i) and Lemma 3.1, we obtain

$\langle x^n-y^{n+1},x^n-x_{op}^{\text{*}}\rangle \le \Vert x^n-y^{n+1}\Vert \Vert x^n-x_{op}^{\text{*}}\Vert \le \frac{2C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{\left(1-\eta \right)\left(n+1\right)}.$ (3.21)

Substituting (3.21) into (3.20), we get

$H_{\mu }\left(y^{n+1}\right)-H_{\mu }\left(x_{op}^{\text{*}}\right)\le \frac{2C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{\left(n+1\right)\left(1-\eta \right)\lambda }.$ (3.22)

We complete the proof.

□

Now, we will show the complexity result of BiG-SAM algorithm.

Theorem 3.2. Suppose that Assumption I holds. Let $\epsilon \in \left(0,\bar{\epsilon }\right)$ for some fixed $\bar{\epsilon }>0$. Let ${\left\{x^n\right\}}_{n\in \mathbb{N}},{\left\{y^n\right\}}_{n\in \mathbb{N}}$ and ${\left\{z^n\right\}}_{n\in \mathbb{N}}$ be generated by BiG-SAM algorithm with smoothing parameter

$\mu =\sqrt{\frac{\alpha }{\beta }}\frac{\epsilon }{\sqrt{\alpha \beta }+\sqrt{\alpha \beta +L_f\epsilon }}.$

Then for any $n$ satisfying

$n\ge \frac{2{\text{Γ}}^2\left(J+2\right){\left(2\sqrt{\alpha \beta }+\sqrt{L_f\epsilon }\right)}^2}{{\epsilon }^2\left(1-\eta \right)},$

where $\text{Γ}=\text{max}\left\{\Vert R_{\delta }-x^0\Vert ,\frac{1}{1-\eta }\Vert \left(I-S\right)R_{\delta }\Vert \right\}$, and $J=\lfloor 2/\left(1-\eta \right)\rfloor$, it holds that $H\left(y^n\right)-H_{opt}\le \epsilon$.

Proof. Using the $\frac{1}{\mu }$-smooth approximation property of $h_{\mu }$ with parameters $\left(\alpha ,\beta \right)$, it follows that for any $y\in {ℝ}^n$,

$H_{\mu }\left(y\right)\le H\left(y\right)\le H_{\mu }\left(y\right)+\beta \mu .$ (3.23)

In particular, the following two inequalities hold:

$H_{\text{opt}}\ge H_{\mu }\left(x_{op}^{\text{*}}\right)\text{and}H\left(y^n\right)\le H_{\mu }\left(y^n\right)+\beta \mu ,n=0,1,\cdots .$ (3.24)

In which, combined with (3.22) and $\lambda =1/\left(L_f+\alpha /\mu \right)$, it yields

$\begin{array}{c}H\left(y^n\right)-H_{\text{opt}}\le H_{\mu }\left(y^n\right)+\beta \mu -H_{\mu }\left(x_{op}^{\text{*}}\right)\\ \le \frac{2C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{\left(n+1\right)\left(1-\eta \right)\lambda }+\beta \mu \\ =2L_f\frac{C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{\left(n+1\right)\left(1-\eta \right)}+\left(\frac{2\alpha C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{\left(n+1\right)\left(1-\eta \right)}\right)\frac{1}{\mu }+\beta \mu \\ \le 2L_f\frac{C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{n\left(1-\eta \right)}+\left(\frac{2\alpha C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{n\left(1-\eta \right)}\right)\frac{1}{\mu }+\beta \mu ,\end{array}$

where $C_{x_{op}^{\text{*}}}:=\text{max}\left\{\Vert x^0-\tilde{x}\Vert ,\frac{1}{1-\eta }\Vert \left(I-S\right)\tilde{x}\Vert \right\}$, and $J=\lfloor 2/\left(1-\eta \right)\rfloor$. Therefore, for a given $N>0$, it holds that for any $n\ge N$,

$H\left(y^n\right)-H_{\text{opt}}\le 2L_f\frac{C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{N\left(1-\eta \right)}+\left(\frac{2\alpha C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{N\left(1-\eta \right)}\right)\frac{1}{\mu }+\beta \mu .$ (3.25)

Minimizing the right-hand side w.r.t. $\mu$, we obtain

$\mu =\sqrt{\frac{2\alpha C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{N\beta \left(1-\eta \right)}}.$ (3.26)

Plugging (3.26) into (3.25), it implies that for any $n\ge N$,

$H\left(y^n\right)-H_{\text{opt}}\le 2L_f\frac{C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{N\left(1-\eta \right)}+2\sqrt{\frac{2\alpha \beta C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{N\left(1-\eta \right)}}.$

Thus, to make sure that $y^n$ is an $\epsilon$-optimal solution for any $n\ge N$, it is enough that $N$ will satisfy

$2L_f\frac{C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{N\left(1-\eta \right)}+2\sqrt{\frac{2\alpha \beta C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{N\left(1-\eta \right)}}\le \epsilon .$

Setting ${\tau }^2=\frac{2C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{N\left(1-\eta \right)}$, the above inequality reduces to

$L_f{\tau }^2+2\sqrt{\alpha \beta }\tau -\epsilon \le 0,$

which, by the fact that $\tau >0$, is equivalent to

$\sqrt{\frac{2C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{N\left(1-\eta \right)}}=\tau \le \frac{-\sqrt{\alpha \beta }+\sqrt{\alpha \beta +L_f\epsilon }}{L_f}=\frac{\epsilon }{\sqrt{\alpha \beta }+\sqrt{\alpha \beta +L_f\epsilon }}.$

We conclude that $N$ should satisfy

$N\ge \frac{\left(2C_{x_{op}^{\text{*}}}^2\left(J+2\right)\right){\left(\sqrt{\alpha \beta }+\sqrt{\alpha \beta +L_f\epsilon }\right)}^2}{{\epsilon }^2\left(1-\eta \right)}.$

In particular, if we choose

$N=N_1\equiv \frac{\left(2C_{x_{op}^{\text{*}}}^2\left(J+2\right)\right){\left(\sqrt{\alpha \beta }+\sqrt{\alpha \beta +L_f\epsilon }\right)}^2}{{\epsilon }^2\left(1-\eta \right)},$

and $\mu$ according to (3.26), meaning that

$\mu =\sqrt{\frac{2\alpha C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{N_1\beta \left(1-\eta \right)}}=\sqrt{\frac{\alpha }{\beta }}\frac{\epsilon }{\sqrt{\alpha \beta }+\sqrt{\alpha \beta +L_f\epsilon }},$

then for any $n\ge N_1$, it holds that $H\left(y^n\right)-H_{\text{opt}}\le \epsilon$. By (3.23) and (3.24),

$H\left(x_{op}^{\text{*}}\right)-\beta \mu \le H_{\mu }\left(x_{op}^{\text{*}}\right)\le H_{\text{opt}}\le H\left(y^0\right),$

which along with the inequality

$\mu =\sqrt{\frac{\alpha }{\beta }}\frac{\epsilon }{\sqrt{\alpha \beta }+\sqrt{\alpha \beta +L_f\epsilon }}\le \sqrt{\frac{\alpha }{\beta }}\frac{\epsilon }{\sqrt{\alpha \beta }+\sqrt{\alpha \beta }}\le \frac{\bar{\epsilon }}{2\beta },$

implies that $H\left(x_{op}^{\text{*}}\right)\le H\left(y^0\right)+\frac{\bar{\epsilon }}{2}$, and hence, by Assumption I (iv), it follows that $\Vert \tilde{x}\Vert \le R_{\delta }$, where $\delta :=H\left(y^0\right)+\frac{\overline{ϵ}}{2}$. Therefore, $C_{x_{op}^{\text{*}}}\le \text{max}\left\{\Vert R_{\delta }-x^0\Vert ,\frac{1}{1-\eta }\Vert \left(I-S\right)R_{\delta }\Vert \right\}=\text{Γ}$. Consequently,

$\begin{array}{c}{N}_1=\frac{\left(2C_{x_{op}^{\text{*}}}^2\left(J+2\right)\right){\left(\sqrt{\alpha \beta }+\sqrt{\alpha \beta +L_f\epsilon }\right)}^2}{{\epsilon }^2\left(1-\eta \right)}\\ \le \frac{\left(2C_{x_{op}^{\text{*}}}^2\left(J+2\right)\right){\left(2\sqrt{\alpha \beta }+\sqrt{L_f\epsilon }\right)}^2}{{\epsilon }^2\left(1-\eta \right)}\\ \le \frac{2{\text{Γ}}^2\left(J+2\right){\left(2\sqrt{\alpha \beta }+\sqrt{L_f\epsilon }\right)}^2}{{\epsilon }^2\left(1-\eta \right)}\\ \equiv N_2.\end{array}$

The second inequality follows from the fact that for any $\gamma ,\delta \ge 0$, it holds that $\sqrt{\gamma +\delta }\le \sqrt{\gamma }+\sqrt{\delta }$. Consequently, for any $n\ge N_2$, we have $H\left(y^n\right)-H_{\text{opt}}\le \epsilon$, thus establishing the desired result.

□

4. BiG-SAM for Nonsmooth Bi-level Optimization Problems

In this section, we adopt the problem (OP) described in Section 2, where the objective function $\omega$ does not necessarily satisfy Assumption II, which satisfies the Assumption III.

Note that, BiG-SAM cannot be directly applied to bi-level problems with Assumption III. However, we can handle this case indirectly. From the strong convexity of $\omega$, we can smooth $\omega$ by the Moreau envelope $M_{s\omega }$. Recall the properties of Moreau envelope in Section 2, $M_{s\omega }$ is continuously differentiable, with a $1/s$-Lipschitz continuous gradient, $1/s>0$, and is strongly convex (see Lemma 2.3). Thus, $M_{s\omega }$ satisfies Assumption II, it maked BiG-SAM algorithm applicable. In this case, step (3.7) can be simplified as follow:

$\begin{array}{c}{z}^n=x^{n-1}-s\nabla M_{s\omega }\left(x^{n-1}\right)\\ =x^{n-1}-s\frac{1}{s}\left(x^{n-1}-{\text{prox}}_{s\omega }\left(x^{n-1}\right)\right)\\ ={\text{prox}}_{s\omega }\left(x^{n-1}\right),\end{array}$ (4.1)

where the second equality follows from (2.13). This implies that computing $z^n$ ( $n\in \mathbb{N}$) requires evaluating the proximal mapping of $\omega$.

Remark 1. Note that the proximal mapping of a strongly convex function is a contraction ( [12] , Lemma 6), making the theory in Section 3.1 applicable here. A direct consequence of Lemma 3.2 applies to the mappings:

$S\left(x\right)=x-s\nabla M_{s\omega }\left(x\right)\text{and}T\left(x\right)={\text{prox}}_{\lambda g}\left(x-\lambda \nabla F_{\mu }\left(x\right)\right),$

where $s>0$ and $\lambda \in \left(0,1/\left(L_f+\alpha /\mu \right)\right]$.

Lemma 4.1. [12] Let ${\left\{x^n\right\}}_{n\in \mathbb{N}}$ be a sequence generated by BiG-SAM. Under Assumptions I, III and IV, for $s>0$, the sequence ${\left\{x^n\right\}}_{n\in \mathbb{N}}$ converges to $x_s^{\text{*}}\in X^{\text{*}}$, where $x_s^{\text{*}}$ satisfies:

$\langle \nabla M_{s\omega }\left(x_s^{\text{*}}\right),x-x_s^{\text{*}}\rangle \ge 0,\forall x\in X^{\text{*}}.$ (4.2)

Thus, $x_s^{\text{*}}$ is the optimal solution of the problem (OP) with respect to the Moreau envelope $M_{s\omega }$, i.e.,

$x_s^{\text{*}}=\underset{x\in X^{\text{*}}}{\arg \min }{M}_{s\omega }\left(x\right),$

where $X^{\text{*}}$ is the set of optimal solutions of problem (Q).

Smoothing the $\omega$ seems to not affect the convergence rate, which is based on the inner function. From the works in [12] , we know that the convergence rate depends on the contraction parameter $\eta$. We have the following result from ( [12] , Lemma 6),

$\eta =\frac{1}{1+s\sigma }.$

Let $\delta >0$ be the required uniform accuracy in terms of the outer objective function, that is,

$\omega \left(x^n\right)-M_{s\omega }\left(x^n\right)\le \delta ,\forall n\in \mathbb{N}$ (4.3)

where it should be noted that $\omega \left(x^n\right)-M_{s\omega }\left(x^n\right)\ge 0$ for all $n\in \mathbb{N}$. Now, we aim to determine the number of iterations $N^{\prime }$ required to achieve an $\epsilon$-optimal solution for the inner problem, that is,

$H\left(y^{N^{\prime }}\right)-H_{opt}\le \epsilon ,$

while keeping the uniform accuracy as given in (4.3). This means that $N^{\prime }$ depends on both $\epsilon$ and $\delta$.

Proposition 4.1. Let $\epsilon \in \left(0,\bar{\epsilon }\right)$ for some fixed $\bar{\epsilon }>0$. Let ${\left\{x^n\right\}}_{n\in \mathbb{N}}$ and ${\left\{y^n\right\}}_{n\in \mathbb{N}}$ be a sequence generated by BiG-SAM and suppose that Assumptions I, III and IV hold true. In addition, suppose that the smoothing parameter is chosen by

$s=\frac{2\delta }{{\ell }_{\omega }^2}$

and

$\mu =\sqrt{\frac{\alpha }{\beta }}\frac{\epsilon }{\sqrt{\alpha \beta }+\sqrt{\alpha \beta +L_f\epsilon }}.$

Then, (4.3) holds true and for

$n\ge \frac{2{\text{Γ}}^2{\left(2\sqrt{\alpha \beta }+\sqrt{L_f\epsilon }\right)}^2}{{\epsilon }^2}\left(2+\frac{3{\ell }_{\omega }^2}{2\sigma \delta }+\frac{{\ell }_{\omega }^4}{4{\sigma }^2{\delta }^2}\right),$

where $\text{Γ}=\text{max}\left\{\Vert R_{\delta }-x^0\Vert ,\frac{1}{1-\eta }\Vert \left(I-S\right)R_{\delta }\Vert \right\}$, it holds that $H\left(y^n\right)-H_{opt}\le \epsilon$.

Proof. Since $\omega$ is ${\ell }_{\omega }$-Lipschitz continuous (see Assumption III) it follows that the norms of the subgradients of $\omega$ are bounded from above by ${\ell }_{\omega }$. Thus, from ( [15] , Lemma 4.2) it follows, for all $x\in {ℝ}^n$, that

$\omega \left(x\right)-\frac{s{\ell }_{\omega }^2}{2}\le M_{s\omega }\left(x\right)\le \omega \left(x\right)$

Therefore, for $s=2\delta /{\ell }_{\omega }^2$, we obtain that

$\omega \left(x^n\right)-M_{s\omega }\left(x^n\right)\le \delta ,\forall n\in \mathbb{N}.$

Using the $\frac{1}{\mu }$-smooth approximation property of $h_{\mu }$ with parameters $\left(\alpha ,\beta \right)$, it follows that for any $y\in {ℝ}^n$,

$H_{\mu }\left(y\right)\le H\left(y\right)\le H_{\mu }\left(y\right)+\beta \mu .$ (4.4)

In particular, the following two inequalities hold:

$H_{\text{opt}}\ge H_{\mu }\left(x_{op}^{\text{*}}\right)\text{and}H\left(y^n\right)\le H_{\mu }\left(y^n\right)+\beta \mu ,n=0,1,\cdots ,$ (4.5)

which, combined with (3.22), yields

$\begin{array}{c}H\left(y^n\right)-H_{\text{opt}}\le H_{\mu }\left(y^n\right)+\beta \mu -H_{\mu }\left(x_{op}^{\text{*}}\right)\\ \le \frac{2C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{\left(n+1\right)\left(1-\eta \right)\lambda }+\beta \mu \\ =2L_f\frac{C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{\left(n+1\right)\left(1-\eta \right)}+\left(\frac{2\alpha C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{\left(n+1\right)\left(1-\eta \right)}\right)\frac{1}{\mu }+\beta \mu \\ \le 2L_f\frac{C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{n\left(1-\eta \right)}+\left(\frac{2\alpha C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{n\left(1-\eta \right)}\right)\frac{1}{\mu }+\beta \mu ,\end{array}$

where $C_{x_{op}^{\text{*}}}:=\text{max}\left\{\Vert x^0-\tilde{x}\Vert ,\frac{1}{1-\eta }\Vert \left(I-S\right)\tilde{x}\Vert \right\}$, and $J=\lfloor 2/\left(1-\eta \right)\rfloor$. Therefore, for a given $N^{\prime }>0$, it holds that for any $n\ge N^{\prime }$,

$H\left(y^n\right)-H_{\text{opt}}\le 2L_f\frac{C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{N^{\prime }\left(1-\eta \right)}+\left(\frac{2\alpha C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{N^{\prime }\left(1-\eta \right)}\right)\frac{1}{\mu }+\beta \mu .$ (4.6)

Minimizing the right-hand side w.r.t. $\mu$, we obtain

$\mu =\sqrt{\frac{2\alpha C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{N^{\prime }\beta \left(1-\eta \right)}}.$ (4.7)

Plugging the above expression into (4.6), we conclude that for any $n\ge N^{\prime }$,

$H\left(y^n\right)-H_{\text{opt}}\le 2L_f\frac{C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{N^{\prime }\left(1-\eta \right)}+2\sqrt{\frac{2\alpha \beta C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{N^{\prime }\left(1-\eta \right)}}.$

Thus, to guarantee that $y^n$ is an $\epsilon$-optimal solution for any $n\ge N^{\prime }$, it is enough that $N^{\prime }$ will satisfy

$2L_f\frac{C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{N^{\prime }\left(1-\eta \right)}+2\sqrt{\frac{2\alpha \beta C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{N^{\prime }\left(1-\eta \right)}}\le \epsilon .$

Denoting ${\tau }^2=\frac{2C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{N^{\prime }\left(1-\eta \right)}$, the above inequality reduces to

$L_f{\tau }^2+2\sqrt{\alpha \beta }\tau -\epsilon \le 0,$

which, by the fact that $\tau >0$, is equivalent to

$\sqrt{\frac{2C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{N^{\prime }\left(1-\eta \right)}}=\tau \le \frac{-\sqrt{\alpha \beta }+\sqrt{\alpha \beta +L_f\epsilon }}{L_f}=\frac{\epsilon }{\sqrt{\alpha \beta }+\sqrt{\alpha \beta +L_f\epsilon }}.$

We conclude that $N^{\prime }$ should satisfy

$N^{\prime }\ge \frac{\left(2C_{x_{op}^{\text{*}}}^2\left(J+2\right)\right){\left(\sqrt{\alpha \beta }+\sqrt{\alpha \beta +L_f\epsilon }\right)}^2}{{\epsilon }^2\left(1-\eta \right)}.$

In particular, if we choose

$N^{\prime }=N_3\equiv \frac{\left(2C_{x_{op}^{\text{*}}}^2\left(J+2\right)\right){\left(\sqrt{\alpha \beta }+\sqrt{\alpha \beta +L_f\epsilon }\right)}^2}{{\epsilon }^2\left(1-\eta \right)}$ (4.8)

Now, substituting $J=\lfloor 2/\left(1-\eta \right)\rfloor$, $\eta =1/\left(1+s\sigma \right)$, and $s=2\delta /{\ell }_{\omega }^2$ into equation (4.8), we obtain

$\begin{array}{c}{N}^{\prime }=N_3\equiv \frac{\left(2C_{x_{op}^{\text{*}}}^2\left(J+2\right)\right){\left(\sqrt{\alpha \beta }+\sqrt{\alpha \beta +L_f\epsilon }\right)}^2}{{\epsilon }^2\left(1-\eta \right)}\\ =\frac{\left(2C_{x_{op}^{\text{*}}}^2\left(\frac{2}{1-\eta }+2\right)\right){\left(\sqrt{\alpha \beta }+\sqrt{\alpha \beta +L_f\epsilon }\right)}^2}{{\epsilon }^2\left(1-\eta \right)}\\ =\frac{4C_{x_{op}^{\text{*}}}^2{\left(\sqrt{\alpha \beta }+\sqrt{\alpha \beta +L_f\epsilon }\right)}^2\left(2-\eta \right)}{{\epsilon }^2{\left(1-\eta \right)}^2}\\ =\frac{4C_{x_{op}^{\text{*}}}^2{\left(\sqrt{\alpha \beta }+\sqrt{\alpha \beta +L_f\epsilon }\right)}^2}{\epsilon }\left(2+\frac{3}{s\sigma }+\frac{1}{{\left(s\sigma \right)}^2}\right)\\ =\frac{4C_{x_{op}^{\text{*}}}^2{\left(\sqrt{\alpha \beta }+\sqrt{\alpha \beta +L_f\epsilon }\right)}^2}{{\epsilon }^2}\left(2+\frac{3{\ell }_{\omega }^2}{2\sigma \delta }+\frac{{\ell }_{\omega }^4}{4{\sigma }^2{\delta }^2}\right)\end{array}$

and $\mu$ according to (4.7), meaning that

$\mu =\sqrt{\frac{2\alpha C_{x_{op}^{\text{*}}}^2\left(J+2\right)}{N_3\beta \left(1-\eta \right)}}=\sqrt{\frac{\alpha }{\beta }}\frac{\epsilon }{\sqrt{\alpha \beta }+\sqrt{\alpha \beta +L_f\epsilon }},$

then for any $n\ge N_3$ it holds that $H\left(y^n\right)-H_{\text{opt}}\le \epsilon$. By (4.4) and (4.5),

$H\left(x_{op}^{\text{*}}\right)-\beta \mu \le H_{\mu }\left(x_{op}^{\text{*}}\right)\le H_{\text{opt}}\le H\left(y^0\right),$

which along with the inequality

$\mu =\sqrt{\frac{\alpha }{\beta }}\frac{\epsilon }{\sqrt{\alpha \beta }+\sqrt{\alpha \beta +L_f\epsilon }}\le \sqrt{\frac{\alpha }{\beta }}\frac{\epsilon }{\sqrt{\alpha \beta }+\sqrt{\alpha \beta }}\le \frac{\bar{\epsilon }}{2\beta },$

implies that $H\left(x_{op}^{\text{*}}\right)\le H\left(y^0\right)+\frac{\bar{\epsilon }}{2}$, and hence, by Assumption I (iv), it follows that $\tilde{x}\le R_{\delta }$, where $\delta :=H\left(y^0\right)+\frac{\overline{ϵ}}{2}$. Therefore, $C_{x_{op}^{\text{*}}}\le \text{max}\left\{\Vert R_{\delta }-x^0\Vert ,\frac{1}{1-\eta }\Vert \left(I-S\right)R_{\delta }\Vert \right\}=\text{Γ}$. Consequently,

$\begin{array}{c}{N}_3=\frac{4C_{x_{op}^{\text{*}}}^2{\left(\sqrt{\alpha \beta }+\sqrt{\alpha \beta +L_f\epsilon }\right)}^2}{{\epsilon }^2}\left(2+\frac{3{\ell }_{\omega }^2}{2\sigma \delta }+\frac{{\ell }_{\omega }^4}{4{\sigma }^2{\delta }^2}\right)\\ \le \frac{4C_{x_{op}^{\text{*}}}^2{\left(2\sqrt{\alpha \beta }+\sqrt{L_f\epsilon }\right)}^2}{{\epsilon }^2}\left(2+\frac{3{\ell }_{\omega }^2}{2\sigma \delta }+\frac{{\ell }_{\omega }^4}{4{\sigma }^2{\delta }^2}\right)\\ \le \frac{2{\text{Γ}}^2{\left(2\sqrt{\alpha \beta }+\sqrt{L_f\epsilon }\right)}^2}{{\epsilon }^2}\left(2+\frac{3{\ell }_{\omega }^2}{2\sigma \delta }+\frac{{\ell }_{\omega }^4}{4{\sigma }^2{\delta }^2}\right)\\ \equiv N_4.\end{array}$

The second inequality follows from the fact that for any $\gamma ,\delta \ge 0$, it holds that $\sqrt{\gamma +\delta }\le \sqrt{\gamma }+\sqrt{\delta }$. The desired result is achieved by selecting $n$ as the upper bound derived above.

□

5. Conclusion

In this paper, we construct a novel bi-level gradient sequential averaging method (BiG-SAM) for solving a more composite convex bi-level optimization problem, where the inner level problem is to find the optimal solution of the sum of three functions, including two non-smooth function and one smoothable. We analyze the convergence rate of the BiG-SAM in two different cases, where the outer objective is smooth or non-smooth, the global convergence rate with respected to inner objective function is $\mathcal{O}\left(1/n\right)$. In the future, we could further explore the convergence rate and complexity analysis of the outer objective function. Additionally, we could design stochastic and parallel variants of BiG-SAM for high-dimensional data or distributed scenarios. This would help reduce computational complexity while ensuring convergence and scalability of both the outer and inner objectives in distributed environments.

NOTES

*First author.

^#Corresponding author.

References

1. Dempe, S., Dinh, N. and Dutta, J. (2010) Optimality Conditions for a Simple Convex Bilevel Programming Problem. In: Burachik, R. and Yao, J.C., Eds., Variational Analysis and Generalized Differentiation in Optimization and Control, Springer, 149-161. >https://doi.org/10.1007/978-1-4419-0437-9_7

2. Dempe, S. (2002) Foundations of Bilevel Programming. Springer Science&Business Media.

3. Tikhonov, A.N. and Arsenin, V.I. (1977) Solutions of Ill-Posed Problems. Winston&Sons.

4. Mangasarian, O.L. and Meyer, R.R. (1979) Nonlinear Perturbation of Linear Programs. SIAM Journal on Control and Optimization, 17, 745-752. >https://doi.org/10.1137/0317052

5. Ferris, M.C. and Mangasarian, O.L. (1991) Finite Perturbation of Convex Programs. Applied Mathematics & Optimization, 23, 263-273. >https://doi.org/10.1007/bf01442401

6. Solodov, M. (2007) An Explicit Descent Method for Bilevel Convex Optimization. Journal of Convex Analysis, 14, 227-237.

7. Cabot, A. (2005) Proximal Point Algorithm Controlled by a Slowly Vanishing Term: Applications to Hierarchical Minimization. SIAM Journal on Optimization, 15, 555-572. >https://doi.org/10.1137/s105262340343467x

8. Boţ, R.I. and Nguyen, D. (2018) A Forward-Backward Penalty Scheme with Inertial Effects for Monotone Inclusions. Applications to Convex Bilevel Programming. Optimization, 68, 1855-1880. >https://doi.org/10.1080/02331934.2018.1556662

9. Malitsky, Y. (2017). Chambolle-Pock and Tseng’s Methods: Relationship and Extension to the Bilevel Optimization. >https://optimization-online.org/2017/06/6103/

10. Yamada, I., Yukawa, M. and Yamagishi, M. (2011) Minimizing the Moreau Envelope of Nonsmooth Convex Functions over the Fixed Point Set of Certain Quasi-Nonexpansive Mappings. In: Bauschke, H., Burachik, R., Combettes, P., Elser, V., Luke, D. and Wolkowicz, H., Eds., Fixed-Point Algorithms for Inverse Problems in Science and Engineering, Springer, 345-390. >https://doi.org/10.1007/978-1-4419-9569-8_17

11. Beck, A. and Sabach, S. (2013) A First Order Method for Finding Minimal Norm-Like Solutions of Convex Optimization Problems. Mathematical Programming, 147, 25-46. >https://doi.org/10.1007/s10107-013-0708-2

12. Sabach, S. and Shtern, S. (2017) A First Order Method for Solving Convex Bilevel Optimization Problems. SIAM Journal on Optimization, 27, 640-660. >https://doi.org/10.1137/16m105592x

13. Xu, H. (2004) Viscosity Approximation Methods for Nonexpansive Mappings. Journal of Mathematical Analysis and Applications, 298, 279-291. >https://doi.org/10.1016/j.jmaa.2004.04.059

14. Shehu, Y., Vuong, P.T. and Zemkoho, A. (2019) An Inertial Extrapolation Method for Convex Simple Bilevel Optimization. Optimization Methods and Software, 36, 1-19. >https://doi.org/10.1080/10556788.2019.1619729

15. Beck, A. and Teboulle, M. (2012) Smoothing and First Order Methods: A Unified Framework. SIAM Journal on Optimization, 22, 557-580. >https://doi.org/10.1137/100818327

16. Shor, N.Z. (1985) Minimization Methods for Nondifferentiable Functions. Springer-Verlag.

17. Ben-Tal, A. and Teboulle, M. (1989) A Smoothing Technique for Nondifferentiable Optimization Problems. In: Dolecki, S., Ed., Optimization, Springer, 1-11. >https://doi.org/10.1007/bfb0083582

18. Bertsekas, D.P. (1975) Nondifferentiable Optimization via Approximation. In: Balinski, M.L. and Wolfe, P., Eds., Nondifferentiable Optimization, Springer, 1-25. >https://doi.org/10.1007/bfb0120696

19. Bertsekas, D.P. (1982) Constrained Optimization and Lagrange Multiplier Methods. Academic Press.

20. Beck, A. and Teboulle, M. (2009) A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems. SIAM Journal on Imaging Sciences, 2, 183-202. >https://doi.org/10.1137/080716542

21. Nesterov, Y. (2012) Gradient Methods for Minimizing Composite Functions. Mathematical Programming, 140, 125-161. >https://doi.org/10.1007/s10107-012-0629-5

22. Beck, A. and Teboulle, M. (2010) Gradient-Based Algorithms with Applications to Signal-Recovery Problems. Journal of Convex Analysis, 17, 445-477.

23. Combettes, P.L. and Pesquet, J. (2011) Proximal Splitting Methods in Signal Processing. In: Bauschke, H., Burachik, R., Combettes, P., Elser, V., Luke, D. and Wolkowicz, H., Ed., Fixed-Point Algorithms for Inverse Problems in Science and Engineering, Springer, 185-212. >https://doi.org/10.1007/978-1-4419-9569-8_10

24. Bauschke, H.H. and Combettes, P.L. (2019) Correction To: Convex Analysis and Monotone Operator Theory in Hilbert Spaces. In: Bauschke, H.H. and Combettes, P.L., Eds., Convex Analysis and Monotone Operator Theory in Hilbert Spaces, Springer, C1-C4. >https://doi.org/10.1007/978-3-319-48311-5_31