Bilevel Optimization recently became a very active research area. This is mainly due to its important applications from machine learning. In this paper, we give a gentle introduction to algorithms, theory, and applications of bilevel optimization. In particular, we will discuss the history of bilevel optimization, its applications in power grid, hyper-parameter optimization, meta learning, as well as algorithms for solving bilevel optimization and their convergence properties. We will mainly discuss algorithms for solving two types of bilevel optimization problems: lower-level problem is strongly convex and lower-level problem is convex. We will discuss gradient methods and value-function-based methods. Decentralized and federated bilevel optimization will also be discussed.

Alternating direction method of multipliers(ADMM) is one of the classical algorithms for solving separable convex optimization problems, but it cannot guarantee the convergence of primal iterates and its subproblems can be computationally demanding. In order to ensure convergence and improve computational efficiency, the golden ratio proximal ADMM using convex combination technique is proposed, where the convex combination factor $\psi$ is the key parameter. Based on the golden ratio proximal ADMM, we enlarge parameter $\psi$ and propose an extended golden ratio proximal ADMM(EgrpADMM). Under very mild assumptions, we establish the global convergence and $\mathcal{O}(1/N)$ ergodic sublinear convergence rate in terms of function value residual and constraint violation of EgrpADMM. Furthermore, the algorithm can achieve $\mathcal{O}(1/N^2)$ ergodic convergence when any of the separable subfunctions of the objective function is strongly convex. Finally, we demonstrate the performance of the proposed algorithms via numerical experiments.

The alternating direction method is one of the classical methods for solving matrix completion problems. Recently, with the rapid development of information, the size of matrices to be processed is very huge. In order to further improve the efficiency of the alternating direction method for solving the large-size matrix completion problems, we borrow the inertia strategy to solve a subproblem of the alternate direction method. Specifically, we obtain the next inertial iteration point by linear combination of the previous iteration point and the previous inertial iteration point of the subproblem. Thus we propose an improved inertial alternating direction method for low rank matrix completion problems in this paper. The convergence analysis of the new algorithm is given under reasonable assumptions. Finally, the superiority of the new algorithm is verified by numerical experimental results of random matrix completion problems and image restoration examples.

Horizontal linear complementarity problem (HLCP) is one of the important generalization of the famous linear complementarity problem (LCP). The projected iterative method and the modulus-based matrix splitting iterative method are two recent proposed very effective methods for solving the HLCP. The research in this paper shows that although the deriving principles of these two methods are different, they are equivalent under certain conditions. In particular, when the parameter matrix Ω in the modulus-based matrix splitting iteration methods is taken as a specific positive diagonal matrix, the projected Jacobi method, the projected Gauss-Seidel method and the projected SOR method are equivalent to the modulus-based Jacobi iteration method, the accelerated modulus-based Gauss-Seidel iteration method and the accelerated modulus-based SOR iteration method, respectively. In addition, for the general positive diagonal matrix Ω, the equivalence of these two methods is also studied. Finally, a numerical example is presented to verify the obtained theoretical results.

The first-order algorithms provide several advantages in tackling large-scale problems, as well as the benefits of simple iteration and little storage. To speed up the convergence, numerous acceleration strategies have been created recently. The gradient method of unconstrained optimization is used as the starting point for this paper, and the common techniques and strategies of the accelerated gradient method are also introduced. These acceleration techniques are further explained in terms of the expressions in the proximal point algorithm, composite optimization problem and stochastic optimization problem. Moreover, this paper provides a summary of additional ways for acceleration strategies using only first-order information and acceleration techniques that are utilized in specific problems.

For solving large sparse linear equations, based on the idea of block Kaczmarz method, this paper proposes a new random block Kaczmarz algorithm——random greedy residual block Kaczmarz (GREBK(k)) algorithm. Firstly, the K-means clustering algorithm is used to partition the standardized residuals and obtain the corresponding row partitioning strategy in the coefficient matrix, and then solves these equations by the random greedy block Kaczmarz for the above blocked mode. The convergence of this algorithm is proved by relevant theoretical analysis. Finally, numerical experiments show that GREBK(k) algorithm greatly improves the existing relevant results and is an effective numerical method.

In this paper, based on the preconditioned modified Hermitian and skew-Hermitian splitting (PMHSS) iteration method, we construct a lopsided variant of the PMHSS iteration method, i.e., LVPMHSS iteration method, to solve the equivalent complex linear system. The convergence conditions of the LVPMHSS iteration method are proposed. By using a special preconditioning matrix, we not only give the detailed theoretical analysis about the spectral properties of the preconditioned matrix, but also obtain the quasi-optimal iterative parameters by minimizing the spectral radius of the iteration matrix. The results of the numerical experiments also illustrate the feasibility and efficiency of the new algorithm.

In this paper, we propose to implement deep neural network to solve the incompressible fluid partial differential equations, the loss function is composed of the equation residual, initial condition and boundary conditions. The sample points are randomly generated at the interior, boundary and initial time as training sets. Compared with traditional numerical methods, the method based on deep neural network is meshfree, and each physical field variable is parallel solved, which is convenient for solving the complex multi-physical field coupling partial differential equations model. Besides, the convergence analysis provides theoretical support for deep neural networks solving partial differential equations. The numerical results show that the method can effectively solve incompressible fluid equations with good accuracy by solving a class of unsteady Stokes equations, a class of viscous Boussinesq equations and a class of Navier-Stokes/Darcy equation.

To begin with, a non-negativity-preserving Du Fort-Frankel finite difference method (FDM) is derived for one-dimensional (1D) delayed Fisher's equation. By applying mathematical induction, we can prove that its numerical solutions are all larger than zero as long as $r_{x}=(\varepsilon\Delta t)/h^{2}_{x}\le 1/2$. Here, $\varepsilon$, $\Delta t$ and $h_{x}$ are diffusion coefficient, time-step size and spatial meshsize in $x$-direction, respectively. Secondly, by using cut-off technique to adjust numerical solutions obtained using this non-negativity-preserving Du Fort Frankel FDM, an improved FDM, which can inherit the non-negativity and boundedness of the exact solutions, is designed. Also, by applying mathematical induction, it is shown that its numerical solutions locate in $[0,1]$. By using the discrete energy method, it is shown that both of the proposed algorithms possess the convergence rates of $\mathcal{O}$ $(\Delta t +(\Delta t/h_{x})^{2}+h^{2}_{x})$ in the maximum norm. Thirdly, by using the techniques similar to 1D case, a non-negativity-preserving Du Fort-Frankel FDM and a non-negativity- and boundedness-preserving FDM are developed for two-dimensional Fisher's equation with delay. Also, theoretical findings can be obtained, similarly. Finally, numerical results confirm the exactness of theoretical results, and high efficiency of the proposed methods.

Based on the structures of SR1 method and spectral conjugate gradient method, by using projection operator, we propose a spectral gradient-type derivative-free projection algorithm for solving nonlinear monotone equations with convex constraints. Its search direction satisfies the sufficient descent property which is independent of line search condition. The algorithm converges under some appropriate assumptions. The experimental results show that the algorithm is robust and effective. Finally, the algorithm is used to recover the sparse signal.

Floquet transform is a mathematical tool for studying operators with periodic translation invariance. This paper discusses the basic mathematical properties of quantum eigenvalue problems of periodic systems from this perspective. The Bloch function is obtained by the Floquet transformation, and the Wannier function is defined by the inverse of Floquet transformation. In this process, the operator H(k) is proved for square-integrable functions of periodic units. The self-adjoint and resolvable set compactness, Wannier function as $L^2\left(\mathbb{R}^d\right)$ Orthogonality and completeness of the basis. The continuous differentiability of the isolated energy band with respect to k is also proved. The smoothing of non-isolated band groups is introduced. Finally, based on the Floquet transform of Wannier function, the interpolation calculation of energy band is introduced.

In this paper, combined with the relaxation algorithm, we present new tensor splitting methods for solving multilinear PageRank problem. The convergence analysis of the proposed algorithms is also shown. It is shown that the proposed algorithms perform well from some numerical experiments when relaxation parameters are properly selected.

Under the condition that the objective function satisfies the Lojasiewicz property, the rates of convergence of the proximal point algorithm on general manifolds are established. The results are new on Riemannian manifolds and improve the corresponding ones in Euclidean space settings.

In this paper, the scattering problem of multiple obstacles under the sea surface in twodimensional space is studied theoretically and numerically. By analyzing the characteristics of the scattering problem, using the Helmholtz equation, and combining different boundary conditions and radiation conditions, the mathematical model is established, and the uniqueness of the scattering problem is proved. Based on the potential theory and the indirect integral equation method, the integral representation of the fields in different regions and the integral boundary equation of the density function on the boundary is obtained. By introducing potential operator, the integral domain is truncated, and the operator equation on the bounded domain is obtained. For the established boundary integral equation system, the numerical scheme is constructed using the Nyström method, and the convergence of the numerical solution is proved. Finally, numerical experiments are used to verify the correctness and effectiveness of the theory. Furthermore, numerical experiments are designed to analyze the effects of different parameters on the scattering problem.

In this article, the backward Euler (BE) fully discrete finite element method of the economical finite difference streamlined diffusion (EFDSD) method for nonlinear convection-dominated diffusion equation is mainly investigated and the superconvergence of order $O(h^2+\tau)$ in $H^1$ norm is derived without the restriction between the time step $\tau$ and the mesh size $h$. Firstly, a time discrete system is established to split the error into two parts, which are the temporal error and spatial error, and with the help of mathematical induction, the regularity of the time discrete system is reduced by the temporal error. Then the finite element solution in $W^{0, \infty}$ norm is bounded by the spatial error and the unconditional superclose and global superconvergence results are gained in $H^1$ norm through interpolation post-processing technique. Lastly, a numerical example is provided to verify the correctness of the theoretical analysis and the effectiveness of the method.

This paper considers the nonseparable nonconvex nonsmooth minimization problem, whose objective function is the sum of a proper lower semicontinuous biconvex function of the entire variables, and two nonconvex functions of their private variables without the global Lipschitz gradient continuity. This paper develops a general inertial alternating structureadapted proximal gradient descent algorithm (GIASAP for short), which not only adopts nonlinear proximal regularization and inertial strategies, but also utilizes constant and dynamical step sizes. The worst case O(1/k) nonasymptotic convergence rate of GIASAP algorithm is established. Furthermore, the bounded sequence generated by GIASAP globally converges to a critical point under the condition that the objective function possesses the Kurdyka-Łojasiewicz property. In addition, numerical results demonstrate the feasibility and effectiveness of the proposed algorithm.

A Lagrange quadratic finite volume method scheme for solving the Stokes equation is constructed on triangular meshes in this paper. The piecewise continuous quadratic finite element space and the discontinuous linear finite element space is used as the trial space for velocity and pressure of the Stokes equation respectively, so that the discrete velocity solution of the finite volume method satisfies the local mass conservation on the macro-element triangular element, and the finite element space pair is naturally satisfied with the so-called inf-sup condition. By adopting the special dual partition and the special mapping, the finite volume method scheme for solving the Stokes equation is transformed into the corresponding finite element method. The unconditional stability (or inf-sup condition) of the finite volume method scheme (without the geometric constraints of the triangular meshes) and the optimal-order error estimates in the $\mathbf{H}^1$-norm for velocity are obtained. Finally, numerical experiments show the validity of the theoretical results and the effectiveness of the finite volume method in the numerical simulation of computational fluid dynamics.

In this paper, for a class of linear equations with block 2×2 structure, the preconditioning techniques of two kinds of Schur complement matrices and their relations are discussed. We also get a new structure-constrained preconditioner in the derivation process, which possesses both theoretical advantages and computing advantages. By minimizing the spectral clustering of the preconditioned matrices, we obtained two kinds of effective parameter selection strategies and exact eigenvalue distribution of the preconditioned matrices. Also, we proved that under certain special conditions, the preconditioned technologies based on Schur’s complement approximation can be further improved and optimized. At the same time, the effects of these two kinds of Schur approximate matrices and their applications are compared. Finally, a general, reliable and effective preprocessing technique is summarized, which is applied to the three most effective preconditioners at present. Several numerical examples show that the theoretical analysis is convincing, and the effectiveness of the optimized preconditioners are also verified.

A two-layer nonlinear finite volume scheme for 2D unstationary diffusion equations is constructed on deformed meshes, which based on a two-point nonlinear discrete scheme of continuous diffusion flux. The scheme uses the idea of Crank-Nicolson (C-N) method to achieve second-order accuracy for time evolution. Since the transpose of the resulting algebraic system of coefficient matrix is an M-matrix, it is guaranteed that the scheme preserves positivity. The existence of discrete solution for the present scheme is proved by using Brouwer fixed-point theorem. Numerical results illustrate that the scheme has secondorder accuracy with a larger time step.

Multidimensional scaling (MDS) is a statistical method used to analyze and visualize the similarity or distance relationships between data points. It represents the relative distances or similarities between data points by mapping them to coordinates in a low-dimensional space. The classical solution to the multidimensional scaling problem involves a double centering process on the squared (non-Euclidean) distance matrix, followed by truncating the eigenvalue decomposition to seek a low-dimensional approximate configuration of points. In this paper, we directly fit the squared distance matrix and reformulate the reconstruction problem as a constrained matrix optimization model in the product manifold composed of zero column and column orthogonal matrices and diagonal matrices. By leveraging the geometric properties of the product manifold and incorporating an extended Riemannian gradient descent algorithm based on Zhang-Hager technique, we design a class of adaptive problem models. Numerical experiments demonstrate that direct fitting yields a smaller error in fitting the Euclidean distance matrix. Moreover, the proposed algorithm exhibits certain advantages in terms of iterative efficiency compared to existing projection gradient flow algorithms and first-order and second-order Riemannian algorithms in the Riemannian optimization toolbox.

In this paper, we introduce a line search criterion and propose a Bregman proximal gradient algorithm with a inertial term to solve a class of nonconvex composite optimization problem, where the objective functions are the sum of a relatively smooth loss function and a nonsmooth regular function. Under the assumption of the generalized concave Kurdyka-Łojasiewicz (KL) property, the global convergence of the algorithm is proved. The numerical results on image restoration and nonconvex sparse approximation with l_1/2 regularization are reported to demonstrate the effectiveness and superiority of the inertial Bregman proximal gradient algorithm.

Immersed finite element methods are a group of effective numerical methods for solving interface problems using unfitted meshes. Currently, there are many works on immersed finite element methods for solving interface problems with traditional interface jump conditions. However, there is limited research on interface problems with Robin type jump conditions. In this paper, an immersed finite element method is proposed for solving one-dimensional interface problems with Robin-type jump conditions. The optimal approximation properties and the optimal convergence of the proposed immersed finite element method are proved rigorously. Some numerical examples are provided to validate the theoretical results.

The fast multipole method (FMM) can accelerate the iterative solver of the large dense linear equations arising from many physical problems. This article is concerned with the convergence of the FMM for three dimensional potential problems. Firstly, derive the expression of the global error, and then give a novel estimate of the error bound. Secondly, the result is applied to the adaptive octree structure, and the specific convergence order is obtained. Finally, an illustrative example is provided to test the proposed results. The method of this paper can also be used to estimate the error of the FMM for elastostatic problems and Stokes flow problems.

The Bordered Block Diagonal (BBD) method is a classical approach for solving the largescale sparse linear equation systems generated in transient analysis of circuits simulations. In this paper, a new BBD method is proposed, which improves upon the traditional BBD method by addressing the issue of load imbalance through a combination of basic column decomposition and pipelined decomposition. During the matrix boundary decomposition, the introduction of pipelined decomposition overcomes the difficulty in parallelizing boundaries in traditional methods. By solving the large-scale sparse linear equations generated from 16 real-world circuits, we have verified the effectiveness of the improved BBD method. Compared to the traditional BBD method, the improved method has certain improvements in solution speed with various numbers of parallel threads.

The article consider hybrid high-order methods (HHO) for the p-Laplace problem when 1$ < p < \infty$. The approximation by HHO methods utilizes a reconstruction of the gradients with piecewise Raviart-Thomas finite elements on a regular triangulation without stabilization. Using high-order gradient $\mathbf{Ru_{h}}$ for local gradient reconstruction in piecewise Raviart Thomas finite element space instead of gradient $\mathbf{Dv}$. From the perspective of energy, we perform gradient reconstruction on the minimum value of discrete energy, and determine the discrete stress in a new framework of distance. The main results are the a priori and a posteriori error estimates with global upper bound and global lower bound. Numerical benchmarks display higher convergence rates for the HHO method.

The numerical methods of highly nonlinear stochastic differential equations can be divided into two types: explicit methods and implicit methods. In general, the explicit method has cheap computational cost but the stable property is bad, in contrast, the implicit method has good stable property but computational cost is expensive. In this paper, we present the implicit partially truncated Euler-Maruyama method and prove that it is strongly convergent and stable in mean-square sense. In addition, the obtained results show that the presented method has approximate computational cost and better stable property compared with the explicit partially truncated Euler-Maruyama method for the case that the drift coefficient contains a linear function, that is, it posses concurrently the merit of explicit and implicit methods.

The Multiple-sets Split Feasibility Problem (MSSFP) is an extension of the Split Feasibility Problem and found applications in many practical problems, such as, image reconstruction and phase recovery. Based on selection techniques, Yao et al. [Optimization,2020,69(2): 269-281] proposed two projection algorithms (SPA) for solving MSSFP in Hilbert space. In this paper, we modify the step-size parameter of SPA and present two modified inertial projection algorithms (MISPA) for solving MSSFP. The weak and strong convergence of MISPA are established, respectively, whenever the solution set of MSSFP is nonempty. Numerical experiments are used to show the feasibility of MISPA. Moreover, inertial technique can be used to accelerate SPA.

In this paper, we consider the generalized double dimensional inverse eigenvalue problem for a kind of pseudo-Jacobi matrices, which is reconstructed from the eigenvalues of these matrices and their $r$×$r$ leading principle submatrices. The eigenvalue distribution of this kind of matrices is related to the size relationship between the eigenvalues of two complementary principle submatrices. When the size relationship is different, the eigenvalue distribution of this kind of matrices will change greatly. Therefore, the eigenvalue distribution of these matrices is discussed according to the distribution of the root of the secular equation, and the necessary and sufficient conditions for the problem to have a solution are given. Then the problem is solved by equivalently converting such a problem into the $k$ problem proposed by Erxiong Jiang. Finally, two numerical examples are given to verify the effectiveness and feasibility of the proposed algorithm.

Radial basis function neural network (RBFNN) is a method applied to interpolation and classification prediction. In this article, we propose an improved algorithm for the RBFNN, based on the singular value decomposition (SVD) technique, in order to greatly simplify the network structure. In particular, the proposed algorithm is able to automatically choose core neurons in the hidden layer, while deleting redundant ones, which can therefore save the CPU memory and computational cost. Meanwhile, we propose to use the $K$-fold cross validation method to determine the radial parameter $\varepsilon$ in RBF, to keep the algorithm accuracy. More importantly, there is no need to load all the sample data into the CPU memory. Instead, we propose to load and deal with the sample data row by row, based on the approximate SVD algorithm proposed by Halko in [2]. All numerical experiments show that, our proposed algorithm greatly improve the computational efficiency and simplify the RBFNN structure, compared to the traditional RBFNN, while not losing the computational accuracy.

he purpose of this paper is to reconstruct the diffusive viscous wave equation with timevarying sources. The source can be divided into an unknown temporal part and a known spatial part. The unknown part is determined by additional detection values within a nonglobal scope. We propose a source reconstruction method based on the additional detection values and prove the existence and uniqueness of the weak solution. Finally, the theoretical results are verified through numerical examples.

With the rise of the AI technology revolution represented by ChatGPT, data-center AI research is rapidly emerging. Data analysis techniques including linear separability have received increasing attention from researchers. Linear separability is a fundamental mathematical problem in data analysis, but in the current big data era, an efficient method for testing linear separability is still an unsatisfied demand. This paper proposes and proves a sufficient and necessary condition for the linear separability between a point and a set based on the sphere model; and based on this necessary and sufficient condition, a parallel rapid preliminary screening method for determining the linear separability between two sets is further proposed and demonstrated. The advantages of the method proposed in this paper are: (1) its inherent parallelization properties enable low time complexity in implementation and more efficiency compared to the existing methods; and (2) the universality of the parallel framework. Any method for determining linear separability can be accelerated using the parallel framework described in this paper. The verification experiments based on benchmark data sets and artificial data sets in this paper also fully demonstrate the accuracy of the method of this paper and the efficiency in implementation.