Bilevel Optimization recently became a very active research area. This is mainly due to its important applications from machine learning. In this paper, we give a gentle introduction to algorithms, theory, and applications of bilevel optimization. In particular, we will discuss the history of bilevel optimization, its applications in power grid, hyper-parameter optimization, meta learning, as well as algorithms for solving bilevel optimization and their convergence properties. We will mainly discuss algorithms for solving two types of bilevel optimization problems: lower-level problem is strongly convex and lower-level problem is convex. We will discuss gradient methods and value-function-based methods. Decentralized and federated bilevel optimization will also be discussed.

The Fast Multipole Method (FMM) is a highly efficient numerical algorithm for handling large-scale multi-particle systems, playing an important role in fields such as molecular dynamics, astrodynamics, acoustics, and electromagnetics. This paper first reviews the history of the Fast Multipole Method, then taking Helmholtz and Maxwell equations as examples, introduces the data structures, mathematical principles, implementation steps, and complexity analysis of the FMM based on kernel analytical expansion in two-dimensional and three-dimensional cases, and describes corresponding adaptive version of FMM. Finally, numerical experiments on multi-particle simulations in two-dimensional and three-dimensional spaces are given on the MATLAB platform.

The alternating direction method is one of the classical methods for solving matrix completion problems. Recently, with the rapid development of information, the size of matrices to be processed is very huge. In order to further improve the efficiency of the alternating direction method for solving the large-size matrix completion problems, we borrow the inertia strategy to solve a subproblem of the alternate direction method. Specifically, we obtain the next inertial iteration point by linear combination of the previous iteration point and the previous inertial iteration point of the subproblem. Thus we propose an improved inertial alternating direction method for low rank matrix completion problems in this paper. The convergence analysis of the new algorithm is given under reasonable assumptions. Finally, the superiority of the new algorithm is verified by numerical experimental results of random matrix completion problems and image restoration examples.

For solving large sparse linear equations, based on the idea of block Kaczmarz method, this paper proposes a new random block Kaczmarz algorithm——random greedy residual block Kaczmarz (GREBK(k)) algorithm. Firstly, the K-means clustering algorithm is used to partition the standardized residuals and obtain the corresponding row partitioning strategy in the coefficient matrix, and then solves these equations by the random greedy block Kaczmarz for the above blocked mode. The convergence of this algorithm is proved by relevant theoretical analysis. Finally, numerical experiments show that GREBK(k) algorithm greatly improves the existing relevant results and is an effective numerical method.

The first-order algorithms provide several advantages in tackling large-scale problems, as well as the benefits of simple iteration and little storage. To speed up the convergence, numerous acceleration strategies have been created recently. The gradient method of unconstrained optimization is used as the starting point for this paper, and the common techniques and strategies of the accelerated gradient method are also introduced. These acceleration techniques are further explained in terms of the expressions in the proximal point algorithm, composite optimization problem and stochastic optimization problem. Moreover, this paper provides a summary of additional ways for acceleration strategies using only first-order information and acceleration techniques that are utilized in specific problems.

In this paper, based on the preconditioned modified Hermitian and skew-Hermitian splitting (PMHSS) iteration method, we construct a lopsided variant of the PMHSS iteration method, i.e., LVPMHSS iteration method, to solve the equivalent complex linear system. The convergence conditions of the LVPMHSS iteration method are proposed. By using a special preconditioning matrix, we not only give the detailed theoretical analysis about the spectral properties of the preconditioned matrix, but also obtain the quasi-optimal iterative parameters by minimizing the spectral radius of the iteration matrix. The results of the numerical experiments also illustrate the feasibility and efficiency of the new algorithm.

In this paper, we propose to implement deep neural network to solve the incompressible fluid partial differential equations, the loss function is composed of the equation residual, initial condition and boundary conditions. The sample points are randomly generated at the interior, boundary and initial time as training sets. Compared with traditional numerical methods, the method based on deep neural network is meshfree, and each physical field variable is parallel solved, which is convenient for solving the complex multi-physical field coupling partial differential equations model. Besides, the convergence analysis provides theoretical support for deep neural networks solving partial differential equations. The numerical results show that the method can effectively solve incompressible fluid equations with good accuracy by solving a class of unsteady Stokes equations, a class of viscous Boussinesq equations and a class of Navier-Stokes/Darcy equation.

The Bordered Block Diagonal (BBD) method is a classical approach for solving the largescale sparse linear equation systems generated in transient analysis of circuits simulations. In this paper, a new BBD method is proposed, which improves upon the traditional BBD method by addressing the issue of load imbalance through a combination of basic column decomposition and pipelined decomposition. During the matrix boundary decomposition, the introduction of pipelined decomposition overcomes the difficulty in parallelizing boundaries in traditional methods. By solving the large-scale sparse linear equations generated from 16 real-world circuits, we have verified the effectiveness of the improved BBD method. Compared to the traditional BBD method, the improved method has certain improvements in solution speed with various numbers of parallel threads.

To begin with, a non-negativity-preserving Du Fort-Frankel finite difference method (FDM) is derived for one-dimensional (1D) delayed Fisher's equation. By applying mathematical induction, we can prove that its numerical solutions are all larger than zero as long as $r_{x}=(\varepsilon\Delta t)/h^{2}_{x}\le 1/2$. Here, $\varepsilon$, $\Delta t$ and $h_{x}$ are diffusion coefficient, time-step size and spatial meshsize in $x$-direction, respectively. Secondly, by using cut-off technique to adjust numerical solutions obtained using this non-negativity-preserving Du Fort Frankel FDM, an improved FDM, which can inherit the non-negativity and boundedness of the exact solutions, is designed. Also, by applying mathematical induction, it is shown that its numerical solutions locate in $[0,1]$. By using the discrete energy method, it is shown that both of the proposed algorithms possess the convergence rates of $\mathcal{O}$ $(\Delta t +(\Delta t/h_{x})^{2}+h^{2}_{x})$ in the maximum norm. Thirdly, by using the techniques similar to 1D case, a non-negativity-preserving Du Fort-Frankel FDM and a non-negativity- and boundedness-preserving FDM are developed for two-dimensional Fisher's equation with delay. Also, theoretical findings can be obtained, similarly. Finally, numerical results confirm the exactness of theoretical results, and high efficiency of the proposed methods.

The article consider hybrid high-order methods (HHO) for the p-Laplace problem when 1$ < p < \infty$. The approximation by HHO methods utilizes a reconstruction of the gradients with piecewise Raviart-Thomas finite elements on a regular triangulation without stabilization. Using high-order gradient $\mathbf{Ru_{h}}$ for local gradient reconstruction in piecewise Raviart Thomas finite element space instead of gradient $\mathbf{Dv}$. From the perspective of energy, we perform gradient reconstruction on the minimum value of discrete energy, and determine the discrete stress in a new framework of distance. The main results are the a priori and a posteriori error estimates with global upper bound and global lower bound. Numerical benchmarks display higher convergence rates for the HHO method.

This paper considers the nonseparable nonconvex nonsmooth minimization problem, whose objective function is the sum of a proper lower semicontinuous biconvex function of the entire variables, and two nonconvex functions of their private variables without the global Lipschitz gradient continuity. This paper develops a general inertial alternating structureadapted proximal gradient descent algorithm (GIASAP for short), which not only adopts nonlinear proximal regularization and inertial strategies, but also utilizes constant and dynamical step sizes. The worst case O(1/k) nonasymptotic convergence rate of GIASAP algorithm is established. Furthermore, the bounded sequence generated by GIASAP globally converges to a critical point under the condition that the objective function possesses the Kurdyka-Łojasiewicz property. In addition, numerical results demonstrate the feasibility and effectiveness of the proposed algorithm.

Based on the structures of SR1 method and spectral conjugate gradient method, by using projection operator, we propose a spectral gradient-type derivative-free projection algorithm for solving nonlinear monotone equations with convex constraints. Its search direction satisfies the sufficient descent property which is independent of line search condition. The algorithm converges under some appropriate assumptions. The experimental results show that the algorithm is robust and effective. Finally, the algorithm is used to recover the sparse signal.

Floquet transform is a mathematical tool for studying operators with periodic translation invariance. This paper discusses the basic mathematical properties of quantum eigenvalue problems of periodic systems from this perspective. The Bloch function is obtained by the Floquet transformation, and the Wannier function is defined by the inverse of Floquet transformation. In this process, the operator H(k) is proved for square-integrable functions of periodic units. The self-adjoint and resolvable set compactness, Wannier function as $L^2\left(\mathbb{R}^d\right)$ Orthogonality and completeness of the basis. The continuous differentiability of the isolated energy band with respect to k is also proved. The smoothing of non-isolated band groups is introduced. Finally, based on the Floquet transform of Wannier function, the interpolation calculation of energy band is introduced.

Under the condition that the objective function satisfies the Lojasiewicz property, the rates of convergence of the proximal point algorithm on general manifolds are established. The results are new on Riemannian manifolds and improve the corresponding ones in Euclidean space settings.

In this paper, combined with the relaxation algorithm, we present new tensor splitting methods for solving multilinear PageRank problem. The convergence analysis of the proposed algorithms is also shown. It is shown that the proposed algorithms perform well from some numerical experiments when relaxation parameters are properly selected.

Immersed finite element methods are a group of effective numerical methods for solving interface problems using unfitted meshes. Currently, there are many works on immersed finite element methods for solving interface problems with traditional interface jump conditions. However, there is limited research on interface problems with Robin type jump conditions. In this paper, an immersed finite element method is proposed for solving one-dimensional interface problems with Robin-type jump conditions. The optimal approximation properties and the optimal convergence of the proposed immersed finite element method are proved rigorously. Some numerical examples are provided to validate the theoretical results.

In this paper, we introduce a line search criterion and propose a Bregman proximal gradient algorithm with a inertial term to solve a class of nonconvex composite optimization problem, where the objective functions are the sum of a relatively smooth loss function and a nonsmooth regular function. Under the assumption of the generalized concave Kurdyka-Łojasiewicz (KL) property, the global convergence of the algorithm is proved. The numerical results on image restoration and nonconvex sparse approximation with l_1/2 regularization are reported to demonstrate the effectiveness and superiority of the inertial Bregman proximal gradient algorithm.

Multidimensional scaling (MDS) is a statistical method used to analyze and visualize the similarity or distance relationships between data points. It represents the relative distances or similarities between data points by mapping them to coordinates in a low-dimensional space. The classical solution to the multidimensional scaling problem involves a double centering process on the squared (non-Euclidean) distance matrix, followed by truncating the eigenvalue decomposition to seek a low-dimensional approximate configuration of points. In this paper, we directly fit the squared distance matrix and reformulate the reconstruction problem as a constrained matrix optimization model in the product manifold composed of zero column and column orthogonal matrices and diagonal matrices. By leveraging the geometric properties of the product manifold and incorporating an extended Riemannian gradient descent algorithm based on Zhang-Hager technique, we design a class of adaptive problem models. Numerical experiments demonstrate that direct fitting yields a smaller error in fitting the Euclidean distance matrix. Moreover, the proposed algorithm exhibits certain advantages in terms of iterative efficiency compared to existing projection gradient flow algorithms and first-order and second-order Riemannian algorithms in the Riemannian optimization toolbox.

In this paper, for a class of linear equations with block 2×2 structure, the preconditioning techniques of two kinds of Schur complement matrices and their relations are discussed. We also get a new structure-constrained preconditioner in the derivation process, which possesses both theoretical advantages and computing advantages. By minimizing the spectral clustering of the preconditioned matrices, we obtained two kinds of effective parameter selection strategies and exact eigenvalue distribution of the preconditioned matrices. Also, we proved that under certain special conditions, the preconditioned technologies based on Schur’s complement approximation can be further improved and optimized. At the same time, the effects of these two kinds of Schur approximate matrices and their applications are compared. Finally, a general, reliable and effective preprocessing technique is summarized, which is applied to the three most effective preconditioners at present. Several numerical examples show that the theoretical analysis is convincing, and the effectiveness of the optimized preconditioners are also verified.

Radial basis function neural network (RBFNN) is a method applied to interpolation and classification prediction. In this article, we propose an improved algorithm for the RBFNN, based on the singular value decomposition (SVD) technique, in order to greatly simplify the network structure. In particular, the proposed algorithm is able to automatically choose core neurons in the hidden layer, while deleting redundant ones, which can therefore save the CPU memory and computational cost. Meanwhile, we propose to use the $K$-fold cross validation method to determine the radial parameter $\varepsilon$ in RBF, to keep the algorithm accuracy. More importantly, there is no need to load all the sample data into the CPU memory. Instead, we propose to load and deal with the sample data row by row, based on the approximate SVD algorithm proposed by Halko in [2]. All numerical experiments show that, our proposed algorithm greatly improve the computational efficiency and simplify the RBFNN structure, compared to the traditional RBFNN, while not losing the computational accuracy.

The numerical methods of highly nonlinear stochastic differential equations can be divided into two types: explicit methods and implicit methods. In general, the explicit method has cheap computational cost but the stable property is bad, in contrast, the implicit method has good stable property but computational cost is expensive. In this paper, we present the implicit partially truncated Euler-Maruyama method and prove that it is strongly convergent and stable in mean-square sense. In addition, the obtained results show that the presented method has approximate computational cost and better stable property compared with the explicit partially truncated Euler-Maruyama method for the case that the drift coefficient contains a linear function, that is, it posses concurrently the merit of explicit and implicit methods.

The Multiple-sets Split Feasibility Problem (MSSFP) is an extension of the Split Feasibility Problem and found applications in many practical problems, such as, image reconstruction and phase recovery. Based on selection techniques, Yao et al. [Optimization,2020,69(2): 269-281] proposed two projection algorithms (SPA) for solving MSSFP in Hilbert space. In this paper, we modify the step-size parameter of SPA and present two modified inertial projection algorithms (MISPA) for solving MSSFP. The weak and strong convergence of MISPA are established, respectively, whenever the solution set of MSSFP is nonempty. Numerical experiments are used to show the feasibility of MISPA. Moreover, inertial technique can be used to accelerate SPA.

With the rise of the AI technology revolution represented by ChatGPT, data-center AI research is rapidly emerging. Data analysis techniques including linear separability have received increasing attention from researchers. Linear separability is a fundamental mathematical problem in data analysis, but in the current big data era, an efficient method for testing linear separability is still an unsatisfied demand. This paper proposes and proves a sufficient and necessary condition for the linear separability between a point and a set based on the sphere model; and based on this necessary and sufficient condition, a parallel rapid preliminary screening method for determining the linear separability between two sets is further proposed and demonstrated. The advantages of the method proposed in this paper are: (1) its inherent parallelization properties enable low time complexity in implementation and more efficiency compared to the existing methods; and (2) the universality of the parallel framework. Any method for determining linear separability can be accelerated using the parallel framework described in this paper. The verification experiments based on benchmark data sets and artificial data sets in this paper also fully demonstrate the accuracy of the method of this paper and the efficiency in implementation.

The Two-Step Backward Difference Formula (BDF2) of variable-step in time has exceptional stability, making it an excellent choice for handling stiff problems and multi-scale dynamics issues. However, there is limited research on the optimal control of Partial Differential Equations. This paper introduces a variable-step method to solve the optimal control problem of source term control for a class of reaction-diffusion equations. Specifically, the BDF2 scheme is employed in time, while in space, we utilize the center-difference method for variable-step difference scheme in the L² norm, provided that the ratio of adjacent time-steps falls within the range of $\frac{1}{4.8645}$ to 4.8645. Furthermore, it achieves second-order convergence accuracy in both time and space. Finally, two numerical examples are provided to validate the feasibility and effectiveness of the proposed scheme.

he purpose of this paper is to reconstruct the diffusive viscous wave equation with timevarying sources. The source can be divided into an unknown temporal part and a known spatial part. The unknown part is determined by additional detection values within a nonglobal scope. We propose a source reconstruction method based on the additional detection values and prove the existence and uniqueness of the weak solution. Finally, the theoretical results are verified through numerical examples.

In this paper, we consider the generalized double dimensional inverse eigenvalue problem for a kind of pseudo-Jacobi matrices, which is reconstructed from the eigenvalues of these matrices and their $r$×$r$ leading principle submatrices. The eigenvalue distribution of this kind of matrices is related to the size relationship between the eigenvalues of two complementary principle submatrices. When the size relationship is different, the eigenvalue distribution of this kind of matrices will change greatly. Therefore, the eigenvalue distribution of these matrices is discussed according to the distribution of the root of the secular equation, and the necessary and sufficient conditions for the problem to have a solution are given. Then the problem is solved by equivalently converting such a problem into the $k$ problem proposed by Erxiong Jiang. Finally, two numerical examples are given to verify the effectiveness and feasibility of the proposed algorithm.

In this paper, we consider a numerical approximation for phase field model of nematic liquid crystal and viscous fluids. An equivalent model of the phase field model of nematic liquid crystal and viscous fluid is obtained based on the convex splitting strategy of the Ginzburg-Landau functional. In the numerical scheme, the backward Euler method is used for temporal discretization, and the hybrid finite element method is used for spacial discretization. Here the pressure correction method is used to decouple the computation of the pressure from that of the velocity. Hence, a new first-order scheme is proposed. This proposed scheme is unconditionally stable, as rigorously proven by theoretical analysis. In addition, numerical simulations are given on the temporal convergence rates with different parameters, the spacial convergence rates with different parameters, the evolution of energies, and the annihilation of singularities for the variables d, u, φ. Ample numerical simulations are performed to validate the accuracy and efficiency of the proposed scheme.

This paper considers a special nonconvex optimization problem, namely DC optimization problem, whose objective function can be written as the sum of a smooth convex function, a proper closed convex function and a continuous possibly nonsmooth concave function. This paper develops a general inertial proximal DC algorithm (GIPDCA), which adopts three different extrapolation points for the inertial direction and the gradient center and the proximal center in solving subproblems based on the classical proximal DC algorithm. The proposed GIPDCA can include some classical algorithms as special cases. Under the assumption that the objective function satisfies the Kurdyka-Łojasiewicz property and some suitable conditions on the parameters, we prove that each bounded sequence generated by GIPDCA globally converges to a critical point. In addition, numerical simulations demonstrate the feasibility and effectiveness of the proposed approach.

In this manuscript we proposed using the implicit-explicit splitting method to solve the linear complementarity problem satisfied by American options in financial option pricing problems. Although implicit-explicit methods have been widely used in jump-diffusion models, they are mostly applied in European options, and there is little stability analysis in numerical solutions for American options. In this paper, we proposed that in terms of time, we adopted three discretization methods: the implicit-explicit Backward differential formula of order two (BDF2), the implicit-explicit Crank-Nikolson Leap-Frog(CNLF), and the implicit-explicit Crank-Nikolson AdamBashforth(CNAB), and proved their stability. In space, finite difference discretization is presented, and due to the nonsmoothness of the initial value function, a local mesh refinement strategy is considered near the strike price to improve accuracy. To verify the theoretical results, numerical results for pricing American options under Merton type and Kou type jump-diffusion models were presented. The numerical experimental results show that our proposed method is stable and effective.

In this paper, a monotone coordinate descent algorithm for solving absolute value equations is presented, and the global convergence of the algorithm is analyzed under appropriate conditions. The feasibility and effectiveness of the proposed algorithm are verified by numerical experiments. Another purpose of this paper is to point out a mistake in the paper by Noor et al. [Optim. Lett., 6:1027-1033, 2012], which is caused by misuse of the second-order Taylor expansion in constructing the descending direction of the objective function.

This study focuses on the numerical solutions of the delayed Fisher’s equations by a class of non-negativity-preserving finite difference methods (FDMs) and a kind of maximumprinciple-satisfying FDMs. At first, by using a class of weighted difference formulas and explicit Euler method to discrete the diffusion term and first-order temporal derivative, respectively, a class of non-negativity-preserving FDMs are established for the delayed Fisher’s equations. Secondly, by applying cut-off technique to adjust the numerical solutions obtained by non-negativity-preserving FDMs, a kind of maximum-principle-satisfying FDMs are developed for the delayed Fisher’s equations. Thirdly, by using the non-negativity and boundedness of numerical and exact solutions, the maximum norm error estimations and stabilities for them are given, rigorously. Numerical results confirm the correctness of theoretical findings and the efficiency of the current methods.

In this paper, we established a new non-convex optimization based on L_*-L_F for low Tucker rank tensor completion problem. Three algorithms for solving the new optimization are designed based on the augmented Lagrange multiplier methods. In theory, we analyze the global convergence of the algorithms. In numerical experiment, the simulation data and real image inpainting for the new non-convex optimization and the traditional convex optimization based on nuclear norm are carried out. Experiments results show the new model outperform the nuclear norm model in CPU times under the same precision.

A new high-order well-balanced finite difference scheme based on weighted compact nonlinear scheme (WCNS) is proposed for the Euler equation with gravitational source on the generalized coordinate system. The basic idea is to reconstruct the gravitational source term using steady-state solution, so that it can correspond to the pressure gradient at the lefthand-side of the equations in an equilibrium state. To ensure that the reconstructed value of the conserved variables is exactly equal to the reconstructed value of the steady-state solution in an equilibrium state, a nonlinear interpolation with scale invariance property is used in the reconstruction procedure. Since the same central difference scheme can be used for both flux derivatives and grid derivatives, the proposed scheme satisfy geometric conservation laws on curvilinear grids. By theoretical analysis and experimental results, it is indicated that the proposed WCNS scheme can preserve the general steady state which include both ispthermal and polytropic equilibria, and geometric conservation laws. Moreover, it can achieve fifth-order accuracy and capture exactly small perturbations near steady-state solutions on curvilinear grids.

Multidimensional scaling (MDS) is a data analysis technology that displays and analyzes the corresponding multidimensional data structure in the low-dimensional space. The Individual Difference Scaling (INDSCAL) is a specific model for simultaneous metric multidimensional scaling (MDS) of several data matrices, which not only analyzes the structure of the analysis object, but also takes into account the difference in scales between subjects. In the present work the orthogonal INDSCAL(O-INDSCAL) problem is considered and the problem of fitting the O-INDSCAL model is constructed as a matrix optimization model constrained by Stiefel manifold and linear manifolds. By leveraging the geometric properties of the product manifold, basing on the strong Wolfe line search, we design an adaptive extended hybrid Riemannian conjugate gradient algorithm for the underlying problem and its global convergence is further discussed. Numerical experiments demonstrate that the hybrid method is feasible and effective for the model. Moreover, the proposed algorithm exhibits certain advantages in terms of iterative efficiency compared to the algorithms in the Riemannian optimization toolbox and other Riemannian first-order algorithms.