Combining Pontryagin's Principle and Dynamic Programming for Linear and Nonlinear Systems

To study optimal control and disturbance attenuation problems, two prominent—and somewhat alternative—strategies have emerged in the last century: dynamic programming (DP) and Pontryagin's minimum principle (PMP). The former characterizes the solution by shaping the dynamics in a closed loop (a priori unknown) via the selection of a feedback input, at the price, however, of the solution to (typically daunting) partial differential equations. The latter, instead, provides (extended) dynamics that must be satisfied by the optimal process, for which boundary conditions (a priori unknown) should be determined. The results discussed in this article combine the two approaches by matching the corresponding trajectories, i.e., combining the underlying sources of information: knowledge of the complete initial condition from DP and of the optimal dynamics from PMP. The proposed approach provides insights for linear as well as nonlinear systems. In the case of linear systems, the derived conditions lead to matrix algebraic equations, similar to the classic algebraic Riccati equations (AREs), although with coefficients defined as polynomial functions of the input gain matrix, with the property that the coefficient of the quadratic term of such equation is sign definite, even if the corresponding coefficient of the original ARE is sign indefinite, as it is typically the case in the $\mathcal {H}_{\infty }$ control problem. This feature is particularly appealing from the computational point of view, since it permits the use of standard minimization techniques for convex functions, such as the gradient algorithm. In the presence of nonlinear dynamics, the strategy leads to algebraic equations that allow us to (locally) construct the optimal feedback by considering the behavior of the closed-loop dynamics at a single point in the state space.


I. INTRODUCTION
I T HAS been widely recognized that the task of imposing the property of asymptotic stability to an equilibrium point of a controlled process-namely, the property that all the trajectories of a dynamical model ensuing from perturbations of the equilibrium condition asymptotically return to the desired steady-state behavior without moving too far away from it-is probably the cornerstone of control theory [1]. It is then not surprising that the ability of carrying out such a task in an optimal or robust way has attracted widespread interest [2]- [8].
To solve the above control problems two main, somewhat antagonistic, strategies have been pursued since their almost simultaneous definition in the mid of the last century, namely, dynamic programming (DP) (see, e.g., [3] and [9]) and Pontryagin's minimum (or maximum) principle (PMP) [10]. The former approach characterizes the optimal feedback in terms of the solution to a certain quadratic partial differential equation (PDE), i.e., the so-called Hamilton-Jacobi-Bellman (HJB) equation [5]. Methods inspired by DP are particularly appealing, since they provide necessary and sufficient conditions for optimality and permit the characterization of the optimal solution, as well as of the optimal cost, for any initial condition in the state space. However, since a closed-form expression for the solution to the underlying PDE can be seldom computed in practical applications, techniques to determine an approximate solution to the HJB equation have been the objective of several studies in the last decades (see, e.g., [11]- [18]). Along a different line to circumvent the computational issues of the HJB equation, an alternative (weaker) notion of solution has been proposed in the case of nondifferentiable value functions (see, e.g., [19] and [20]), introducing the notion of viscosity solution of the HJB equation.
Nonetheless, despite the intense research effort focused on techniques based on DP, the vast majority of practical implementations of optimal control laws are instead tackled via approaches inspired by the PMP. These provide, in general, only necessary conditions for optimality-hence, they are employed merely to identify candidate optimal solutions or extremalsand, additionally, characterize the (open-loop) solution typically for a specific initial condition. Therefore, the widespread use of such strategies is essentially motivated by the simplicity of the underlying conditions, provided in terms of ordinary differential equations (instead of PDEs) that should be satisfied by the optimal process together with an auxiliary variable (the costate). However, in the case of infinite-horizon problems determining boundary conditions for such a dynamical system does not represent a straightforward task, while, at the same time, avoiding in general an explicit solution to the HJB equation. The latter consequently seems to constitute a bottleneck also for the application of methods based on PMP in the specific scenario of infinite-horizon problems.
As far as robust stabilizing control laws are concerned-hence solving what is typically referred to as the H ∞ control problem in the case of linear as well as of nonlinear systems-challenges and approaches similar to those related to optimal control problems have been identified, essentially with an identical discussion about advantages and drawbacks of each strategy (see, e.g., [7], [21]- [24]). Differently from the specialization of optimal control problems to the case of linear systems and quadratic cost functionals, namely the so-called linear-quadratic regulator (LQR) problem, it is worth mentioning that in the case of the linear H ∞ control problem, an additional challenge is represented by the fact that the (matrix) coefficient of the quadratic term in the underlying algebraic Riccati equation (ARE) is sign indefinite whenever the controlled input and disturbance are not matched [25]. As a consequence, numerical techniques based, for instance, on Newton's iterations, which are extremely popular in LQR problems [26], are not straightforwardly viable in the context of disturbance attenuation.
It is worth mentioning that the developments in the theories of DP and PMP have crossed paths several times in the past. Such a long-standing interaction has consisted, on one hand, in employing the former approach to provide a more concise proof of the latter strategy (see, for instance, [6,Sec. 1.6], [27], [28,Sec. 7], and the references therein). On the other hand-and on a more interesting note-the so-called sensitivity relations have been established, showing that the costate arc obtained by PMP is related to the value function of DP, also in the settings in which the latter is not continuously differentiable (see, e.g., [6,Sec. 12.5] and [29]- [31]).
The main contribution of this article consists of combining the ideas of DP with those of PMP by condensing knowledge derived from each approach into unified conditions both for infinite-horizon optimal control and for H ∞ (or disturbance attenuation) problems. Interestingly, the proposed results bring together ideas inspired by the theory concerning the sensitivity relations mentioned above with structural conditions on the data of the problem, such as real analyticity of the underlying value function. As a consequence, the novel conditions yield additional value and insight in the linear as well as in the nonlinear scenarios for the two above problems. In the former setting, the conditions are derived in terms of algebraic matrix equations with two prominent features: these equations are quadratic in the matrix that defines the value function (similarly to the classic ARE), for which additional constraints are required such as positivity and symmetry, while polynomial in the input gain matrix, which is, however, not additionally constrained; moreover, the equations possess a sign-definite quadratic term in the former variable regardless of the sign of the corresponding term in the underlying "classic" Riccati equation. In the nonlinear setting, instead, the conditions permit the construction or the approximation, via finite truncation of a power series, of the optimal solution by considering only algebraic conditions, rather than PDEs, at a single point in the state space.
The rest of this article is organized as follows. The main ideas are first discussed by revisiting the LQR problem, namely, optimal control in the presence of linear dynamics and quadratic cost functional, in Section II. Similar constructions are then extended in Section III to the disturbance attenuation task, namely, the so-called H ∞ control problem, for which the advantages of deriving alternative conditions that hinge upon a sign-definite algebraic equation are particularly appealing from the computational point of view. This aspect is exploited in Section IV, in which a gradient-based algorithm defined in the Riemannian manifold of symmetric positive-definite matrices is proposed and discussed. Finally, the derivations are extended to the case of nonlinear dynamics in Section V. In this case, the specialization of the strategy that suggests to combine DP and PMP allows deriving algebraic conditions that permit the construction (or the approximation) of the optimal solution, without the need for the solution of any PDE.
Notation: N defines the set of positive integers, N 0 = N ∪ {0}, and R 0 (R >0 ) defines the set of nonnegative (positive) real numbers. σ(A) denotes the spectrum of the matrix A ∈ R n×n . Given a symmetric matrix M = M , M 0, denotes a positive-semidefinite matrix, namely, such that v Mv 0 for any vector v; similarly, M 0 denotes a positive-definite matrix. I n denotes the identity matrix of dimension n. Given two vectors x 1 ∈ R n 1 and x 2 ∈ R n 2 , col(x 1 , x 2 ) denotes the vector [ x 1 x 2 ] . For a vector x = col(x 1 , x 2 ) ∈ R n 1 +n 2 , define the projection operator Π x i : R n 1 +n 2 → R n i as Π x i (x) = x i . For a continuously differentiable function f : R n → R, ∇f denotes the column vector of the partial derivatives of f , namely, ∇f = [ ∂f (x)/∂x 1 , . . . , ∂f(x)/∂x n ] . Given an ordinary differential equationẋ = f (x) with boundary condition x(0) = x 0 and the vector field f : R n → R n sufficiently smooth, the mapping ϕ f : R × R n → R n denotes the corresponding flow such that ∂ϕ f (t; x 0 )/∂t = f (ϕ f (t; x 0 )) for any t 0 and x 0 . The notation C κ indicates the space of functions that possess continuous derivatives of order less than or equal to κ.

II. REVISITING THE LQR PROBLEM
The aim of this section consists of providing an alternative characterization of the solution to the classic LQR problem, by combining information provided by the trajectories of the associated Hamiltonian dynamics and by those of the closed-loop (optimal) plant. To this end, consider a linear time-invariant system described by the equatioṅ with x(t) ∈ R n , x(0) = x 0 , and u(t) ∈ R m denoting the state and the control input of the plant (1), respectively, together with the quadratic cost functional with Q = Q 0 and R = R 0 constant matrices of appropriate dimensions. The following standing assumption holds throughout this article.
Assumption 1: The pair (A, B) is reachable and the pair (A, C), with C such that Q = C C, is observable. Moreover, the matrix B ∈ R n×m is full column rank.
• By relying on arguments based on DP [3], it has been shown (see, e.g., [2], [4], [5], [32], and references therein) that, provided that Assumption 1 holds, the unique feedback input that minimizes the cost functional (2) while stabilizing the origin of the closed-loop system is given by 0 the unique positive-definite solution of the ARE The main result of this section yields an alternative characterization of such a unique solution, which may prove to be particularly useful in the case of the H ∞ control problem, as explained below in detail. To this end, let denote the Hamiltonian matrix naturally associated 1 with the Riccati equation (3) and define the extended state z(t) = col(x(t), λ(t)) ∈ R 2n , with λ denoting the costate. Define then the function Υ : R × R n → R n , parameterized with respect to P and K, where Π x is the projection operator introduced above. The following statement provides a fixed-point 2 characterization of the solution of the LQR problem, with respect to the function Υ introduced in (5). Theorem 1: Consider the LQR problem described by the system (1) and the cost functional (2), and suppose that Assumption 1 holds. Then, the pair of matrices P = (P ) and K yields the optimal solution to the problem if and only if: for all x 0 ∈ R n and all t 0, i.e., any x 0 ∈ R n is a fixed point of the function Υ P ,K (t, ·) for any t ∈ R 0 . Proof of Theorem 1: (⇒) By PMP (see, e.g., [33] for an infinite-horizon formulation), the optimal process col(x (t), λ(t)) satisfies for any t 0 and for any x 0 ∈ R n , where H is the Hamiltonian matrix defined in (4) and P denotes the maximal solution of the ARE (3). Therefore, by letting ϕ (A+BK)x (t; x 0 ) denote the flow of the system (1) in closed loop with u = Kx, namely, ϕ (A+BK)x (t; x 0 ) = e (A+BK)t x 0 , it follows, by uniqueness of the optimal trajectory, that the composition of the flows satisfies for all t 0, which implies necessity of item (ii) of the statement by recalling the definition of the function Υ P,K in (5). Necessity of item (i), instead, is straightforward from Assumption 1, which implies that the optimal feedback is stabilizing.
(⇐) Note that the backward evolution of the Hamiltonian system ϕ Hz restricted to an invariant subspace of the form V {(x, λ) ∈ R n × R n : λ = P x} may be replicated by the evolution of the closed-loop system ϕ (A+BK)x (t; ·) in several ways by incorporating (via the selection of different matrices K) an internal model of any 3 selection of n eigenvalues of the matrix H ∈ R 2n×2n , namely, such that σ(A + BK) = σ(H| V ). Therefore, sufficiency follows instead by relying on item (i) and by the fact that the requirements in Assumption 1 and [7, Lemma 3] imply that the Hamiltonian matrix H ∈ dom(Ric) 4 , hence guaranteeing that H possesses precisely n eigenvalues with positive and negative real parts, respectively; hence, items (i) and (ii) together univocally identify the optimal pair K and P = Ric(H) by the full-rank condition on B in Assumption 1.

Remark 1:
The following interpretation may be given to the fixed-point characterization above. By PMP, the dynamics of the optimal process, i.e., the Hamiltonian system, is known, while the initial condition (for the costate variable) must be determined. In other words, even without computing the solution to the Riccati equation (3), it can be claimed that the (unique) optimal trajectory x (t) must verify col(ẋ (t),λ(t)) = Hcol(x (t), λ(t)), together with the continuous function λ, the boundary conditions of which would, however, require explicit knowledge of P , i.e., λ(0) = P x 0 . In contrast, in the case of the flow ϕ (A+BK)x (t; x 0 ), the initial condition, x 0 ∈ R n , is known, whereas the optimal dynamics, namely, the selection of the state feedback u = Kx, must be determined. Therefore, by relying on the DP approach, it can be claimed that the (unique) optimal trajectory x (t) must solve the ordinary differential equatioṅ x(t) = (A − BR −1 B P )x(t), with the boundary condition x(0) = x 0 . As a consequence, the statement of Theorem 1 entails that the optimal solution can be completely characterized by combining together the two sources of information above. The discussion is visually summarized in the diagrams of Fig. 1.
While providing an alternative characterization of the optimal solution to an LQR problem, the statement of Theorem 1 does not seem to yield a computationally viable way of determining , thus avoiding the lifting operation. The red lines describe the composition for the optimal gain matrix K and from the initial condition λ 0 = P x 0 , for any x 0 ∈ R. The black lines describe the flow composition for generic gain matrix K and initial condition the optimal feedback u = K x, due to the dependence of the mapping Υ on time. This aspect is addressed, and circumvented, in the following statement. Theorem 2: Consider the LQR problem described by the system (1) and the cost functional (2), and suppose that Assumption 1 holds. Then, the pair of matrices P = (P ) and K yields the optimal solution to the problem if and only if where H ∈ R 2n×2n denotes the Hamiltonian matrix defined in with M and N (P ) defined in the statement above. Provided that the state feedback u = K x stabilizes the zero equilibrium point for the closed-loop systemẋ = (A + BK )x, by item (ii) of Theorem 1, the pair of matrices P and K yields the optimal solution to the LQR problem if and only if P ,K (t) = 0, for all t 0. Since P,K (·) is an analytic function, 5 the latter is equivalent to requiring that the function and its time derivatives, 5 A function f : X → R n is analytic in the open set X , denoted as f ∈ C ω (X ), if for every v ∈ X , there is a neighborhood U such that the Taylor series centered at v converges pointwise to f (x) for x ∈ U. evaluated at t = 0, are equal to zero. By definition, P,K (0) = I n − I n = 0, for any matrices P and K. By straightforward computations, it can be shown that for i ∈ N, obtained iteratively for any i by relying on the fact that the previous i − 1 time derivatives must be equal to zero. By solving (10) for i = 1 and by noting that MN(P ) = I for any matrix P , one obtains MHN(P ) = A + BK, which, replaced, in turn, in (10) for i = 2, yields the modified equation The system of equations in (8) is then obtained by applying recursively the above reasoning for i = 1, . . ., 2n. To conclude the proof, note that, since the characteristic polynomial of the matrix A + BK with the optimal matrix K must be entirely contained in that of the Hamiltonian matrix H, denoted as p H (λ) = λ 2n + c 2n−1 λ 2n−1 + · · · + c 1 λ + c 0 , it follows, by the Cayley-Hamilton theorem (see, e.g., [35]), that Therefore, provided that the first 2n time derivatives of P,K are equal to zero for t = 0, as dictated by the equations in (8), the (2n + 1)th expression becomes which is trivially satisfied.

Remark 2:
The interest of the statement of Theorem 2 lies in the fact that the optimal solution to the LQR problem is characterized in terms of a system of equations that are linear in the matrix P , which must satisfy additional requirements, as in the classic formulation, such as symmetry and positivity, although nonlinear in the matrix K, which, however, must not verify any additional condition, apart from item (i) in the statement of Theorem 1.
Remark 3: By inspecting the structure of (8), which is indeed linear in the (unknown) matrix P , it may be desirable to derive an equivalent system of equations that are linear also in the variable K by projecting the equations on appropriate subspaces. To this end, since the first equation in (8) is already linear in K-and obviously in P -consider the second equation, which yields which is obtained by assuming for simplicity that R = I m and by recalling the definition of the Hamiltonian matrix H in (4). Therefore, by projecting the equation (13) on a full-rank matrix which is, however, linearly dependent on the first equation of (8), namely, (A − BB P ) − (A + BK) = 0. By iterating the above strategy of projecting the equations in (8) with i 3 on appropriate matrices, it can be easily shown that there are intrinsically only mn linear equations in (8), while there are mn + (n + 1)n/2 variables, as illustrated also by Example 1.

Example 1:
To visualize the structure of the equations in (8), consider a double integrator, namely letting in (1) and define Q = I 2 and R = 1 in (2). Then, selecting K = [k 1 , k 2 ] and defining P = p 11 p 12 p 12 p 22 (15) the first equation in (8) is whereas the second, third, and fourth sets of equations are described by and respectively. Therefore, by considering the linear equations for the entries of P , namely, p 11 = k 1 k 2 , p 12 = −k 1 , and p 22 = −k 2 , and the stability requirement, resulting in k 1 < 0 and k 2 < 0, the optimal solution is characterized by the pair of equations which are derived from the off-diagonal elements of (18). In fact, by adding (20a) to (20b), one obtains (k 1 + 1)(k 2 2 + k 1 ) = 0, which is solved by letting either k 1 = −1, which replaced in (20a) yields k 2 = ± √ 3, or k 1 = −k 2 2 , which replaced in (20a) leads to the equation k 2 2 + 1 = 0 that does not admit real solutions. Therefore, the optimal solution is indeed k 1 = −1 and k 2 = − √ 3. The next result shows that the equations in (8) lead to a (pseudo-)ARE, namely, a matrix equation that remains quadratic in the unknown (symmetric positive-definite) matrix P and the coefficients of which are polynominal functions of the gain matrix K. Such a matrix function has the property of possessing a sign-definite quadratic term in the matrix P , regardless of the sign of the input block (North-East entry of H) of the underlying Hamiltonian matrix, as it will be shown in the case of the H ∞ control problem. Toward this end, consider the definitions and partitions where i ! denotes the factorial of the positive integer and define Theorem 3: Consider the LQR problem described by the system (1) and the cost functional (2), and suppose that Assumption 1 holds. Then, the pair of matrices P = (P ) and K yields the optimal solution to the problem if and only if σ(A + BK ) ⊂ C − and satisfies the (pseudo-)ARE where

Proof of Theorem 3:
The result is obtained by combining the right-hand sides of the equations in (8) which is equal to zero if and only if the equations in (8) are satisfied. Straightforward computations show that the matrix in (25) can be written as in (24) by relying on the definitions in (21)- (23). Positive semidefiniteness of the matrix X follows by applying a Schur complement argument to the matrix in (21), which is positive semidefinite by construction. Finally, note that the matrix W can be equivalently written as (26) Hence, it is positive semidefinite.
The next result shows that the matrix X is actually positive definite under the same hypotheses of Theorem 3, i.e., provided that the mild conditions of Assumption 1 are satisfied.
Proposition 1: Suppose that the hypotheses of Theorem 3 hold. Then, the matrix X in (24) is positive definite, namely X 0.
Proof of Propsition 1: By inspecting the structure of the lefthand side of (21), to prove the claim, it is sufficient to show that there does not exist a nonzero vector v ∈ R 2n such that or, equivalently, a vectorv = Hv such that where H is nonsingular, since it belongs to dom(Ric) (see [7]), hence σ(H) ⊂ C \ C 0 . Therefore, the nonexistence of such a vectorv is implied by observability of the pair (M, H). By recalling that M = [I n , 0] and that observability properties of the pair (M, H) are equivalent to those of the pair (M, H + GM ) for any matrix G ∈ R 2n×n , it follows that observability of the pair obtained by selecting G = [0, Q ] , is equivalent to observability of the original pair (M, H). The former is, in turn, equivalent to the observability of the pair (A ,R B ), withR such that RR = R −1 , which is, by duality, equivalent to reachability of (A, B), thus concluding the proof.

III. H ∞ CONTROL PROBLEM FOR LINEAR SYSTEMS
The main objective of this section is to extend the results above also to the context of the H ∞ control problem. To this end, consider a (perturbed) linear time-invariant system described bẏ and u(t) ∈ R m denoting the state, the output, the exogenous disturbance input, and the control input, respectively. In addition, the disturbance signal w : The classic formulation of the H ∞ control problem consists of ensuring that the L 2 -gain 7 from the disturbance input w to the output col(y, u) for the closed-loop system is smaller than a given γ > 0 (see [7] and [34] for more detailed definitions and discussions). As it is well known, the classic solution to the above problem revolves around the ARE in which the quadratic term is typically indefinite in sign, whenever the control and disturbance inputs are not matched. Suppose that P is a symmetric positive-semidefinite solution to (31); the solution to the control problem is then given by The aim of this section is to provide an alternative characterization of the solution, similarly to what has been pursued in the previous section in the case of the LQR problem. To this end, let denote the Hamiltonian matrix naturally associated with the Riccati equation in (31). The following structural assumption is considered and subsequently motivated in Remark 4.
• The statements of the previous section can be naturally extended to the case of the H ∞ control problem.
Theorem 4: Consider the H ∞ control problem for the system (30), and suppose that Assumption 2 holds and that the pair (H, M ), with M = [I n , 0] , is observable. Then, the triple of matrices P = (P ) , K 1 , and K 2 yields the solution to the 7 Given the systemẋ = Ax + Bu, y = Cx + Du, with A Hurwitz, i.e., with the property that σ(A) ⊂ C − , the L 2 -gain from u to y is defined as [8]). 8 If the Hamiltonian matrix H belongs to dom(Ric), then the matrix P P 2 P −1 1 , with P 1 and P 2 such that im[P 1 , where H ∞ ∈ R 2n×2n denotes the Hamiltonian matrix in (32), Proof of Theorem 4: The claim is proved by combining the arguments employed in the proofs of Theorems 1 and 2. More precisely, consider the function (32), and note that the triple P , K 1 , and K 2 yields a solution to the underlying H ∞ control problem if and only if any x 0 ∈ R n is a fixed point of T P ,K 1 ,K 2 for any t 0, namely T P ,K 1 ,K 2 (t, x 0 ) = x 0 . Derivations identical to those discussed in the proof of Theorem 2 then lead to the system of equations (33), hence concluding the proof.
Remark 4: The rank condition of Assumption 2 entails that the somewhat more interesting case of unmatched disturbance attenuation problems is addressed. To further illustrate the relevance of such an assumption, consider the scalar systeṁ namely as in (30) with A = 1, B 1 = 1, and B 2 = √ 2, for which clearly Assumption 2 does not hold. Selecting γ = 1, the corresponding ARE (31) is given by 0 = 1 + 2P − P 2 , which admits the solutions P = 1 ± √ 2. Therefore, P = 1 + √ 2 0, K 1 = 1 + √ 2, and K 2 = −(2 + √ 2). Moreover, Consider, instead, the equations in (33) with n = 1, namely By replacing (36b) into (36a), one univocally obtains, as expected, P = 1 + √ 2. Equation (36b) entails that the matrices K 1 and K 2 must be such that the closed-loop system possesses the stable eigenvalue of the underlying Hamiltonian matrix. However, whenever Assumption 2 does not hold, the latter requirement is achieved by a set-a subspace, in this example-of matrices defined as , which solve (36) [i.e., (33)] for any α ∈ R. Interestingly, recalling the zero-sum (min/max) dynamic game formulation of the H ∞ control problem with respect to the cost functional (see [36]) (37) with C = 1 and γ = 1, and noting that, for fixed initial condition, the trajectories x α (t) coincide for any α ∈ R pointwise in t, it follows that the cost (37) is minimized for any such that K(α ) recovers the classic solution.
The following statement provides a characterization of the solution to the H ∞ control problem in terms of a matrix equation that is quadratic with respect to the variable P as in (31) and with coefficient of the quadratic term sign definite. The proof of the result can be obtained by straightforward combination of the arguments employed in the proofs of Theorem 3 and Proposition 1; hence, it is omitted.
Theorem 5: Consider the H ∞ control problem for the system (30). Suppose that Assumption 2 holds and the pair is observable. Then, the triple of matrices P = (P ) , K 1 , and K 2 yields the solution to the problem if and only if σ(A K 2 ) ) 0 and X 0, with the matrices J 1 , J 2 , and V , obtained as straightforward adaptations of (22) and (23), respectively.
The interest in the conclusions of Theorem 5 lies in the fact that the right-hand side of (38) is a convex (matrix-valued) function of the variable P , for fixed K 1 and K 2 that are, in principle, uniquely related to P , once the latter has been computed. More precisely, letting M : R n×n → R n×n denote the matrix-valued function that maps P into the right-hand side of (38), it can be shown that for any h ∈ [0, 1] and any P i ∈ R n×n , i = 1, 2, which is positive semidefinite, provided that X is positive definite, as shown by a straightforward Schur complement argument. This feature renders (38) particularly suitable for iterative zero-finding techniques, such as the gradient algorithm, as detailed in the following section.

IV. DISCUSSION ON COMPUTATIONAL ASPECTS
The aim of this section is to design iterative algorithms to find a solution to (8) or to (24), by exploiting the properties of positive definiteness of the coefficient of the quadratic term in (24) and (38), respectively, and convexity of the right-hand side with respect to the matrix P . To this end, a brief review of geometrical concepts and Riemannian manifolds is in order for completeness (see, e.g., [37] for a complete review).
Let M denote a Riemannian manifold, and let T x M denote the tangent space to M at x ∈ M. The definition of an inner product X, Y x ∈ R, for X, Y ∈ T x M, allows turning the Riemannian manifold into a metric space. More precisely, the length of a curve φ x,X : (40) whereas, given any x ∈ M and X ∈ T x M, the curve γ x,X : [0, 1] → M of the shortest length is referred to as geodesic curve. The length (γ x,X ) of the latter curve is the geodesic distance between the two points γ x,X (0) ∈ M and γ x,X (1) ∈ M; hence, the Riemannian distance is defined as Moreover, given a continuously differentiable function f : M → R and given any smooth curve φ x,X : Let S + (n) denote the manifold of symmetric positive-definite matrices, namely, S + (n) = {X ∈ R n×n : X = X , X 0}.
The tangent space at a point X ∈ S + (n) is defined as T X S + (n) = {V ∈ R n×n : V = V }, namely, by the space of symmetric matrices. The manifold S + (n) becomes a Riemannian manifold by introducing the Riemannian metric as for V 1 ∈ T X S + (n) and V 2 ∈ T X S + (n), where tr(·) denotes the trace of a matrix. Finally, the geodesic curve at the point X ∈ S + (n) in the direction V ∈ T X S + (n) is defined as and it is entirely contained in S + (n) for any s ∈ [0, 1].
It is now possible to envision a gradient algorithm on the Riemannian manifold of the positive-definite matrices to iteratively compute the solution to (24) and (38). To this end, the discussion contained in the paragraph before (21) is now particularly useful. In fact, the (pseudo-)ARE remains quadratic in P , which is constrained to belong to S + (n), while becoming sign definite. This is achieved at the price of polynomial dependence of the coefficients on the matrix K, which is instead not constrained to belong to a specific manifold.

(45)
The following technical statement provides a characterization of the gradient of the function J with respect to P and K i , i = 1, 2.

Algorithm 1: Gradient Algorithm on S + (n).
1: parameters: w 1 , w 2 , ε, ∈ R + , s 0 ∈ [0, 1], α ∈ (0, 1) 2: input: P 0 ∈ S + (n), K 0 1 ∈ R p×n , K 0 2 ∈ R m×n 3: init: s ← αs 10: repeat: Steps 6, 7 11: end while 12: Lemma 1: Consider the function J defined in (45). Then, the Riemannian gradient of J with respect to P , namely, ∇ P J ∈ T P S + (n), is (46) Moreover, the gradient of J with respect to K j is Proof of Lemma 1: Expressions (46) and (47) are obtained from J in (45) by relying on classic properties of the trace of a matrix and by recalling the equivalent formulations of A(P, K 1 , K 2 ) in (38) and (25), respectively. Algorithm 1 describes an iterative strategy inspired by the gradient method that minimizes the cost function J and that incorporates also a line-search method to select the corresponding step, hence obtaining a triple solving the H ∞ control problem.
Remark 5: Provided that the update s ∈ R + in steps 12 and 13 of Algorithm 1 is sufficiently small, the retraction γ X,V defined in (44)-which requires the, typically daunting, computation of a matrix exponential-can be approximated as Newton's approaches to the problem yield quadratic convergence. These methods include, for instance, [26] in the case of optimal control problems and [25] for sign-indefinite Riccati equations. In particular, the latter requires, at each step, to solve a linear and a quadratic matrix equation. Compared to the strategies above, the scheme proposed in Algorithm 1, together with (48), does not involve additional computations other than matrix multiplications, even in the case of sign-indefinite Riccati equations. Note that, instead, in [38], a two-player policy iteration strategy, involving only linear equations, is proposed for the nonlinear H ∞ control problem, which relies on the use of a quasi-norm for the control input to satisfy requirements of input saturation and, simultaneously, to ensure sign definiteness of the quadratic term in the Hamilton-Jacobi-Isaacs equation. Thus, even in the case of the LQR problem, Algorithm 1 has a lower computational complexity than the classic Kleinman iterations: indeed, the cost of a single iteration of Algorithm 1-involving a O(1) number of multiplications between n × n matrices-is O(n 2.373 ) by implementing the optimized Coppersmith-Winograd algorithm proposed in [39], whereas the cost of a single iteration of Kleinman's method-involving the solution of a Lyapunov equation, i.e., the inversion of an n 2 × n 2 matrix-is O(n 4.746 ). Remark 6: Whenever the observability condition of Theorem 5 is satisfied, and hence X is guaranteed to be positive definite, it may be possible to weigh with μ ∈ R 0 also the original formulation of the (possibly sign-indefinite) Riccati equation in the cost function J in (45), which is then employed in Algorithm 1, provided that X + μ(γ −2 B 1 B 1 − B 2 B 2 ) remains positive definite.
Example 2: To assess the performance of Algorithm 1, consider the LQR problem described by system (1) [26] and due to the structure of the updates in Algorithm 1, it is not required that P 0 = (P 0 ) > 0 and K 0 possess specific structures relative to each other or even that K 0 is stabilizing for system (1)  and it is such that the manifold of symmetric positive-definite matrices S + (n) is invariant with respect to the updates in step 12 of Algorithm 1. This, in turn, derives from the properties of the geodesic curve (44) and the initialization, with s 0 1 and α < 1, of the algorithm. This aspect is further illustrated by the following example. The top graph of Fig. 2 shows the natural logarithm of the cost function J in (45) against the iteration number k, while the bottom graph depicts the logarithm of the matrix infinity norm 9 of the right-hand side of the ARE (3), which allows visualizing the convergence of P k and K k to the underlying optimal values. In particular, for k = 10 4 , the algorithm yields  which, in turn, lead to the pseudo-ARE (24) given by The right-hand side of (53) is a convex function of the variable p that admits a unique minimum with respect to p, for fixed k. Algorithm 1 is then applied to (53) by considering the discretetime updates with ∇ p J and ∇ k J computed as in (46) and (47), respectively. Since the proposed algorithmic approach is significantly different from classic Kleinman's iterations, the former possesses advantages and drawbacks with respect to the latter, which are both intimately related to the shape of the basins of attraction of the distinct solutions to the classic ARE. More precisely, on one hand, there may be initial pairs (p 0 , k 0 ) (such as the ones contained in the shaded red region, e.g., described by the black square, in Fig. 3 ) with p 0 > 0 and k 0 stabilizing for the closedloop system such that the updates (54) do not converge to the optimal solution, denoted by the red star in Fig. 3. Note that, since p 0 > 0 implies p i > 0 for any i ∈ N, it follows that such trajectories are, in fact, not attracted by unstable solutions to the classic ARE. On the other hand, the nature of the basin of attraction is also such that there are pairs (p 0 , k 0 ) (such as the ones in the shaded green region, e.g., described by the black circle, in Fig. 3) with k 0 not stabilizing for the closed-loop system with the property that the updates (54) do converge to the optimal solution, differently from any algorithm inspired by ideas similar to those of Kleinman's iterations. The shaded yellow region of Fig. 3 depicts the values of the control gain k, for which the closed-loop system is unstable. Example 4: A numerical simulation similar to the one in Example 2 is carried out in the case of the H ∞ control problem described by the perturbed system (30) defined by the randomly generated matrices Note that, since the disturbance is unmatched, the resulting ARE (31) is sign indefinite; hence, an iterative scheme, as that suggested in Algorithm 1, would not be feasible if applied directly to (31). The latter algorithm is then implemented on the equivalent matrix equation (38) with w 1 = w 2 = 1, ε = 10 −4 , s 0 = 1, α = 0.9, and μ = 0. The application of Algorithm 1 from the randomly selected initial matrices Similarly to Fig. 2 in Example 2, the top graph of Fig. 4 displays the logarithm of the cost function J in (45) against the iteration number k, while the bottom graph depicts the logarithm of the matrix infinity norm of the right-hand side of the ARE (31).

V. ALGEBRAIC CHARACTERIZATION OF OPTIMAL AND H ∞ CONTROLS FOR NONLINEAR SYSTEMS
The objective of this section is to extend the previous constructions to the solution of optimal and H ∞ control problems in the nonlinear context. To this end, consider first an infinite-horizon optimal control problem with a nonlinear dynamic constraint, i.e., where the positive-definite function q : R n → R >0 , the vector field f : R n → R n , and the mapping g : R n → R n×m are assumed to be analytic functions. Suppose that rank g(x) = m, for any x ∈ R n . The underlying value function V : R n → R 0 is defined as V (x 0 ) inf u {Q x 0 }, for any initial condition x 0 ∈ R n (see [6]). The following assumption is introduced to guarantee asymptotic stability of the origin for the closed-loop optimal system. Assumption 3: The dynamics in (59) is zero-state detectable from the output y = q(x), namely, any trajectory such that u(t) ≡ 0 and y(t) ≡ 0 is such that lim t→∞ x(t) = 0.
• It has been shown (see, e.g., [5]) that by relying on the DP approach, the optimal solution to the problem (59) is given by the feedback control law for any x ∈ R n . The latter PDE represents the nonlinear counterpart of the algebraic equation (3). In particular, (61) is derived by the HJB PDE for any x ∈ R n . As it is reasonable to expect, the value function V is a natural candidate solution for the HJB PDE (62): the function V indeed solves the latter PDE, provided that V (x 0 ) is continuously differentiable and Q x 0 admits a continuously differentiable minimizer for any x 0 ∈ R n . The approach based on DP and described above yields the information about the optimal solution and the minimal cost for any initial configuration of the plant, provided, however, that one is able to determine a closed-form solution to the HJB PDE (61) or (62). Such a computation is typically a daunting task, and a closed-form expression of the solution can be rarely determined in practice. Therefore, despite the fact that (62) yields necessary as well as sufficient conditions for optimality if the value function of (Q) is continuously differentiable, alternative design strategies based on Hamilton conditions (see [6]) are usually preferred in applications. The latter conditions are trajectory-based, hence computed only for specific initial conditions x 0 ∈ R n and yield, in general, only necessary conditions. More precisely, let H : R n × R n → R denote the (minimized) Hamiltonian function naturally associated with the optimal control problem (59), namely where λ(t) ∈ R n is the costate variable. Then, provided that the underlying value function V is twice continuously differentiable with respect to the initial condition x 0 ∈ R n in (59), the optimal process satisfies the dynamics (Hamilton conditions) with In the case of finite-horizon optimal control problems, the dynamics in (64)-together with readily available boundary conditions, typically directly related to the cost functionalrepresent the most common design strategy to identify extremals for (Q x 0 ), namely, candidate optimal solutions. In the case of infinite-horizon optimal control problems, as in (59), the use of (64) is essentially prevented by the fact that the boundary conditions are intrinsically related to a solution to the PDE (61), knowledge (seldom available) of which would, however, bypass the conditions in (64) and directly lead to the implementation of (60). The main objective of this section is then to circumvent the above obstruction by combining the information yielded by the characterization of the optimal solution as in (60) and (61) with that as in (64), namely combining DP with Pontryagin's approach, mimicking the strategy proposed in Section II in the linear setting. To this end, recall that ϕ J∇H (t; x 0 , λ 0 ) denote the flow of the system (64) at time t 0 and from the initial condition (x 0 , λ 0 ) ∈ R n × R n . In addition, given a feedback control input π : R n → R m , let ϕ f +gπ (t; x 0 ) denote the flow of the dynamics in (59) in closed loop with u = π(x).
Theorem 6: Consider the infinite-horizon optimal control problem defined in (59). Suppose that Assumption 3 holds. Then, the feedback control input u = π(x) is the optimal solution of (59) if and only if there exists an open set X , containing the origin, such that for any x 0 ∈ X ⊆ R n , there exists λ 0 ∈ R n Fig. 5. Graphic representation of the composition of flows (66). The red lines describe the composition for the optimal feedback π and from the initial condition λ 0 = ∇ x V (x 0 ), for any x 0 ∈ R. The black lines describe the flow composition for generic feedback π and initial condition with the property that for all t 0, and the closed-loop system is locally asymptotically stable with basin of attraction containing X . Proof of Theorem 6: By relying on [11] and [40]-in which a formulation identical to Q x 0 is considered-and on analyticity of the problem data, the value function is a real analytic function of x 0 in a neighborhood of the origin. Since the value function V is continuously differentiable, it is also a smooth solution to the HJB equation (62), which yields necessary and sufficient conditions for optimality, and hence, the optimal solution u exists; it is continuously differentiable and provided by (60) with V replaced by the value function V . By Assumption 3, the optimal feedback u stabilizes the zero equilibrium of the closed-loop system, as certified by the function V employed as a Lyapunov function for the latter system. Moreover, since the value function is, in fact, twice continuously differentiable, then the costate λ of the Hamiltonian dynamics (64) satisfies (see, e.g., [6]) λ(t) = ∇ x V (x (t)) for any t 0. Therefore, (66) must hold with λ 0 = ∇ x V (x 0 ), for any x 0 ∈ R n , thus showing necessity of the conditions in the statement. Fix now x 0 ∈ R n and suppose, instead, that (66) holds for some λ 0 ∈ R n . The condition implies that the trajectory of the dynamics in (59) in closed loop with π ensuing from x 0 coincides with a trajectory of the Hamiltonian dynamics (64) for any time t 0; hence, it is optimal, thus concluding the proof. Fig. 5 provides a graphic representations of the fixed point representation for the composition of flows in the nonlinear setting of Theorem 6. To provide a more constructive interpretation of the fixed-point condition stated in Theorem 6-similarly to Theorem 2 in the linear case-consider first the following preliminary definitions and technical results.
Definition 1: Given a smooth vector field f : R n → R n , consider the operator D i (f ) recursively defined as D 0 (f ) = f and for i ∈ N, where ∇f denotes the Jacobian matrix of the vectorvalued function f . • The operator introduced in Definition 1 can be employed to compactly characterize repeated time derivatives of the flow of an ordinary differential equation, as detailed in the following statement.
Lemma 2: Consider an ordinary differential equation described byẋ = f (x), and let ϕ f (t; x 0 ) denote the corresponding flow from x 0 ∈ R n at time t. Then for i ∈ N. Theorem 7: Consider the infinite-horizon optimal control problem defined in (59). Suppose that Assumption 3 holds. Then, the feedback control input u = π(x) is the optimal solution of (59) if and only if there exists an open set X , containing the origin, such that for any x 0 ∈ X ⊆ R n , there exists λ 0 ∈ R n with the property that for any i ∈ N 0 , with M = [I n , 0] , and the closed-loop system is locally asymptotically stable with basin of attraction containing X .

Proof of Theorem 7:
Let ω(t) denote the left-hand side of (66), namely, Since ω is an analytic function of time t, it follows that the function and its time derivatives, evaluated at t = 0, must be equal to zero. To begin with, ω(0) = ϕ f +gπ (0; Mϕ J∇H (0; x 0 , λ 0 )) − x 0 = 0, by definition, for any λ 0 ∈ R n . Then, note that the function ω(·) is identically equal to zero if and only if the functionω is equal to zero for any t ∈ R 0 . Therefore, the ith time derivative of the functionω is x 0 ) (71) which, evaluated at t = 0 and by Lemma 2, yields the ith equation in (69).
Remark 8: The requirement that the data of the problem as well as the underlying value function be analytic with respect to their arguments are instrumental for showing that the algebraic equations (69) completely characterize the optimal state feedback in a neighborhood of the origin. Nonetheless, while the rigorous analysis relies on the restrictive analyticity assumption, it is worth observing that the synthesis-namely, a control design strategy based on the solution of a certain (finite) number of the algebraic equations (69) at a specific state-may still be viable even in the presence of functions that are not analytic in a neighborhood of the desired state. In fact, since, in practice, (69) must be considered for any i κ, for a prescribedκ < ∞, if the function V belongs to C κ , with κ max{2,κ + 1}, then solving (69) permits to approximate the optimal feedback via a finite power series with respect toπ ij for any i κ, namely with the multi-index power defined as x α = x α 1 1 · . . . · x α n n , at least in a suitably defined (potentially sufficiently small) neighborhood of x 0 .
Example 5: Consider an infinite-horizon optimal control problem described by the cost functional and the dynamics in (59), with q(x) = x 2 , f (x) = x 2 , and g(x) = 1, for any x ∈ R, respectively. This specific problem admits a closedform solution for the optimal feedback, namely, The objective of the case study is to approximate such an optimal feedback by solving the algebraic conditions (69) at different points of the state space, which are subsequently patched together via continuity constraints. More precisely, consider a collection of points x 0,i , i ∈ N, and consider the task of approximating the optimal feedback at each x 0,i , with such an approximation valid in the intervals [x 0,i − δ l,i , x 0,i + δ r,i ], with δ l,i such that x 0,i − δ l,i = x 0,i−1 + δ r,i−1 . To this end, consider the piecewise linear approximation π a (x), with each linear function defined around x 0,i as π a,i (x) =π i,0 +π i,1 x, where the unknowns π i,0 and π i,1 (and the corresponding values of λ 0,i ) are computed for each i by determining a solution to (69), limited to the operator D of order 0 and 1, and combined with continuity requirements for π a,i−1 and π a,i , namely respectively. The equations in (73) with the above constraint admit two real solutions, only one of which locally stabilizes the zero equilibrium of the closed-loop system. Fig. 6 shows the comparison between the optimal feedback π (x) (dashed gray line) and the piecewise linear approximation π a (x) (solid black line) obtained by solving the system of algebraic equations (73) with the points x 0,i selected as {−3/2, −1/2, 1/2, 3/2}, which are indicated in Fig. 6 by black dots, and δ l,i = δ r,i = 1/2, for i = 1, . . ., 4. Note that one may envision a design method, in which equations in (73) are solved in a receding-horizon fashion, i.e., without considering an a priori grid on the state space. Finally, Fig. 7 shows the approximate control laws obtained by considering a single state, i.e., x 0 = 0.1, and increasing order of the polynomials defining π i a , i = 1, 2, 3. In particular, the dashed line describes the linear approximation π 1 a (x) =π 1 x, the dotted line shows the quadratic  approximation π 2 a (x) =π 1 x +π 2 x 2 , whereas the dash-dotted line depicts the cubic approximation π 3 a (x) =π 1 x +π 2 x 2 + π 3 x 3 . It can be noted that while increasing the order of the approximation, the function π i a is more and more capable of capturing the one-sided saturation-like behavior of the optimal control law. Moreover, note that the approximate value of the optimal control is such that π (x 0 ) − π i a (x 0 ) is monotonically decreasing with respect to the order i.
Example 6: Consider an infinite-horizon optimal control problem described by the cost functional in (2) with q(x) = x 2 2 and the controlled Van der Pol dynamicṡ while the associated Hamiltonian function is Let the state x 0 = [0.5, 0.5] be given, let π a (x) denote a quadratic approximation of the optimal solution, which is u = −x 1 x 2 , namely π a (x) =π 11 x 1 +π 12 x 2 +π 21 x 2 1 +π 22 x 2 2 +π 23 x 1 x 2 (76) and consider (69) for i = 0, . . ., 3, thus obtaining a system of seven algebraic equations (since the first equation for i = 0 is identically equal to zero) in the seven unknownsπ = {π 11 ,π 12 ,π 21 ,π 22 ,π 23 } and λ 0 ∈ R 2 . To find solutions to such a system of nonlinear equations by means of local methods, consider a uniform grid for the values of The best approximation is then determined by considering the numerical evaluation of the cost functional along the trajectories of the system in closed loop with the alternative sets of solutions from the initial condition x 0 . To this end, note thatπ 4 induce diverging trajectories, while Jπ 1 = 1.4908 > Jπ 2 = 0.2885 > Jπ 3 = 0.2498, hence identifying π a (x)|π 3 as the best approximation, in fact the optimal solution. Fig. 8 depicts the trajectories of system (74) in closed loop with (76) for the set of parameters inπ 1 (dashed gray line),π 2 (dotted gray line), andπ 3 (solid black line), respectively To conclude the section, consider the nonlinear formulation of the H ∞ control problem dealt with in Section III in the case of linear systems. Toward this end, consider a nonlinear input-affine system described by equations of the forṁ where x(t) ∈ R n denotes the state of the system, u(t) ∈ R m is a control input, and w(t) ∈ R p is an exogenous signal. Assume that f and g i , i = 1, 2, are analytic vector fields and that rank [ g 1 (x) g 2 (x) ] = m + p, for any x ∈ R n . Moreover, let y(t) ∈ R q denote a (measured) output of system (78). Assuming initially that u(t) = 0, for all t 0, consider a function s : R p × R q → R, referred to as supply rate. Definition 2 (see [8]): The system (78), with u(t) ≡ 0, is said to be dissipative with respect to the supply rate s if there exists a function V : R n → R 0 , called storage function, such that for all x 0 ∈ R n , T 0 and input functions w V (x(T )) V (x 0 ) + 1 2 T 0 s(w(τ ), y(τ ))dτ.
Moreover, if (79) holds with the equality sign, then system (78) is lossless with respect to s. • The notion of L 2 -gain of a nonlinear system and the solution to the (nonlinear) H ∞ control problem are then recalled in the following definition and proposition, respectively.
Definition 3 (see [8]): Let γ > 0. The system (78), with u(t) ≡ 0, has L 2 -gain less than or equal to γ if it is dissipative with respect to the supply rate s(w, y) = γ 2 w 2 − y 2 . • Proposition 2: see [21] Consider the nonlinear system (78) with output y = h(x) and let γ > 0. Suppose that there exists a smooth solution V : R n → R 0 of the Hamilton-Jacobi PDE with V (0) = 0. Then, system (78) in closed loop with u = π 2 (x) = −g 2 (x) (∂V /∂x) has L 2 -gain less than or equal to γ from the input w to the output [y, v] , with V as a storage function. Moreover, π 1 (x) = γ −2 g 1 (x) (∂V /∂x) describes the worst-case disturbance. As it appears evident, the PDE (80) represents, in H ∞ or disturbance attenuation control problems, the counterpart of (61) arising in optimal control problems: as a matter of fact, the latter can be recovered from the former by letting the desired attenuation gain γ tend to infinity (hence without posing any disturbance attenuation requirement). Moreover, a characterization of the resulting closed-loop trajectories similar to (64) can be given by adapting the latter Hamiltonian dynamics to the Hamiltonian function Therefore, considering the ingredients in the statement of Proposition 2 together with the Hamilton conditions (64) with H replaced by H ∞ defined in (81), the straightforward extension of the results in Theorems 6 and 7 to the H ∞ control design for nonlinear systems is contained in the two following statements, respectively. Theorem 8: Let γ > 0. Consider the H ∞ control problem for the nonlinear system (78) with output y = h(x) and suppose that the underlying storage function is analytic. Then, the feedback control input u = π 2 (x) solves the problem and w = π 1 (x) is the worst-case disturbance if and only if there exists an open set X , containing the origin, such that for any x 0 ∈ X ⊆ R n , there exists λ 0 ∈ R n with the property that ϕ f +g 1 π 1 +g 2 π 2 (−t; Π x • ϕ J∇H ∞ (t; x 0 , λ 0 )) − x 0 = 0 (82) for any t 0, and the closed-loop system is locally asymptotically stable with basin of attraction containing X .
Theorem 9: Let γ > 0. Consider the H ∞ control problem for the nonlinear system (78) with output y = h(x) and suppose that the underlying storage function is analytic. Then, the feedback control input u = π 2 (x) solves the problem and w = π 1 (x) is the worst-case disturbance if and only if there exists an open set X , containing the origin, such that for any x 0 ∈ X ⊆ R n , there exists λ 0 ∈ R n with the property that 0 = M D i (J∇H ∞ )(x 0 , λ 0 ) − D i (f + g 1 π 1 + g 2 π 2 )(x 0 ) (83) for any i ∈ N 0 , with M = [I n , 0] , and the closed-loop system is locally asymptotically stable with basin of attraction containing X .

VI. CONCLUSION
The optimal control and the disturbance attenuation problems have been addressed in the linear as well as in the nonlinear setting. The proposed novel conditions for analysis and synthesis are derived by aiming at combining the two main approaches introduced to tackle such problems, namely, those inspired by DP and those based on PMP. The result is achieved by matching the corresponding optimal trajectories yielded by each of the methods, thus exploiting and combining the alternative sources of information, which include knowledge of the complete initial condition from DP and of the optimal dynamics from PMP.
The proposed analysis and the design strategy have then provided valuable insights on linear as well as nonlinear systems. In the case of linear systems, the derived conditions have led to matrix algebraic equations, similar to the classic ARE, with the property that the coefficient of the quadratic term of such equation is (strictly) sign definite, even if the original ARE is sign indefinite, as it is typically the case in the H ∞ control problem. This feature has been then efficiently employed to design a gradient-based minimization algorithm in the manifold of positive-definite matrices for (matrix) convex functions. Finally, in the presence of nonlinear dynamics, the strategy has led to algebraic equations that allow us to (locally) construct the optimal feedback by considering conditions at a single point.
Finally, a similar strategy based on the algebraic characterization of fixed points for compositions of flows appears promising also toward the extension to different contexts, encompassing dynamic games, the design of optimal filters and observers, and the solution to two-point boundary value problems arising, for instance, in finite-horizon optimal control problems.