In the modelling of uncertainty, the theory of stochastic processes [1] provides established mathematical tools for the modelling and the analysis of systems with random dynamics. Furthermore in application, the possibility to control sequences of events subject to random disturbances is highly desirable for real applications. In this paper, we elucidate the connection between the well established Hamilton-Jacobi-Bellman (HJB) control frame- work [2] [3] and a control strategy based on the Fokker-Planck (FP) equation [4] [5] . Illustrative examples allow gaining additional insight on this connection.

We focus on a representative n-dimensional continuous-time stochastic processes described by the following model

where is a Lipschitz-continuous n-dimensional drift function and is a m-dimensional Wiener process with stochastically independent components. The dispersion function with values in is assumed to be smooth and full rank; see [6] . This is the well-known Itō stochastic differential equation (SDE) [1] where we consider also the action of a d-components vector of controls, that allows driving the random process towards a certain goal [3] . We denote with the set of Markovian controls that contains

all jointly measurable functions, where is a given compact set [3] . In determinis-

tic dynamics, the optimal control is achieved by finding a control law that minimizes a given objective defined by a cost functional; see, e.g., [2] .

In the non-deterministic case, the state is random, so that inserting a stochastic process into a deterministic cost functional will result into a random variable. Therefore, in stochastic optimal control prob- lems the expected value of a given cost functional is considered [7] . In particular, we have

This is a Bolza type cost functional in the finite-horizon case and it is assumed here that the controller knows the state of the system at each instant of time (complete observations). For this case, the method of dynamic programming can be applied [2] [7] [8] in order to derive the HJB equation for with as the optimization function. Some other cases of the cost structure of are quoted in [8] , that have applications in finance, engineering, and in production planning and forest harvesting. Each will lead to a different form of the HJB equation that can be analysed with appropriate methods of partial differential equa- tions; see, e.g., [9] .

A control approach close to the HJB formulation consists in approximating the continuous stochastic process by a discrete Markov decision chain. In this approach the information of the controlled stochastic process, carried by the transition probability density function of the approximating Markov process, is utilized to solve the Bellman equation; for details see [10] .

However, the common methodology to find an optimal controller of random processes consists in reformulat- ing the problem from stochastic to deterministic. This is a reasonable approach when we consider the problem from a statistical point of view, with the purpose to find out the collective “ensemble” behaviour of the process. In fact, the average of the functional of the process is omnipresent in almost all stochastic optimal control problems considered in the scientific literature.

The value of the cost functional before averaging is a way to measure the cost of a single trajectory of the process. However, the knowledge of the single realization is not useful for the statistical analysis, that would require to determine the average, the variance, and other properties associated to the state of the stochastic process.

On the other hand, a stochastic process is completely characterized by its law which, in many cases, can be represented by the probability density function (PDF). Therefore, a control methodology that employs the PDF would provide an accurate and flexible control strategy that could accommodate a wide class of objectives. For this reason, in [11] - [14] PDF control schemes were proposed, where the cost functional depends, possibly non- linearly, on the PDF of the stochastic state variable; see, e.g., [11] - [14] for specific applications.

The important step in the Fokker-Planck control framework proposed in [4] [5] is to recognize that the evolution of the PDF associated to the stochastic process (1.1) is characterized as the solution of the Fokker- Planck (also known as forward Kolmogorov) equation; see, e.g., [15] [16] . This is a partial differential equation of parabolic type with Cauchy data given by the initial PDF distribution. Therefore, the formulation of objec- tives in terms of the PDF and the use of the Fokker-Planck equation provide a consistent framework to formu- late an optimal control strategy of stochastic processes.

In this paper, we discuss the relationship between the HJB and the FP frameworks. We show that the FP control strategy provides the same optimal control as the HJB method for an appropriate choice of the objectives. Specifically, this is the case for objectives that are formulated as expected cost functionals and assuming that both the HJB equation and the FP equation admit a unique classical solution. The latter assumption is motivated by the purpose of this work to show the connection between the HJB and FP frameworks, without aiming at finding the most general setting, e.g. for viscosity solution of the HJB equation [9] [17] [18] or FP equation with irregular coefficients [19] , where this connection holds. Furthermore, we remark that the FP approach allows accommodating any desired functional of the stochastic state and its density, that is now represented by the PDF associated to the controlled stochastic process.

In the next section, we illustrate the HJB framework. In Section 3, we discuss the FP method. Section 4 is devoted to specific illustrative examples. A section of conclusions completes this paper.

We consider the optimal control of the state whose evolution is governed by drift and random diffusion as follows

The control function use the current value to affect the dynamics of the stochastic process by adjusting the drift and the dispersion function.

We define the expected cost for the admissible controls as follows

which is an expectation conditional to the process taking the value at time. Here, solves the stochastic differential equation (1.3) with control and the following functions

are smooth and bounded. We call the running cost and the terminal cost. Our goal is to find an optimal

control which minimizes the expected cost for the process (1.3), namely

We assume that this control is unique. Further, we define the following value function, also known as the cost to go function,

It is well known [2] [3] that solves the Hamilton-Jacobi-Bellman equation and that the optimal control can be reconstructed from. Assume that the optimal control is unique and is attained, then we have

In the following, to ease notation we use the Einstein summation convention: when an index variable appears twice in a single term this means that a summation over all possible values of the index takes place. For exam-

ple, we have that. Moreover, , denotes derivative with respect to the

variable, and denotes partial derivative with respect to the time variable.

Theorem 1. Assume that solves (1.3) with a Markov control function and that the function defined by (1.6) is bounded and smooth. Then satisfies the following Hamilton-Jacobi-Bellman equation

with the Hamiltonian function

and.

Notice that the optimal control satisfies at each time and state the following optimality condition

Existence and uniqueness of solutions to the HJB equation often involve the concept of uniform parabolicity; see [3] [18] [20] . The HJB equation is called uniformly parabolic [3] if there exists a constant such that, for all and, the following holds

where represents a bounded or unbounded set.

If the non-degeneracy condition holds, results from the theory of PDEs of parabolic type imply existence and uniqueness of solutions to the HJB problem (1.7) with the properties required in the Verification Theorem [3] . In particular, we have the following theorem due to Krylov.

Theorem 2. If the non-degeneracy assumption holds, and in addition we have that is compact, is bounded with, the drift, the diffusion, and the Lagrange functions are sufficiently smooth on the space-time cylinder, and the final condition is, then the HJB has a unique solution

.

In this section, we discuss an alternative to the HJB approach that is based on the formulation of a Fokker- Planck optimal control problem. We suppose that the functions and of (1.3) yield a stochastic process for which it exists an absolutely continuous measure. Thus, denotes the PDF of the state variable, where the process starts at with initial value distributed according to the density.

The time evolution of the PDF is governed by the Fokker-Planck equation

Also in this case, the FP problem can be defined in a bounded or unbounded set in. Existence and uniqueness to this problem often relay on the concept of uniform parabolicity as in Theorem 2. For the case, we refer to [19] [20] ; see also [21] and the references therein. Furthermore, we remark that in the case of bounded domains, boundary conditions for the FP model must be chosen that ought to be meaningful for the underlying stochastic process. This is a delicate issue that is not the focus of this work and therefore, in the following, we consider the common case where.

Now, we consider a cost functional that is linear in. In correspondence to this cost functional, we define the following PDE optimal control problem

It is important to recognize that if, then we can write the cost functional introduced in (1.4) as follows

that gives (1.4) in terms of the probability measure,.

To characterize the optimal solution to (1.11) where the cost functional (1.4) is considered, we introduce the Lagrange functional [22] .

where the function represents the Lagrange multiplier.

We have that the optimal control solution is characterized as the solution to the following optimality system

where and.

Now, we illustrate the equivalence between the HJB and the FP formulation. The key point is to notice that (1.16) corresponds to the first-order necessary optimality condition (1.9) for the minimization of the Hamil- tonian in the HJB formulation, once we identify the Lagrange multiplier with the value function in (1.7). Therefore, provided that the minimization problem (1.11) admits a unique solution in terms of the first- and second-order derivatives of, then we can replace such into the backward Kolmogorov equation (1.15), thus obtaining the HJB equation (1.7). This procedure results in the equation (1.15), because of the formal equivalence between (1.16) and (1.9). With our setting, since the solution of (1.7) is unique, then the uni- queness of and follows; see [3] .

Notice that with the above setting, the optimal control does not depend explicitly on the density, but only on the Lagrange multiplier, that is the value function. (This explains why the feedback control is based on the value function.) Hence, the equations (1.15)-(1.16) determine the optimal control. This will not be the case in the more general situation in which the cost functional in (1.12) is not linear in. This happens for instance when does not represent an expected cost; see [4] [5] for the case of a tracking functional for the density.

We also note that the solution to the adjoint FP equation (1.15) and to the optimality condition equation for the control function (1.16) do not depend on the initial condition of the forward FP equation (1.14). Hence, according the HJB formulation, the solution to the backward Kolmogorov equation is not affected by the initial state of the system.

In this section, we consider two examples that illustrate that the FP optimal control formulation may provide the same control strategy as the HJB method. In the following, the first example refers to a Itōstochastic process, while the second example considers a piecewise deterministic process.

In this paper, the connection between the Hamilton-Jacobi-Bellman dynamic programming scheme and a recently proposed Fokker-Planck control framework was discussed. It was shown that the two control strategies were equivalent in the case of mean cost functionals. To illustrate the connection between the two control strategies, the cases of an Itō stochastic process and of a piecewise-deterministic process were considered.

We would like to thank the Mathematisches Forschungsinstitut Oberwolfach for the kind hospitality and for inspiring this work.

M. Annunziato would like to thank the support by the European Science Foundation Exchange OPTPDE Grant.

A. Borzì acknowledges the support of the European Union Marie Curie Research Training Network “Multi- ITN STRIKE-Novel Methods in Computational Finance”.

R. Tempone is a member of the KAUST SRI Center for Uncertainty Quantification in Computational Science and Engineering.

F. Nobile acknowledges the support of CADMOS (Center for Advances Modeling and Science).

Supported in part by the European Union under Grant Agreement “Multi-ITN STRIKE-Novel Methods in Com- putational Finance”. Fund Project No. 304617 Marie Curie Research Training Network.