Stochastic kernels represent system models, control policies, and measurement channels, and thus offer a general mathematical framework. We will first present several kernel topologies, which we classify as weak topologies (which includes weak* (also called Borkar), Young, and kernel mean embedding topologies), and strong topologies (in various forms), and study their relations. Regularity of optimal cost on models and policies will then be presented with a unified framework.

We will show that weak topologies are versatile for control policy spaces where the associated compactness/continuity properties result in new existence and approximate optimality results in both discrete-time and continuous-time, such as (i) discrete-time approximations for controlled diffusions with full, partial, and decentralized information, (ii) near optimality of Lipschitz policies, and (iii) finite approximations for stochastic control with decentralized information.

Strong kernel topologies, however, are needed for a general robustness theory on mismatched models: When an optimal control policy designed for an incorrect model is applied to a true system, we show that the expected induced mismatch cost is continuous under continuous weak convergence and uniform convergence of transition kernels. Under stronger regularity, a modulus of continuity is also applicable. As implications we study (i) robustness to finite approximations, (ii) robustness to empirical or Bayesian model learning, (iii) near optimality of quantized Q-learning for MDPs with standard Borel spaces, and (iv) robustness to incorrectly modeled noise or prior distributions. Comparison with several robustness paradigms (such as distributional robustness) will also be discussed.

[Joint work with Ali D. Kara, Omar Mrani-Zentar, Somnath Pradhan, and Naci Saldi].