Markov Decision Processes is a common mathematical framework employed in decision making to factor in uncertainty during the optimisation process. We explore variations to the standard setup which involve controlled observations. Exploration through the control of the quality and accuracy of observations is combined with the exploitation of the gathered information to maximise expected rewards. This differs from standard partial information setups, where one has access to a stream of noisy information which is not subject to active controls. The trade-off is akin to the exploration vs exploitation problem in reinforcement learning. We demonstrate this idea of controlled observations by requiring a payment for observing at each instant, in this case the value functions satisfy a system of quasi-variational inequalities, and we discuss aspects of the numerical implementation with some numerical examples.