Structure-driven Deep Reinforcement Learning for Wireless Networked Control

Abstract

Wireless networked control systems (WNCSs) consisting of plants, distributed sensors, remote estimator actuators, and controllers are a key component for Industry 4.0 and have been widely applied in many areas such as industrial automation, vehicle monitoring systems, building automation, and smart grids. In particular, providing a high quality real-time remote estimation of dynamic system plant states is important in ensuring control performance and stability of WNCSs. For large-scale WNCSs, the limited communication resources and unreliable transmission in wireless communications have a significant effect on the remote estimation quality. Therefore, it is challenging to design a transmission scheduling of wireless sensors to guarantee the remote estimation performance. Most existing works on transmission scheduling in wireless communication systems merely use general methods such as value iteration and deep reinforcement learning (DRL) without considering the inherent features of the sensor scheduling problems that are different from other problems. This thesis derives several structural properties of the sensor scheduling problem and then develops new DRL algorithms to minimize the remote estimation mean square error (MSE) by using these properties. First, we derive the threshold structure of the optimal scheduling policy and then propose a structure-driven DQN and DDPG. Second, we prove the monotonicity of the Q function and use it to develop different types of monotonic critic neural networks (NNs). Finally, we obtain the convexity of the optimal value function and the greedy structure of the optimal scheduling policy. Then, we develop a structure-guided unified dual on-off policy (SUDO) DRL framework based on derived structural properties. Numerical results illustrate that our proposed DRL algorithms can significantly improve the convergence speed and estimation performance when compared to general DRL algorithms.