Terra - An Open Digital Twin Framework for Simulation-to-Real AI Development
Access status:
Open Access
Type
ThesisThesis type
Masters by ResearchAuthor/s
Mo, YaminAbstract
The development of artificial intelligence (AI) based robot policies such as embodied visual AI is a rapidly emerging field that promises to fully automate robot sensing and control, reducing the need for expert knowledge and information about system dynamics. The recent surge in ...
See moreThe development of artificial intelligence (AI) based robot policies such as embodied visual AI is a rapidly emerging field that promises to fully automate robot sensing and control, reducing the need for expert knowledge and information about system dynamics. The recent surge in the availability of digital simulation environments has helped to alleviate the data scarcity problem faced by AI training and yielded many tasks for embodied AI to explore. In contrast to conventional computer vision databases (e.g. ImageNet, COCO), they provide interactive environments tailored to the training of embodied AI agents (e.g. virtual robots). Despite their success, there remain several unaddressed issues. First, these environments primarily focus on in-simulation policy training and are not well suited for real-world performance evaluation. Second, current platforms do not model the dynamics of the physical environment on-the-fly, limiting their potential for the development of robot navigation policies in challenging environments. Third, the associated real-world evaluation environments are both expensive and limited to large interior scenes. This limits access to the physical environments, subsequently making sim-to-real policy evaluation difficult. Fourth, current simulations rely on third party robots for policy training and evaluation, which can be costly while limiting the robot’s customizability and optimisation for the associated task and environment. To tackle these issues, we propose a novel robot-centred smart digital twin framework called Terra. Terra leverages the power of digital twin (DT) systems (digital replicas of the physical world), commonly used for monitoring and evaluating physical systems. Terra includes a comprehensive DT representation which encodes useful real-time dynamics of both the physical world and the robot agent deployed therein. The DT is updated through a multi-view, multi-modality perception module, which obtains high-level semantics, delivering a precise description of the current status of the physical environment and the robot agent. By mapping the perceived results to the virtual replica of the physical environment, Terra actively updates the action policy and sends it back to the agent, forming an integral and real-time information feedback loop. In practice, to help demonstrate the proposed framework's effectiveness and feasibility, we deliberately set up a challenging unordered physical environment with many obstacles and a custom robot tasked with a simple navigation task. Our physical scene is inexpensive and small, while our novel low-cost robot is customised for the environment. Experiment results show that the proposed Terra framework successfully enables the robot to accomplish a simple navigation task, demonstrating its suitability for real-world robot performance evaluation.
See less
See moreThe development of artificial intelligence (AI) based robot policies such as embodied visual AI is a rapidly emerging field that promises to fully automate robot sensing and control, reducing the need for expert knowledge and information about system dynamics. The recent surge in the availability of digital simulation environments has helped to alleviate the data scarcity problem faced by AI training and yielded many tasks for embodied AI to explore. In contrast to conventional computer vision databases (e.g. ImageNet, COCO), they provide interactive environments tailored to the training of embodied AI agents (e.g. virtual robots). Despite their success, there remain several unaddressed issues. First, these environments primarily focus on in-simulation policy training and are not well suited for real-world performance evaluation. Second, current platforms do not model the dynamics of the physical environment on-the-fly, limiting their potential for the development of robot navigation policies in challenging environments. Third, the associated real-world evaluation environments are both expensive and limited to large interior scenes. This limits access to the physical environments, subsequently making sim-to-real policy evaluation difficult. Fourth, current simulations rely on third party robots for policy training and evaluation, which can be costly while limiting the robot’s customizability and optimisation for the associated task and environment. To tackle these issues, we propose a novel robot-centred smart digital twin framework called Terra. Terra leverages the power of digital twin (DT) systems (digital replicas of the physical world), commonly used for monitoring and evaluating physical systems. Terra includes a comprehensive DT representation which encodes useful real-time dynamics of both the physical world and the robot agent deployed therein. The DT is updated through a multi-view, multi-modality perception module, which obtains high-level semantics, delivering a precise description of the current status of the physical environment and the robot agent. By mapping the perceived results to the virtual replica of the physical environment, Terra actively updates the action policy and sends it back to the agent, forming an integral and real-time information feedback loop. In practice, to help demonstrate the proposed framework's effectiveness and feasibility, we deliberately set up a challenging unordered physical environment with many obstacles and a custom robot tasked with a simple navigation task. Our physical scene is inexpensive and small, while our novel low-cost robot is customised for the environment. Experiment results show that the proposed Terra framework successfully enables the robot to accomplish a simple navigation task, demonstrating its suitability for real-world robot performance evaluation.
See less
Date
2021Rights statement
The author retains copyright of this thesis. It may only be used for the purposes of research and study. It must not be used for any other purposes and may not be transmitted or shared with others without prior permission.Faculty/School
Faculty of Engineering, School of Computer ScienceAwarding institution
The University of SydneyShare