Manipulation of Natural Deformables: Simulation, Inference, and Learning

Abstract

Over the past decades, machine learning has transformed robotic manipulation, enabling autonomous systems to perform complex rigid body tasks. Yet, this progress has not translated to deformable objects, characterised by high-dimensional state spaces and non-linear dynamics. Handling natural variants of deformables, such as tree branches and plant stems, is an even greater challenge, exacerbated by their non-uniform geometry and asymmetric dynamics. Viable solutions must quantify uncertainties from imperfect sensors and inaccurate models to account for the probabilistic nature of the world. To address these challenges, we present an ensemble of frameworks to model the intricate plant topology, estimate dynamic parameters, and learn efficient control strategies.

First, we present a simulation-driven inverse inference approach to model the uncertain dynamics of plant stems under deformation, in a real-to-sim context. Framing system identification as a Bayesian inference problem, we estimate the multi-modal, spring parameter posterior density. Our non-parametric method can incorporate biological assumptions, quantify the estimation uncertainty, and tolerate contact instabilities.

Next, we train a non-prehensile, contact-sensitive, reinforcement learning policy to interact with tree branches, in a sim-to-real setting. The spring abstractions are integrated with a parametric L-system model to build a procedural forest. The novel proprioceptive approach transfers zero-shot from simulation to real, manipulating stems with unseen geometry and dynamics, exhibiting unique contact strategies.

Finally, we expand the sim-to-real strategy, emphasising manipulation with the whole arm, treating the deformables as a collection. To learn a computationally efficient policy, we leverage a distributional state representation. Our blank slate policy learning approach can autonomously discover creative de-occlusion strategies for clearing electrical power lines of overhanging foliage.