Physics as inverse graphics model

Stronger inductive biases for deep learning

Standard architectures for neural networks have numerous problems with interpretability, flexibility and generalisation. I believe that this is in large part due to a lack of stronger inductive biases in models and architectures, and have recently been pushing (see my job talk at Monash) to include stronger biases in deep learning models.

Switching controller front-ends

For example, by embedding known controllers in a model, in addition to knowledge of a switching structure, we gain better performance in settings where hybrid control is required, along with greater interpretability.

M Burke, Y Hristov, S Ramamoorthy, Switching Density Networks for Hybrid System Identification,  Conference on Robot Learning (CoRL) 2019. (arxiv link)

Relational representations

Similarly, by autoencoding with light supervision, we can ground perception networks in symbolic concepts that align with natural language for greater interpretability, while allowing for planning and symbolic reasoning.

Y Hristov, D Angelov, M Burke, Alex Lascarides, S Ramamoorthy, Disentangled Relational Representations for Explaining and Learning from Demonstration,  Conference on Robot Learning (CoRL) 2019. (arxiv link)

Video to physical parameters

The same idea can allow for parameter estimation from video, and the incorporation of physical dynamics into a model.

M Asenov, M Burke, D Angelov, T Davchev, K Subr, S Ramamoorthy, Vid2Param: Modelling of Dynamics Parameters from Video, Robotics and Automation Letters (RA-L) (arxiv link).

Integrated physics

The approaches above inject constraints through training, but for generalisation, we may need even stronger priors built into models. Our work on physics-as-inverse graphics does this by including differentiable physical equations in the model, and exhibits much stronger extrapolation performance.

M Jacques, M Burke, T Hospedales, Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Video, International Conference on Learning Representations (ICLR 2020) (open review link)


True sequence

Physics as Inverse Graphics

Physics as Inverse Graphics

Interaction networks, inverse graphics



Switching density networks for hybrid control systems

Hybrid system identification can be particularly challenging, particularly in the context of visuomotor control. We introduce switching density networks (SDNs), which can be used to identify switching control systems in an end-to-end learning fashion from demonstration data.

We show that SDNs, when paired with a general purpose family of proportional-integral-derivative control laws, can identify the pump, spin and balance controllers required to keep an inverted pendulum upright (see header). We also use them to identify the joint angle goals that make up an inspection task on a PR2 robot, and those needed to open a suitcase.

Switching density networks are particularly useful for options learning, as the controllers identified using these can be re-used elsewhere. Importantly, by embedding structure into the network, SDNs become more interpretable, and allow for hierarchical learning that is not possible with their closely related counterparts, mixture density networks.

M Burke, Y Hristov, S Ramamoorthy, Switching Density Networks for Hybrid System Identification,  Conference on Robot Learning (CoRL) 2019. (arxiv link)

Inducing explainable robot programs

End-to-end learning is able to solve a wide range of control problems in robotics. Unfortunately, these systems lack interpretability and are difficult to reconfigure if there is a minor task change. For example, a robot inspecting a range of objects needs to be retrained if the order of inspection changes.

We address this by inducing a program from an end-to-end model using a generative model consisting of multiple proportional controllers. Inference under this model is challenging, so we use sensitivity analysis to extract controller goals and gains from the original model. The inferred controller trace (a sequence of controller goal states) is then simplified and controller specific grounding networks trained to predict controller goals for visual inputs, producing an interpretable and reconfigurable program describing the original learned behaviour.

Michael Burke, Svetlin Penkov, Subramanian Ramamoorthy, From explanation to synthesis: Compositional program induction for learning from demonstrationRobotics: Science and Systems (R:SS), 2019. arXiv link

Tracked robot control

Tracked robots can traverse a wide range of terrains, but can be hard to control automatically in uneven terrain because of track slip. Improved robot control requires a real-time estimate of slip, but this can be difficult to obtain without good forward velocity measurements. This project showed that a slip estimate can be obtained using only a rate gyroscope, allowing for improved path following control.

This work has been used by the CSIR for a mine safety robot and in autonomous 3D mapping projects. Work is ongoing in this domain with a Masters student, Ditebogo Masha, looking to apply slip estimation to terrain classification and characterisation problems.

M. Burke, “Path-following control of a velocity constrained tracked vehicle incorporating adaptive slip estimation,” 2012 IEEE International Conference on Robotics and Automation, Saint Paul, MN, 2012, pp. 97-102.

D. Masha, M. Burke, and B. Twala, “Slip estimation methods for proprioceptive terrain classification using tracked mobile robots.” Pattern Recognition Association of South Africa and Robotics and Mechatronics (PRASA-RobMech), 2017. IEEE, 2017.

D. Masha, M. Burke, “Importance Sampling Forests for Location Invariant Proprioceptive Terrain ClassificationArtificial Intelligence Research – First Southern African Conference for AI Research, SACAIR 2020, in Communications in Computer and Information Science, Springer, 2020.