A graphical visualisation of latent spaces, the newtonian latent space is smooth, while others are discontinuous.

NewtonianVAE: Proportional Control and Goal Identification from Pixels via Physical Latent Spaces

Miguel Jaques, Tim Hospedales and I recently published a CVPR paper (best paper nominee) on learning latent dynamics models for proportional control from pixels. Miguel has written a great blog post about this idea. This work builds on prior work around hybrid system identification using proportional controllers (Burke et. al, Corl 2019) and programming induction […]

A reward signal inferred by PTR, slowly increasing, as ultrasound quality improves.The inferred reward correlates strongly with human image ratings.

Robotic ultrasound scanning: Learning from exploratory demonstrations using probabilistic temporal ranking

Recently, we have been exploring the use of time as a supervisory signal for learning from demonstration. As an example use case, we considered ultrasound scanning, where a technician is required to search for a scanning position and contact force that produces an optimal image. We propose a probabilistic temporal ranking (PTR) model that allows […]

Physics as inverse graphics model

Stronger inductive biases for deep learning

Standard architectures for neural networks have numerous problems with interpretability, flexibility and generalisation. I believe that this is in large part due to a lack of stronger inductive biases in models and architectures, and have recently been pushing (see my job talk at Monash) to include stronger biases in deep learning models. Switching controller front-ends […]

On inter-sectional bias and its remediation in data-driven models

It’s well established that machine learning has a problem with bias. Our datasets reflect the inequalities and prejudices of our daily lives, and the models we train and deploy exacerbate these even further. We even notice this in robotics (despite being generally removed from people), where localisation and mapping systems perfected in green European settings […]

Switching density networks for hybrid control systems

Hybrid system identification can be particularly challenging, particularly in the context of visuomotor control. We introduce switching density networks (SDNs), which can be used to identify switching control systems in an end-to-end learning fashion from demonstration data. We show that SDNs, when paired with a general purpose family of proportional-integral-derivative control laws, can identify the […]

Inducing explainable robot programs

End-to-end learning is able to solve a wide range of control problems in robotics. Unfortunately, these systems lack interpretability and are difficult to reconfigure if there is a minor task change. For example, a robot inspecting a range of objects needs to be retrained if the order of inspection changes. We address this by inducing […]

Finding interesting images

I obtained a young researcher’s establishment grant from the CSIR to investigate what makes images interesting and to find algorithms that flag images of potential interest to users. At present, I am exploring the use of pairwise image comparisons to estimate image interest. These interest estimates can be improved for video by imposing temporal smoothness constraints. Further improvements are obtained […]

Particle filter shoreline tracking

Coastal scientists are interested in tracking the surf-zones over time, so I spent some time exploring curve models to accomplish this. A generative curve model, positioned using a motion model driven by tidal dynamics, is able to track the changing shape and position of a stretch of coastline in Fish Hoek. The tracker struggles in […]

Automatic conference organisation

I’ve been involved in the organisation of the PRASA-Robmech conference a few times, and it’s usually a painful experience. In an attempt to simplify things, we dashed together an automatic conference scheduler, which also gave me a chance to explore latent topic models. The scheduler takes a conference schedule skeleton and full text papers, then uses latent […]

Pose estimation for human-robot interaction

Human-robot interaction using gesture recognition typically requires that the 3D pose of a human be tracked in real time, but this can be challenging, particularly when only a single, potentially moving, camera is available. We use a mixture of random walks model for human motion that allowed for fast Rao-Blackwellised tracking, and provide a useful mechanism to […]

Gesture recognition

Gesture recognition can be a valuable interface for human-robot interaction, but typical approaches to gesture recognition use pre-defined dictionaries of signs for recognition. This is often not particularly intuitive, so we explored the pantomimic gestures for human robot interaction. Pantomimic gestures are those that mimic a desired behaviour or action, and can be used to improve […]

Tracked robot control

Tracked robots can traverse a wide range of terrains, but can be hard to control automatically in uneven terrain because of track slip. Improved robot control requires a real-time estimate of slip, but this can be difficult to obtain without good forward velocity measurements. This project showed that a slip estimate can be obtained using only […]

Robot planning in crowds

We built a pamphlet distributing robot at the CSIR, which worked well in testing, but tended to get stuck in crowded conference venues. A planning strategy was required to help the robot escape crowds. Unfortunately, we couldn’t rely on localisation, as this proved infeasible in crowded indoor environments. An honours student I co-supervised experimented with random […]

Robot localisation

Robot localisation is crucial for field robot deployment, but can be challenging. Global positioning systems can be noisy and affected by multi-path in urban canyon environments. For outdoor mobile robots, we can solve this by fusing GPS and robot odometry, but in indoor environments alternative solutions are required. Simultaneous localisation and mapping (SLAM) can address […]

Human (and cereal) following robots

This project focused on feature-based object recognition and tracking using a single camera for a target-following mobile robot. Robot controls are generated so as to maximise the chances of successfully detecting and tracking the object of interest while navigating. This was a particular challenge at the time (2010 – before convnet fame) as object recognition approaches were […]