• Home
  • Learning language and perception with a robot

Learning language and perception with a robot


The task of the project is to learn a mapping between natural language descriptions on one hand and sensory observations and commands issued to a simple mobile robot (Lego NX) using machine learning. The project would involve building a corpus of descriptions paired with actions - one person is guiding the robot and another person is describing. Multimodal ML models would then be built from this corpus both to predict a description, action or perceptual observation. Finally, the models should be integrated with a simple dialogue manager with which humans can interact and test the success of learning in context.

The system should be implemented in ROS (Robot Operation System) which provides access to sensors and actuators of the robot and allows writing new models in a simplified (well-organised) manner in Python.

Contributions/possible research directions of this thesis:

  • to examine to what extend and for what situations Lego NX robot can be used for learning multimodal semantics;
  • to examine whether bag-of-features approach (features being both linguistic and perceptional/action features) can be used to learn multimodal semantic representations;
  • examine how such models can be integrated with dialogue;
  • examine ML techniques that would actively learn/update the models through the interaction with a user (clarification, correction).

Supervisors: Simon Dobnik and possible others from the Dialogue Technology Lab

To the top

Page updated: 2014-11-12 15:11

Send as email
Print page
Show as pdf