Background

To improve the performance of dialogue systems, analyses of interactions are an important source of knowledge. When dialogue systems are deployed, the interaction can be logged for later analysis. Talkamatic AB are about to begin collection of logs resulting from dialogue system interactions, which will be available to the student.

Task

The task is to build a toolbox for analysing logs of dialogue system interactions and presenting results from such analyses. Examples of relevant analysis features include:

• speech recognition error % (requires transcribing user utterances)
• task completion time
• number of turns
• dialogue complexity (e.g. number of subdialogues, degree of subdialogue embedding)
• number of grounding subdialogues
• estimated task success (measured by observing behaviour after system response)

Such basic analysis dimensions can then be used to detect and diagnose problems with systems which need fixing.

Supervisors

Staffan Larsson, Dialogue Technology Lab, and Talkamatic AB.