• research_dialogue_lab


End-of-utterance detection

The current dialogue system used at the Dialogue Lab, GoDiS, depends on cut-off values to control turn-taking. This means that when the user has not spoken for a period of time, the system assumes the user is finished and takes the turn. This can lead to both interruptions and unnecessary long waits for the user.

To solve this problem, the system has to be able to detect when a speaker is finished or when he is just making a pause within his utterance. If the system can reliably detect the users end-of-utterance, it can take the turn more rapidly when the user is finished (and avoid interrupting the user when he/she is not finished).

To detect end of utterance, we assume that the system needs information from several sources: syntactic information, prosodic information and also information state. We will create a statistical language model for end-of-utterance detection, using machine learning. For this we will use the Weka toolkit.

We will attempt to create a model that allows the system to differentiate between user pauses within an utterance, and user pauses at the end of an utterance.

Funding: CLT internal.

Duration: August 2011 - October 2012

Researchers: Kristina Lundholm Fors, Staffan Larsson (supervisor).


SIMSI - Safe In-vehicle Multimodal Speech Interfaces

External project site

Driver distraction is a common cause of accidents, and is often in turn caused by the driver interacting with technologies such as mobile phones, media players or navigation systems. A Multimodal HMI system complements traditional human-machine interaction modalities (visual output and haptic input) with spoken interaction. Speech solutions generally aim to increase safety but immature solutions may end up distracting the driver and decreasing safety.

In the SIMSI project, we  aim to integrate an existing safety-oriented multimodal HMI system based on academic research into a commercial-grade HMI platform and uses this integrated system for research on dialogue strategies for cognitive load management and integrated multimodality. We expect to achieve an HMI solution which can be reliably shown to increase safety by reducing distraction, cognitive load and head-down time considerably when compared to other state-of-the-art in-vehicle interaction models. The HMI will be evaluated in simulators and real traffic.

Funding agency: Vinnova (FFI programme)

Partners: Talkamatic AB, Mecel AB

Contact person at CLT: Staffan Larsson

Dialogue Technology for Second Language Learning

In his thesis entitled "The Virtual Language Teacher: Models and
applications for language learning using embodied conversational agents"
[1], Preben Wik presents a framework for computer assisted language
learning using a virtual language teacher. At least one implementation
is also available at [2].

The purpose of the project is to investigate the feasibility of
implementing some of Wik's applications on an platform such as Voxeo or

Supervisor: Torbjörn Lager

[1] <http://www.speech.kth.se/~preben/thesis/thesisPrebenWik.pdf>
[2] <http://www.speech.kth.se/ville/swell.html>


Building an Alternate Reality Game on a platform for unified communication

According to Wikipedia, an Alternate Reality Game (ARG) is an
interactive narrative that uses the real world as a platform, often
involving multiple media and game elements, to tell a story that may be
affected by participants' ideas or actions. ARGs generally use
multimedia, such as telephones, email and mail but rely on the Internet
as the central binding medium. [1]

The purpose of the project is to design and prototype an ARG using an
existing platform for unified communication [2] such as Voxeo Prophecy
or Tropo and standards such as VoiceXML, SRGS, SSML, CCXML and SCXML. It
is suggested that the game is to be played in the house in which FLOV

Supervisor: Torbjörn Lager (and possibly Johan Roxendal)

[1] <http://en.wikipedia.org/wiki/Alternate_reality_game>
[2] <http://en.wikipedia.org/wiki/Unified_communications>

Developing and evaluating a TDM app (2016)


To develop a voice interface to an Android smartphone application.


Voice interfaces give users the possibility to interact with a device without using their eyes or hands. This can be particularily useful in situations where interaction with a device would otherwize interfere with other tasks. A typical example is driving, in which the simultaneous use of a screen-based device such a smartphone can constitute a direct danger. This project focuses in the potential benefits of voice enabled smartphone apps in driving and similar scenarios.

Problem description

The problem consists of developing a spoken dialog interface to an Android application. The user should be able to switch freely between touch-screen and voice interaction. To facilitate the development of this multimodal interface, Talkamatic Dialog Manager (TDM) will be used.

The problem mainly consists of the following tasks:

  • Develop a Use Case for the application; this may involve market surveys and interviews
  • Write a TDM device resource, which communicates with the app functionality (Python and Android SDK)
  • Write a TDM domain and ontology specification, describing the structure of the app (XML)
  • Write a TDM grammar, describing the verbal phrases used by the user and the app (XML, Grammatical Framework)
  • Evaluate the app


  • Python
  • Grammar formalisms
  • XML

Experience of Android app development is valuable but not required.


Staffan Larsson, FLoV, together with Talkamatic AB. Talkamatic is a university research spin-off company based in Göteborg.


A talking and playing robot for children with communicative disabilities

En talande och lekande robot för barn med funktionshinder

Project description (in English)

Lekbot is a collaboration between DART, Talkamatic and the Dept. of philosophy, linguistics and theory of science, University of Gothenburg. It is funded by VINNOVA, and runs from March 1, 2010 to August 31, 2011.

The project uses current theory and technology in human communication, computer communication and dialogue systems to develop a toy that is fun and interesting for young people with communicative disabilities. The toy helps the development of dialogical communication, an area that is normally problematic for these children. Children with severe disabilities often have few opportunities to play independently and to interact on equal terms with children without disabilities, and here Lekbot enables children with and without disabilities to interact and learn from each other.

The Lekbot toy developed in the project is a radio-controlled robot that can be used by children with severe physical and/or cognitive disabilities, such as cerebral palsy or autism. The robot is controlled by the child through touch-screen symbols. The symbols are translated into spoken language, so that the touch screen "talks" to the robot and acts as the child's voice. The robot can, in turn, talk to the child using spoken language, and the child can again answer using the touch screen.

The Lekbot project is a development of TRIK.

The robot is built using Lego Mindstorms NXT, and the dialogue system is developed using GoDiS. The project is supported by Acapela, whose speech synthesis is used for both the touch screen and the robot.


  • A leaflet describing Lekbot (in Swedish): [lekbot-broschyr.pdf]
  • Abstract for poster at the third Swedish Language Technology Conference (SLTC 2010), Linköping 28-29th October 2010: [lekbot-sltc2010.pdf]
  • Paper describing Lekbot and our evaluation, presented at SLPAT 2011, 2nd Workshop on Speech and Language Processing for Assistive Technologies, Edinburgh 30th July 2011: [lekbot-slpat2011.pdf]
  • A news story from Swedish TV4, June 21st 2011: [tv4play, link apparently dead]
  • A news story from Swedish SVT Västnytt, May 8th 2014: [svt.se]

Contact: lekbot "at" talkamatic.se

Project description (in Swedish)

Lekbot är ett samarbete mellan DART, Talkamatic och Inst. för filosofi, lingvistik och vetenskapsteori, Göteborgs universitet. Det finansieras av VINNOVA, och pågår 1 mars 2010 till 31 augusti 2011.

Projektet utnyttjar aktuell teknik inom mänsklig kommunikation, datakommunikation och dialogsystem för att förse unga människor med kommunikationssvårigheter med en rolig och spännande leksak, som dessutom hjälper dem att utveckla sin förmåga inom ett område där deras funktionshinder i vanliga fall hindrar dem, nämligen dialog. För barn med stora rörelsehinder finns få möjligheter att leka självständigt och att samspela på lika villkor med barn utan funktionshinder. Genom en leksak som barn kan använda oavsett funktionshinder eller ej, ges barnen möjlighet att samspela och lära av varandra.

I Lekbot utvecklas en radiostyrd robot som kan användas av barn och ungdomar med svåra fysiska och /eller kommunikativa funktionshinder, såsom cerebral pares eller autism. Roboten styrs av barnet genom att peka på symboler på en pekskärm. Symbolerna översätts till talat språk, så att pekskärmen "pratar" med roboten och berättar vad barnet vill. Roboten å sin sida kan ställa frågor tillbaka, som barnet svarar på genom att peka på symboler.

Lekbotprojektet är en vidareutveckling av TRIK.

Roboten är byggd med Lego Mindstorms NXT, och dialogsystemet utvecklas i GoDiS. Projektet stöds av Acapela, vars talsyntes används för både pekskärmen och roboten.


  • En broschyr som beskriver Lekbot: [lekbot-broschyr.pdf]
  • Abstract för poster på den tredje Swedish Language Technology Conference (SLTC 2010) Linköping 28-29 oktober 2010 (på engelska): [lekbot-sltc2010.pdf
  • En artikel som beskriver Lekbot och vår evaluering, presenterad på SLPAT 2011, 2nd Workshop on Speech and Language Processing for Assistive Technologies, Edinburgh 30 juli 2011: [lekbot-slpat2011.pdf]
  • Ett nyhetsinslag i TV4, 21 juni 2011: [tv4play, länken är död]
  • Ett nyhetsinslag i SVT Västnytt, 8 maj 2014: [svt.se]

Kontakt: lekbot "at" talkamatic.se

GPCC: Personcentrerad vård vid pediatrisk diabetes (och obesitas)

Detta projekt är inte främst ett språkteknologiskt projekt. Däremot är insamling, transkribering och kodning av samtal en del av projektet, och dessa aktiviteter utförs inom DTL:s ramar.

Project coordinator:
Ewa Wikström, Företagsekonomiska institutionen.

Scientific management leaders:
Gun Forsander, Institutionen för Kliniska Vetenskaper/Barnmedicin, Inga-Lill Johansson, Företagsekonomiska institutionen, Christian Munthe, Institutionen för Filosofi, Lingvistik och Vetenskapsteori, Marianne Törner, Arbets- och Miljömedicin.

Objective (including the person-centred angle):
Det övergripande syftet med projektet är att genom tvärvetenskapliga team ta fram och utvärdera individ- och familjeanpassade behandlings¬program som identifierar och använder patientens syn på sin situation i form av behov, resurser, vilja och preferenser. Detta ger möjlighet till individualisering och optimering av rehabilitering och behandling med målet att öka patienttillfredsställelsen och effektiviteten i behandlingen av barn och ungdomar med diabetes och fetma som tillhör minoritetsgrupp.

Expected results/Specific research questions:
Hur kan familjens och patientens syn på sin situation i form av behov, resurser, vilja och preferenser identifieras och användas i behandling och vård? Finns det etniska och kulturella skillnader i detta avseende?
Hur kan individ och familjeanpassade alternativ utvecklas i tvärvetenskapliga vårdgivarteam?
Hur kan patientberättelsen ligga till grund för en förändrad vårdpraktik avseende samverkan mellan sjukhusvård, öppenvård och förskola/skola?

Method (-s):
Observations- och intervjumetoder används för datainsamling. Djupintervjuer med patienter och deras anhöriga. Videoinspelning av samtal/möten mellan patient/anhöriga och vårdpersonal. Förändringslaboratorier används för att åstadkomma önskad förändring av vårdpraktiken.

Timetable – when will the project start and be finalised, milestones:
Pilotstudie 2010 april – november
Huvudstudie 2010 december – 2012
Uppföljning av effekter fortgår 2013
Slutrapport i början av 2014

Project homepage




Semantic analysis of interaction and coordination in dialogue

VR project 2009-1569, 2010-2012

Robin Cooper, University of Gothenburg
Jonathan Ginzburg, King's College London
Staffan Larsson, University of Gothenburg

The aim of the project is to integrate aspects of traditional model theoretic semantics developed in the main for sentence semantics and discourse with recent developments in dialogue analysis.  The project aims to give a theoretical account of how dialogue participants manage to remain coordinated during relatively intricate linguistic interaction. We aim to synthesize work on a key mechanism of interaction,  namely repair – clarification questions (CQs), self-corrections, hesitations etc – with work in formal semantics that has modelled many of the central elements of natural language meaning, such as quantifier terms, anaphora, and attitude reports. This is a two way street: the perspective from interaction will provide useful extra cognitively--based evidence for the field of semantics -- a domain overfull with theories underdetermined by evidence, while helping solve age--old puzzles; conversely, in tackling the intricate linguistic phenomena analyzed by semanticists theories of interaction can aspire to provide comprehensive theories of cognition.  We will focus on dialogic aspects of some classical concerns of formal semantics: quantification, anaphora and intensionality.

Project homepage

Semantic Coordination in Dialogue

In the last decade or so problems related to linguistic, and especially semantic, coordination have received increased attention from several strands of research in psycholinguistics, computational linguistics and discourse analysis, among other disciplines. Work on semantic coordination in psycholinguistics has produced very interesting results and some informal theory; we want to use these results as a basis for a general formal theory of semantic coordination.

The project Semantic Coordination in Dialogue (2008-2011) is funded by the Swedish Tercentenary Foundation (Project P2007/0717)

Project homepage


A talking and drawing robot for children with communicative disabilities

This project is a cooperation between DART (Centre for Alternative and Augmentative Communication, and Assistive Technology; in Swedish: "Kommunikations- och dataresurscenter för personer med funktionshinder"), and the University of Gothenburg (department of Philosophy, Linguistics and Theory of Science). The project was financed by the Promobilia Foundation and Magn. Bergvall's Foundation.

TRIK started in September 2008 and ended in December 2009. The main contact person is Peter Ljunglöf. It was followed by the Lekbot project.

Abstract (in English)

In this project we develop and evaluate setup involving a communication board (for manual sign communication) and a drawing robot, which can communicate with each other via spoken language. The purpose is to help children with severe communication disabilities to learn language, language use and cooperation, in a playful and inspiring way. The communication board speaks and the robot is able to understand and talk back. This encourages the child to use the language and learn to cooperate to reach a common goal, which in this case is to get the robot to draw figures on a paper.

Abstract (in Swedish)

I detta projekt utvecklar vi och utvärderar en teckentavla och en ritande robot som kan kommunicera med varandra med hjälp av talat språk. Syftet är att hjälpa barn och ungdomar med svåra språkliga funktionshinder att tillgodogöra sig språket, språkliga regler och språkligt samarbete, på ett lekfullt och inspirerande sätt. Genom att teckentavlan pratar och roboten förstår och kan svara själv, så uppmuntras barnet att använda språket och lär sig att samarbeta för att nå ett gemensamt mål, vilket i detta fall är att få roboten att rita figurer.


  • Two (normal developed) children starting to play with TRIK: [youtube or download quicktime, 5.6MB]
  • The same session, 10 minutes later: [youtube or download quicktime, 10.3MB]



The robot has been presented at the following workshops and conferences:

  • NordiCHI'08 Robot Workshop, 17 oktober 2008, Lund: [www]
  • Nodalida'09, 17th Nordic Conference on Computational Linguistics, 14–16 maj 2009, Odense, Danmark: [www]
  • 8:e Västsvenska Kommunikationskarnevalen, 1–2 juni 2009, Göteborg: [www]
  • DiaHolmia'09, 13th Workshop on the Semantics and Pragmatics of Dialogue, 24–26 juni 2009, Stockholm: [www]
  • ID-dagarna 2009, 7–9 oktober 2009, Stockholm: [www]


TRIK has been mentioned in the press too:

  • Vårt Göteborg, 1 juni 2009: [www]
  • Metro Teknik, 19 november 2009: [www]


Building instructions for the TRIK robot, together with some unsorted Python utilities for connecting to and drawing with the robot from a Mac computer:

  • [generic-turtle-controller.zip] — note that there is no program that can be run directly in this archive, you have to make a wrapper program yourself. For this you need to know at least some Python programming. The building instructions can be viewed in Lego Digital Designer.