A team of students from the School of Computing Science have taken second place in the Alexa Prize TaskBot Challenge 2.

The team will take home $100,000 in prize money after judges appraised their work in building GRILLBot, an AI-powered multimodal conversational assistant, against others developed by finalists from universities in the USA and Portugal.

Their success follows a first-place prize in the inaugural Alexa Prize TaskBot Challenge competition the previous year, in which the team won $500,000. A team from NOVA School of Science and Technology (FCT NOVA) in Portugal won first place this year.

GRILLBot is a digital assistant that guides users through complex real-world tasks. It helps users with cooking and everyday tasks in the home.

During the finals event, judges cooked and performed DIY tasks successfully with the help of GRILLBot. GRILLBot adapts instructions to accommodate the user’s environment and expertise and responds to unseen questions, making it suitable for real-world task assistance and learning new skills.

During the Alexa Prize TaskBot Challenge 2, a key aspect was to go beyond just voice to effectively incorporate visual aids fluently. The Alexa Prize TaskBot Challenge is the first conversational AI challenge to incorporate multimodal customer experiences. In addition to verbal instructions, customers were also presented with step-by-step instructions and visuals to enhance the user experience.

Innovative ideas on improving the presentation of visual aids, as well as the coordination of visual and verbal modalities, were part of the judging criteria.

PhD student Sophie Fischer is the GRILL team’s leader. She said: “Working on the TaskBot Challenge 2 gave us the unique opportunity to develop and deploy cutting-edge large language models (LLMs). We learned that it’s not just about model size or improved training, but about using models in new and creative ways to help people.”

GRILLBot uses LLMs to generate recipes and tasks specific to each user in a controlled and safe environment to take the user’s expertise and equipment into account. Not only textual instructions are AI-generated, but also images to match the written and spoken instructions. This enables the agent to provide the user with recipes that suit the personal dietary requirements and preferences.
 
The GRILL team also uses LLMs for live task modification and extension during the conversation. For example, GRILLBot can adapt a recipe that the user has started cooking already based on the user’s preferences and available ingredients. Furthermore, if the user has not performed a cooking skill before, more detailed relevant instructions are provided fluently.
 
To polish off the experience, GRILLBot can respond to user questions and engage in domain-specific small talk. It also tells jokes and fun facts and asks follow-up questions. GRILLBot’s fluency and task-specific knowledge made the experience stand out and allowed the team to place second.

The team was advised by Dr Jeff Dalton, the leader of the Generalized Representation and Information Learning Lab (GRILL) research group. As of October 2023, the research group has moved to the University of Edinburgh.

Dr Dalton said: “I’m proud of the team rising to meet new AI research challenges in the second year of the Amazon Alexa TaskBot Challenge. This includes deploying LLMs for knowledge augmentation, social chat and improving task-specific QA. Placing second demonstrates the world-leading talent and determination of the students and their ability to engage with research.”

Each university selected for the challenge received a $250,000 research grant, Alexa-enabled devices, free Amazon Web Services (AWS) cloud computing services to support their research and development efforts, access to Amazon scientists, the CoBot (conversational bot) toolkit, and other tools such as automated speech recognition through Alexa, neural detection and response generation models, conversational datasets, and design guidance and development support from the Alexa Prize team.

Throughout the competition, customers throughout the US engaged with the university teams’ Taskbots by saying, “Alexa, let’s work together.” After initiating the interaction, customers received a brief message informing them that they were interacting with an Alexa Prize university Taskbot before being randomly connected to one of the participating Taskbots.

After exiting the conversation with the Taskbot, the customer was prompted for a verbal rating, followed by an option to provide additional feedback. The interactions, ratings, and feedback were shared with the teams to help them improve their Taskbots. Customer ratings were also used to determine which university teams advanced to the semifinals and finals.

Success in the previous TaskBot Challenge required teams to address many difficult AI obstacles, including knowledge representation and inference, commonsense and causal reasoning, and language understanding and generation.
The GRILL team are:

·       Sophie Fischer (Team Lead), PhD in Computing Science 
·       Niklas Tecklenburg, Bsc Informatics
·       Philip Zubel, BSc (Hons) Computing Science
·       Ekaterina Terzieva, MSc Computer Science
·       Daniel Armstrong, MSs Information Technology
·       Eva Kupcova, BSc (Hons) Computing Science
·       Carlos Gemmell, PhD in Computing Science
·       Federico Rossetto, PhD in Computer Science 
·       Iain Mackie, PhD in Computing Science