Talks and presentations

Multimodal Conversational AI Agents Towards Smarter Assistant

February 07, 2024

Invited Talk, Microsoft Research, Cambridge, United Kingdom

Artificial Intelligence (AI) is playing an increasingly vital role in scientific research, particularly in enhancing Human-Computer Interaction (HCI) through the development of smarter agents. Conversational AI agents, in particular, have demonstrated considerable potential in enriching the user experience within multimodal immersive systems, facilitated by large language model (LLM)-powered autonomous agents. In this talk, drawing from my practical expertise, we will delve into the application of large foundation models and frameworks in natural language processing (NLP) and information retrieval (IR) tasks. Furthermore, we will address the revision of the frontier research and the potential for fostering interdisciplinary collaborations between researchers and practitioners in the realms of NLP and HCI.

Multimodal Dialogue Agents Towards Intelligent XR

December 04, 2023

Invited Talk, Utrecht University, Utrecht, Netherlans

Interactive personal assistants have been playing a leading role in recent applications in extended reality. In this talk, we introduce our recent work related to dialogue agents and vision-language models and discuss the challenges and opportunities in multlimdal dialoge agents.

Maximising efficiency in NLP model training and XR environments

November 29, 2023

Invited Panel Talk, Immersive Tech Week, Rotterdam, Netherlands

In the realm where Natural Language Processing (NLP) intersects with immersive technology, a narrative unfolds. NLP, driving change in digital interactions, has reshaped our virtual landscape. However, the resource-intensive nature of NLP models raises concerns about environmental impact, energy consumption, and scalability. Extended Reality (XR) environments are now integrated with NLP tools, influencing virtual meetings, training simulations, and immersive experiences. The demand is clear: efficiency, optimisation, and seamless fusion of language models to minimise latency and enhance the user experience. The roundtable assembled a group of experts to explore strategies for optimising training pipelines, addressing environmental concerns, and developing sustainable approaches. The focus extended to improving efficiency, reducing latency, and maximising the user experience. More information see https://voxreality.eu/voxreality-unleashes-xr-revolution-nlp-mastery-and-tech-wizardry-take-center-stage-at-immersive-tech-week-2023/

Language Model Powered Dialogue Agents and Virtual KB

September 19, 2023

Invited Talk, Sony AI, Barcelona, Spain

In this talk, we will explore the emerging integration of advanced language models with virtual knowledge bases (KBs) to develop enhanced dialogue agents capable of providing more informed and contextually relevant interactions. As AI continues to evolve, the synergy between deep learning-based language models and structured knowledge representation plays a critical role in crafting dialogue agents that exhibit higher levels of understanding and functionality. We begin by examining the current landscape of language model-powered dialogue systems, focusing on their ability to generate human-like responses in various conversational contexts. We then introduce the concept of a virtual KB, a dynamic, structured repository of information that dialogue agents can query and update in real-time during interactions. The core of the discussion revolves around the methods and technologies used to seamlessly integrate these virtual KBs with language models, enabling more informed, accurate, and context-aware interactions. By leveraging case studies and recent advancements, we demonstrate how this integration facilitates a significant leap in handling complex user inquiries and maintaining coherent long-term conversations. Finally, the talk addresses potential challenges such as privacy concerns, data integrity, and the ongoing need for system adaptability. We conclude with future directions for research in enhancing the scalability and efficiency of these systems, aiming to further bridge the gap between human and machine communication.

Advancements and Prospects in Dialogue Agents and LLMs

July 24, 2023

Invited Talk, Bosch Center of Artificial Intelligence (BCAI), Renningen, Germany

In this talk, we will delve into the exciting advancements and prospects within the fields of Dialogue Agents and Language Model Models (LLMs). As the landscape of natural language processing and artificial intelligence continues to rapidly evolve, these two areas play pivotal roles in transforming human-computer interactions and enabling more sophisticated language understanding and generation.

Generative AI towards Scientific Discovery

April 06, 2023

Invited Talk, Microsoft Research AI4Science, Amsterdam, Netherlands

Artificial Intelligence (AI) is becoming increasingly important for scientific research, with many applications in fields like chemistry and biology. Generative AI, in particular, has shown great promise in accelerating scientific discovery by generating hypotheses and insights that may be difficult for humans to uncover. In this talk, we will use my previous research as examples to explore how generative AI models can be applied to natural language processing (NLP) and information retrieval (IR) tasks. We will also discuss the challenges and limitations of using generative AI, including concerns around uncertainty and interpretability. Furthermore, we will discuss the potential applications of generative AI in scientific discovery, such as accelerating the discovery of new molecules and simulating biological processes. Generative AI also has the potential to facilitate interdisciplinary collaborations between AI researchers and domain experts, such as chemists and biologists.

Learning Embeddings to Represent Information Retrieval Systems

October 14, 2022

Invited Conference Talk, Amazon Machine Learning Conference (AMLC), Seattle, WA, US

In this talk, I introduce our paper accepted by Amazon Machine Learning Conference (AMLC) 2022 Workshop on Deep Metric Learning and Semantic Similarity Search.

Frontiers of Collaborative Task-oriented Dialogue Systems

July 06, 2022

Invited Talk, University College London (UCL), London, UK

In this talk, I introduce our exploration of collaborative dialogue models and our recent work on task-oriented dialogue systems. Task-oriented dialogue systems (TDSs), as an important branch of dialogue systems, have raised considerable interest due to their broad applicability. Recent end-to-end single-module TDSs have many attractive characteristics, e.g., global optimization and easy adaptation to new domains. However, we think it is impractical to use a single general agent to handle all complex cases in TDSs. For example, an agent that is specialized in booking a restaurant is unlikely to work well in scheduling meetings. Inspired by this intuition, we call for studies on a new series of collaborative task-oriented dialogue system (CTDS) frameworks, where multiple parallel and/or hierarchical agents work in a collaborative manner to achieve better performance than a single, general agent.

Transformer Uncertainty Estimation with Hierarchical Stochastic Attention

April 29, 2022

Invited Talk, Search Engine Amsterdam (SEA), Amsterdam, Netherlands

In this talk, I introduce the stochastic transformers, which have been published at AAAI 2022 conference and discuss about uncertain estimation in NLP/IR.Transformers are state-of-the-art in a wide range of NLP tasks and have also been applied to many real-world products. Understanding the reliability and certainty of transformer model predictions is crucial for building trustable machine learning applications, e.g., medical diagnosis. Although many recent transformer extensions have been proposed, the study of the uncertainty estimation of transformer models is under-explored. [Link]

Jiahuan Pei | 裴家欢

Talks and presentations