Speech by Dr. Pascal Poupart from the University of Waterloo: 

Inverse Constraint Learning and Risk Averse Reinforcement Learning for Safe AI

Dr. Pascal Poupart is a professor at the David R. Cheriton School of Computer Science at the University of Waterloo in Canada. He also serves as a CIFAR AI Chair at the Vector Institute and is a member of the AI Institute at Waterloo. Since 2022, he has been part of the advisory board for the NSF AI Research Institute at Georgia Tech. Previously, from 2018 to 2020, he held the roles of Research Director and Chief Research Scientist at the Waterloo Borealis AI Research Lab at the Royal Bank of Canada. His research primarily focuses on developing machine learning algorithms for natural language processing and materials discovery, with a particular emphasis on reinforcement learning. His team is currently engaged in several notable projects, including Inverse Constraint Learning, Mean-Field Reinforcement Learning, Foundational Models for Reinforcement Learning, Bayesian Federated Learning, Uncertainty Quantification and Calibration, Probabilistic Deep Learning, Conversational Agents, Transcription Error Correction, Sports Analytics, Adaptive Satisfaction, and materials to facilitate desirable chemical reactions for CO2 conversion and CO2 capture.

In his speech, Professor Poupart highlighted the crucial role of real-world constraints in the practical implementation of reinforcement learning (RL) and control systems, and proposed feasible algorithmic solutions. These constraints are essential for ensuring the feasibility of implementation, safety, and adherence to key performance indicators. However, some constraints are challenging to define precisely, particularly in complex applications such as autonomous driving. While establishing target reward functions is relatively straightforward, accurately articulating the implicit constraints that expert drivers adhere to for safe, smooth, and comfortable driving proves to be much more difficult.

Professor Poupart introduced the concept of Inverse Constraint Learning (ICL). While Inverse Reinforcement Learning (IRL) traditionally focuses on determining the reward functions that explain expert behavior, this approach is often insufficient in practical applications. Understanding the constraints underlying behavior is equally important, as these constraints frequently provide a more intuitive rationale for actions than reward functions, which is especially crucial in safety-critical contexts. By reverse-engineering these constraints, we can uncover the implicit logic behind expert behavior, enabling the design of autonomous driving strategies that more closely mimic human behavioral patterns.

He also examined methods for learning soft constraints from expert trajectories. This strategy relies on a known reward function and utilizes expert trajectories to derive soft constraints. In real-world applications of machine learning and reinforcement learning, challenges often arise from noise in sensor data or imperfect expert demonstrations. This necessitates a balance between the data's reliability and the model's performance. Unlike traditional hard constraints (like energy usage limits), soft constraints permit the model to occasionally breach certain restrictions, seeking for a balance between the reward function and constraints. As a result, the model can implement more adaptable strategies in response.

Moreover, Professor Poupart presented a risk-averse reinforcement learning approach that utilizes Gini deviation. In various real-life scenarios, such as avoiding collisions in autonomous driving or minimizing substantial financial losses in portfolio management, risk avoidance is crucial. While traditional reinforcement learning focuses on maximizing expected returns, risk-averse reinforcement learning also considers risk management. Gini deviation offers an alternative to conventional variance-based methods, allowing for a more effective assessment of potential risks during strategy implementation, especially in high-risk decision-making contexts.

This presentation introduces an array of new research tools and offers practical insights into implementing safe artificial intelligence in real-world applications. It enhances the feasibility of developing intelligent systems capable of autonomously adapting to complex environments and meeting rigorous safety standards. The insights and methodologies provide valuable guidance for the fields of machine learning and reinforcement learning, particularly in addressing uncertainty and managing risk. This alignment ensures that these technologies are better suited to meet the demands and challenges of real-world applications.