RAIDER: Tool-Equipped LLM Agent for Robotic Action Issue Detection, Explanation and Recovery

Silvia Izquierdo-Badiola1,2, Carlos Rizzo1, Guillem Alenyà2
1Eurecat Technology Center of Catalonia 2Polytechnic University of Catalonia
Paper arXiv

RAIDER LLM agent dynamically detects, explains and resolves issues for action execution.

RAIDER system overview

Abstract

As robots increasingly interact in dynamic human-centric environments, improving their ability to detect, explain, and recover from action-related issues becomes crucial. Traditional methods often lack context-awareness and adaptability, hindering their real-world effectiveness. While methods relying on Visual or Large Language Models offer greater flexibility, they struggle to ground the extracted information to tangible constraints, limiting their performance in embodied systems. To address this gap, we propose RAIDER, a novel agent integrating LLMs with grounded tools for efficient and selective information gathering. RAIDER identifies and explains issues through a unique "Ground, Ask&Answer, Issue" procedure, dynamically generating context-aware precondition questions and calling appropriate tools to resolve them, all managed through a regulated flow incorporating self-correction mechanisms. Our results outperform existing methods reliant on predefined action models, full scene descriptions or standalone trained models. Additionally, the system's explanations substantially improve the success of recovery plans, which may involve human interactions where issues cannot be independently resolved. RAIDER's modular architecture allows for straightforward adaptation to various scenarios, showcasing its potential as a versatile solution in robotic issue detection and explanation.

Assistive Scenario: Detailed Demo in a Handover

This video shows the details of the iterative flow between the LLM and the tools, regulated through the program flow manager, in a handover. Before executing the instructed action, the agent analyses the current context, decides what information is needed, and selects the appropriate tools to query. In this case, the issue identified is that the user is not looking at the robot, triggering a replan involving an interaction with the human to resolve the issue.

Household Simulator: Detailed Examples

RAIDER detailed examples

BibTeX

@article{izquierdo2025raider,
  author    = {Izquierdo-Badiola, Silvia and Rizzo, Carlos and Alenyà, Guillem},
  title     = {RAIDER: Tool-Equipped Large Language Model Agent for Robotic Action Issue Detection, Explanation and Recovery},
  journal   = {arXiv 2503.17703},
  year      = {2025}
}