Faulty Tools or Disruptive Teammates? A New Theory of Human-AI Conflict and Compromise

Advances in AI are increasing the scope for “reasonable disagreements” between humans and artificial systems. Intelligent robots and virtual agents are increasingly tasked with complex, potentially open-ended assignments that require the AI to function autonomously. AI judgments and decisions can lack transparency and explainability, leading to conflict with the human operator or user, requiring compromise to resolve the conflict. This article presents a new theory of human-AI conflict and compromise, drawing on research on decision-making, conflict in human teams, trust in human-robot teaming, and the challenges for humans of interacting with AI. It is proposed that the nature of conflict depends on the human’s mental model of AI functioning. If the human sees the AI as an advanced tool, conflict arises from differences in choice and implementation of algorithms for joint decision-making. Research on multi-cue judgment within the framework of the Brunswik Lens Model illustrates this type of conflict and its resolution. By contrast, if the mental model attributes humanlike characteristics to the AI, including a Theory of Mind, conflict can arise from different person-centered narratives for framing the task or team functioning. When narratives clash, the robot may be perceived as unsupportive or in violation of team role expectancies. In this case, system design may require using AI natural language capabilities and dialogue can facilitate compromise through context-sensitive matching of the human’s mental model to robot functionality.