This article explores the application of semi-automated decision support systems in the administration of social insurance, focusing on the Swedish Social Insurance Agency’s utilisation of an AI system called SAMU (structured analysis of medical records). SAMU employs cognitive analysis models to assist administrators in identifying pertinent information in medical certificates for evaluating eligibility for sickness or activity compensation. The article assesses the implementation of SAMU, examining its potential impact on a comprehensive evaluation of an individual’s work capacity, as well as its implications for legally correct and secure decision-making. The analysis adopts a sociotechnical perspective, considering the mutual influence between society and technology and the resulting framework that shapes the decisions of caseworkers in the realm of automation-related risks in case management.