Contemporary welfare organizations engage in various evaluation practices to assess the quality of their services. In this paper we report a qualitative exploration of how technology-enabled evaluations are understood by organizational members who participate in quality assurance activities in Swedish social services. The study contributes to critical information systems literature, focusing on the tensions professionals experience in relation to the digital systems they use for evaluations. For example, "quantities" take precedence over the qualities of such work, as information systems constrain ambitions to realize knowledge-based social services. The results reveal three tensions in professionals' evaluation-related activities arising from conflicting uses or desires. One is between desires for flexible systems that enable reflection and standardized digital support systems. Another is between uses or desires for indicators that are meaningful at the operational level and for general, comparable measures at the management level. The third is between desires to use evaluation procedures for learning and control. The study contributes to both theory and practice related to technology-enabled evaluation of welfare services, and critical perspectives on information systems.