Characterizing the energy performance of building components and locating anomalies is necessary for effectively refurbishing existing buildings. It is often challenging because defects in building envelopes deteriorate without being visible. Passive infrared thermography (PIRT) is a powerful tool used in building inspection. However, thermal image interpretation requires significant domain knowledge and is prone to artifacts arising from a complex interplay of factors. As a result, PIRT-based inspections require skilled professionals, and are labor-intensive and time-consuming. Artificial intelligence (AI) holds great promise to automate building inspection, but its application remains challenging because common approaches rely on extensive labeling and supervised modeling. It is recognized that there is a need for a more applicable and flexible approach to leverage AI to assist PIRT in realistic building inspections. In this study, we present a label-free method for detecting anomalies during thermographic inspection of building envelopes. It is based on the AI-driven prediction of thermal distributions from color images. Effectively the method performs as a one-class classifier of the thermal image regions with a high mismatch between the predicted and actual thermal distributions. The algorithm can learn to identify certain features as normal or anomalous by selecting the target sample used for training. The proposed method has unsupervised modeling capabilities, greater applicability and flexibility, and can be widely implemented to assist human professionals in routine building inspections or combined with mobile platforms to automate the inspection of large areas.