In a previous paper, we proposed an algorithm for ontology-driven conceptual model abstractions [18]. We have implemented and tested this algorithm over a FAIR Catalog of such models represented in the OntoUML language. This provided evidence for the correctness of the algorithm’s implementation, i.e., that it correctly implements the model transformation rules prescribed by the algorithm, and its effectiveness, i.e., it is able to achieve high compression (summarization) rates over these models. However, in addition to these properties, it is fundamental to test the validity of this algorithm, i.e., that it achieves what it is intended to do, namely provide summarizing abstractions over these models whilst preserving the gist of the conceptualization being represented. We performed three user studies to evaluate the usefulness of the resulting abstractions as perceived by modelers. This paper reports on the findings of these user studies and reflects on how they can be exploited to improve the existing algorithm.