In 2000, when the first Programme for International Student Assessment (PISA) test of 15-year-olds reading skills was launched by the Organisation for Economic Cooperation and Development (OECD), it marks the start of an OECD led discourse on the quality of national education systems. Test scores of students’ knowledge, ability to reflect and apply their knowledge and experience to real-world issues were used to measure the quality of education systems (OECD, 2001). OECD/PISA is more than an international student test and should be described and analyzed as a programme for transnational governance of education systems (OECD, 2009; Grek, 2010, 2012; Lundgren, 2011; Popkewitz, 2011). PISA produces student performance data every third year together with policy reports for actors in the school sector (Carvalho, 2012; Mangez and Hilgers, 2012). PISA has become a global phenomenon permeating national education discourses world-wide and has influenced policy reforms in participating countries, according to members of PISA Governing Board (Breakspear, 2012). If those in power make the same judgment is not explored, but the paper will critically look into the assumptions regarding how PISA will attain its intended effects, and how consistent these assumptions are. If PISA’s programme theory (PT) is consistent, PISA is more likely to achieve intended effects.
The purpose of this article is to unfold and describe PISA’s PT, that is, what PISA intends to achieve and how, and to critically probe these assumptions, and to discuss how PISA’s PT can be used for further research. A special attention is paid to if the PT has support in research on educational governance, accountability and school development.
PISA has recently been assessed in a special issue of the European Educational Research Journal. The issue focused on how PISA is ‘fabricated’, that is, constructed and reconstructed in and across different contexts, involving many actors and different kinds of knowledge (Ozga, 2012, 166). Although PISA is partly developed at the OECD level and partly at the national level, the “fabrication of PISA” in different member states, should be kept apart from OECD’s policy making. What this special issue, and other analysis of PISA, has not made quite clear is how the various activities and assumptions of PISA are linked together at the transnational programme level and if these assumptions are consistent. PISA can be analysed as a transnational programme implemented in member states, and/or as a transnational programme being fabricated while implemented in member states. This paper elaborates an analysis of PISA as a transnational programme which we argue needs an analysis in its own right.
The article integrates knowledge from three broad research fields: governance research; (education) policy research; and research on evaluation systems. The framework reflects OECD’s steering of PISA and the interplay between (transnational) governance and PISA. Conceived through this framework OECD develops PISA to meet OECD’s knowledge needs and OECD filters PISA results through policy reports and synthesis reports, and integrates knowledge from various evaluation studies to achieve its intentions.
Methodology, Methods, Research Instruments or Sources UsedThe critical PT-analysis includes three steps and searches answers to the following questions: Step 1:Reconstruction of PISA’s PT: 1. What are the assumptions of PISA? 2. What is the problem PISA is aimed to resolve? 3. What are the pre-requisites for PISA, and which activities are assumed to produce which effects? 4. Are the problems that PISA intends to manage described and substantiated? 5.Are intended effects clearly described/ specified? Step 2:Assessment of the PT’s internal validity 6. Is PISA’s PT consistent? (e.g. is there a logical and coherent description of how programme activities shall achieve intended effects?) Step 3:Assessment of PT’s external validity 7.Does PISA have scientific support? 8.Is the knowledge that PISA produces relevant to help resolving the problem that PISA was set up to manage? 9.What value does the knowledge produced have for governance, accountability and school development? The PT-analysis is based on document analysis and interviews with well-informed persons at the OECD and country level (Sweden). Conclusions, Expected Outcomes or FindingsThe expected result of this analysis is a contribution to existing knowledge of PISA, mainly of the nature of PISA’s PT. The paper generates knowledge of how assumptions are linked together and of the consistency of PISA’s PT. This knowledge provides further insight into PISA including which objectives and intended effects PISA has a potential to achieve. PISA’s PT provides a construct that can be inter-subjectively shared which has a value in its own right; advocates and critics of PISA can deliberate about PISA’s nature, assumptions and the programme’s potential. The PT analysis focuses on intended effects, of importance when exploring intended and unintended effects and consequences of PISA in the context of educational governance. PISA’s PT can also be used for further research of use of PISA for accountability and school development in different countries and at different levels of government.