Umeå University's logo

This study explores the type and frequency of extramural English (EE) (e.g. reading, speaking and gaming) and relationships between EE activities and performance on a standardised English reading comprehension (ERC) test in a sample of young adults, with a particular focus on potential sex differences and online gaming as an EE activity. Participants consist of 6,079 test-takers of the Swedish Scholastic Aptitude Test (SweSAT), a test used for admission to higher education, which has ERC as one of the subtests and the only SweSAT subtest that assesses English proficiency. The type and frequency of engagement in EE activities were assessed through a self-report questionnaire. Correlation, ANOVA and multiple linear regression analyses were conducted to explore how EE activities influence ERC performance, as well as differences between males and females. Overall, the results show positive correlations and significant relationships between EE activities and ERC ability, especially reading, watching English content and moderate-to-high frequency of gaming. Sex differences in EE exposure and test performance were observed. In summary, results from this large-scale study corroborate previous findings that English language acquisition also takes place outside the formal context and that leisure-time activities may enhance ERC ability. Implications are discussed.

Detta projekt syftar till att belysa betydelsen av rollspel med artefakter i högre utbildning för att träna konflikthantering.

I högre utbildning används rollspel för att få studenterna att aktivt träna på professionella färdigheter som är svårt att genomföra och examinera inom de högskoleförlagda delarna av utbildningen (Wigert et al., 2021). Rollspel används inom flertalet akademiska professionsutbildningar som omvårdnad och medicin (Westrup & Planander, 2013). 

Även om forskningen visar att drama och rollspel ger positiva resultat så har att användningen minskat inom lärarutbildningen Österlind et al., (2016). Inom förskollärarutbildningen finns få studier om rollspel inom utbildningen men några om att dramatisera med dockteater (Forsberg Ahlcrona & Östman, 2018). Inom förskolans profession finns studier som visar på möjligheten att kunna träna förskolebarnens sociala kompetens och konflikthantering genom rollspel med dockor (Choi & Md-Yunus, 2011). Även andra artefakter som föremål och bilder har visat sig vara funktionella i förskolans undervisning och används som stöd i till exempel tydliggörande pedagogik (Edfelt et al., 2019).

I detta projekt har en workshop om konflikthantering inom förskollärarprogrammet utvecklats. Tydligare fall har konstruerats, fler artefakter har lagts till, digitala verktyg i form av ipads har integrerats och bedömningssanvisningar har formulerats. För att på ett tillförlitligt och giltigt sätt utvärdera rollspelet, utvecklades en bedömningsmatris enligt de steg som Moskal (2000) beskriver och de riktlinjer som beskrivs i Mertler (2000), Moskal (2003) och Brennan (2006).

Pilotstudier har gjorts på två kurser inom förskollärarprogrammet under våren 2023, en i början och en i mitten av utbildningen. Under hösten 2023 har uppläggets reviderats något och genomförs åter på motsvarande kurser och utvärderas med hjälp av studentenkäter.

Preliminära resultat visar att studenterna uppfattar workshopen som inspirerande och lärorik. Artefakter, digitala verktyg och bedömningsanvisningar ses som stöd för lärandet, men även vissa hinder för lärandet uppmärksammas som svårt att förstå bedömningsanvisningar och vissa tekniska problem.

Det finns många funderingar om bedömningssituationer, prov, och om hur de bör planeras och utformas för att vara relevanta vid bedömning i skolan. Tanken med boken är att bistå med några svar på de frågorna, samt att presentera metoder och arbetssätt som underlättar lärarens bedömningsarbete. Denna andra utgåva är uppdaterad utifrån innehållet i "Betyg och prövning" (Skolverket, 2022)

Gender fairness in testing can be impeded by the presence of differential item functioning (DIF), which potentially causes test bias. In this study, the presence and causes of gender-related DIF were investigated with real data from 800 items answered by 250,000 test takers. DIF was examined using the Mantel-Haenszel and logistic regression procedures. Little DIF was found in the quantitative items and a moderate amount was found in the verbal items. Vocabulary items favored women if sampled from traditionally female domains but generally not vice versa if sampled from male domains. The sentence completion item format in the English reading comprehension subtest favored men regardless of content. The findings, if supported in a cross-validation study, can potentially lead to changes in how vocabulary items are sampled and in the use of the sentence completion format in English reading comprehension, thereby increasing gender fairness in the examined test.

Validity is a crucial part of all forms of measurement, and especially in instruments that are high-stakes to the test takers. The aim of this thesis was to examine theory and validity evidence for a recently revised large-scale instrument used for selection to higher education in Sweden, the Swedish Scholastic Assessment Test (SweSAT), as well as identify threats to its validity. Previous versions of the SweSAT have been intensely studied but when it was revised in 2011, further research was needed to strengthen the validity arguments for the test. The validity approach suggested in the most recent version of the Standards for education and psychological testing, in which the theoretical basis and five sources of validity evidence are the key aspects of validity, was adopted in this thesis.

The four studies that are presented in this thesis focus on different aspects of the SweSAT, including theory, score reporting, item functioning and linking of test forms. These studies examine validity evidence from four of the five sources of validity: evidence based on test content, response processes, internal structure and consequences of testing.

The results from the thesis as a whole show that there is validity evidence that supports some of the validity arguments for the intended interpretations and uses of SweSAT scores, and that there are potential threats to validity that require further attention. Empirical evidence supports the two-dimensional structure of the construct scholastic proficiency, but the construct requires a more thorough definition in order to better examine validity evidence based on content and consequences for test takers. Section scores provide more information about test takers' strengths and weaknesses than what is already provided by the total score and can therefore be reported, but subtest scores do not provide additional information and should not be reported. All four quantitative subtests, as well as the Swedish reading comprehension subtest, are essentially free of differential item functioning (DIF) but there is moderate DIF that could be bias in two of the four verbal subtests. Finally, the equating procedure, although it appears to be appropriate, needs to be examined further in order to determine whether it is the best practice available or not for the SweSAT.

Some of the results in this thesis are specific to the SweSAT because only SweSAT data was used but the design of the studies and the methods that were applied serve as practical examples of validating a test and are therefore likely useful to different populations of people involved in test development, test use and psychometric research.

Suggestions for further research include: (1) a study to create a more clear and elaborate definition of the construct, scholastic proficiency; (2) a large and empirically focused study of subscore value in the SweSAT using repeat test takers and applying Haberman’s method along with recently proposed effect size measures; (3) a cross-validation DIF-study using more recently administered test forms; (4) a study that examines the causes for the recurring score differences between women and men on the SweSAT; and (5) a study that re-examines the best practice for equating the current version of the SweSAT, using simulated data in addition to empirical data.

When subscores on a test are reported to the test taker, the appropriateness of reporting them depends on whether they provide useful information above what is provided by the total score. Subscores that fail to do so lack adequate psychometric quality and should not be reported. There are several methods for examining the quality of subscores, and in this study seven such methods, four of which are based on classical test theory and three of which are based on item response theory, were reviewed and applied to empirical data. The data consisted of test takers' scores on four test forms – two administrations of a first version of a college admission test and two administrations of a second version – and the analyses were carried out on the subtest and section levels. The two section scores were found to have adequate psychometric quality with all methods used, whereas the results for subtest scores ranged from almost all scores having adequate psychometric quality to none having adequate psychometric quality. The authors recommend using Haberman's method and the related utility index because of their solid theoretical foundation and because of various issues with the other subscore quality methods.