Umeå universitets logga

I oktober 2009 infördes krav på förarbevis för att köra moped klass II. För att få förarbevis måste provtagarna genomgå en obligatorisk utbildning och klara ett kunskapsprov. För att få tillstånd att förrätta prov för moped klass II fick provförrättare lämna in ett exempel på ett prov till Transportstyrelsen tillsammans med ansökan. Vi har analyserat ett urval av dessa prov. Dels gjordes en innehållsanalys av frågorna i proven för att undersöka i vilken utsträckning de täcker det innehåll som stipulerats i föreskriften, och dels undersöktes om de inlämnade proven följer riktlinjer om antal frågor samt hur frågorna följer generella riktlinjer för frågekonstruktion.

Bland de 34 utvalda provförrättarna hade 11 konstruerat egna prov, övriga angav att de använde de prov som Sveriges Trafikskolors Riksförbund (STR) utformat. Föreskriften (TSFS 2010:80) anger att provet ska innehålla 25 till 35 uppgifter. Samtliga studerade prov innehöll minst 25 uppgifter. Med ett par undantag förekommer uppgifter inom samtliga fyra innehållsområden som nämns i föreskriften. Tyngdpunkten ligger på området Trafikregler. Målen för utbildningen och målen för provet i föreskriften stämmer inte helt överens, vilket förmodli­gen är anledningen till att många uppgifter gäller innehåll som inte definieras i föreskriftens krav på provet. De flesta av dessa kan dock kopplas till utbildningsmål i föreskriften.

När det gäller vilka brister som uppträder kan man exempelvis konstatera att vissa frågor kunde vara mer tydligt formulerade, en del svarsalternativ avviker eller är för osannolika, vilket gör antalet reella svarsalternativ färre, vilket är olyckligt.

Både innehåll och utformning av proven varierar avsevärt varför det vore bäst att låta Transportstyrelsen utforma även provet för förarbevis för moped klass II. Det skulle innebära ökad enhetlighet och likvärdighet och därmed en mer rättssäker prövning. Det skulle vara möjligt att sträva efter parallella prov. Det skulle också göra det lättare att övervaka resul­taten och kvaliteten i provet över tid än vad som är möjligt med nuva­rande tillsynsförfarande.

Mot bakgrund av att Riksrevisionen har ifrågasatt likvärdigheten i det praktiska körprovet var syftet med denna studie att göra en undersökning av likvärdigheten i bedömningen av provtagarnas prestation på kör­provet. För att undersöka detta utformades studien så att två bedömare, den ordinarie förarprövaren och en s.k. kusk, fick bedöma samma prov­tagare. Studien omfattade sammanlagt 535 prov och bedömningen gjor­des på en 2-gradig skala (godkänd/underkänd). Eftersom utfallet av provet kan påverkas av faktorer som är förknippade med provtagaren och de två bedömarna utvecklades enkäter som besvarades av kuskar, förarprövare och provtagare. Information om provens genomförande erhölls via ett speciellt utarbetat s.k. kuskprotokoll. Resultatet från studien visade att det var en mycket god överensstämmelse mellan förarprövarnas bedömningar. För 93 procent av proven gjorde kusken och förarprövaren exakt samma bedömning på den 2-gradiga skalan. I de fall bedömningarna skilde sig åt visade analyserna endast ett fåtal systematiska skillnader utifrån möjliga förklaringsvariabler. Dessa var dock inte problematiska ur likvärdighetssynpunkt. Erhållna resultat indikerade att de flesta prov genomfördes på ett tillfredsställande sätt. Sammanfattat indikerar detta en hög likvärdighet, såväl vid en nationell som vid en internationell jämförelse.

Den första september 2008 infördes en ny provmodell, ”sammanhållet prov”, för körkort behörighet B. Införandet av sammanhållet prov  innebar att kunskapsprovet och körprovet bokas samtidigt och genomförs samma dag eller nära i tid. Provtagarna börjar med att genomföra kunskapsprovet och oavsett resultat på kunskapsprovet genomför de därefter kör­provet. Provtagarna måste få godkänt på båda proven inom en tvåmånadersperiod från det första godkända provet för att få körkort. I annat fall får de göra om båda proven och betala nya provavgifter.

Denna studie hade som syfte att undersöka om införandet av sammanhållet prov medfört att godkännandefrekvensen på provets två delar ‑ kunskapsprovet respektive körprovet ‑ förändrats, samt om integrationen mellan proven förbättrats.

För att jämföra provtagarnas resultat på kunskapsprovet och körprovet före och efter införandet av sammanhållet prov togs stickprov från registerdata från perioden före och efter införandet av sammanhållet prov.

Provtagarnas resultat på kunskapsprovet undersöktes genom att jämföra resultatet på de kunskapsprov som genomfördes vecka 50 2007 med kunskapsprov som genomfördes motsvarande vecka 2008. Resultatet visar att andelen godkända prov har ökat från 51 till 62 procent. Genomsnittspoängen för provet har ökat och ligger i stick­provet från 2008 ovanför gränsen för godkänt, 52 poäng (52,6 jämfört med tidigare 50,4 poäng).

Resultaten av de körprov som genomförts från december 2008 fram till sista februari 2009 jämfördes med resultatet på de körprov som genomförts under motsvarande period ett år innan. För att få jämförbara grupper ingick endast provtagare som klarat kunskapsprovet i jämförelsen. Analyserna visade att andelen godkända kör­prov ökat från 59 till 66 procent.

Analyserna visade också att det fanns ett samband mellan resultat på kunskapsprovet och körprovet då provtagare med högre poäng på kunskapsprovet hade högre sannolikhet att bli godkända på körprovet.

Slutsatserna från studien är att såväl resultaten på kunskapsprovet som körprovet har förbättrats efter införandet av sammanhållet prov och att sambandet mellan kunskapsprovet och körprovet blivit starkare.

Over the last few years there have been discussions regarding whether the Swedish system for selection to higher education is suitable for selecting the most appropriate students. The reasons for these discussions are problems with recruitment, student completion and selection. In the light of these problems and within the limits of the alternative selection, the Swedish National Agency for Higher Education started a project in 2003, with the ambition of developing an entrance test for the technology section in higher education. The prototype consisted of three separate tests: a verbal test, a problem-solving test and a knowledge test.

The knowledge test, which was developed at the Department of Educational Measurement at Umeå University, was intended to measure general knowledge in technology. The assumption was that there is a relationship between knowledge and interest, and one related question was whether the result from this test also could be an indirect measure of interest. If there is a positive relationship between interest, motivation and study success, the idea of using such a test for recruitment and selection is supported.

The first purpose of this study was to examine if there is any evidence for a positive relationship between the score on a general knowledge test in technology and study success in education in technology. The second purpose was to find relevant psychological literature on interest and motivation which could be used in the verification of the hypotheses. For this purpose, a literature search in several databases was conducted. The third purpose was, given that there was a positive relationship between knowledge and study success, to describe the possible uses of the test in the Swedish selection system.

The result, concerning the first purpose, indicated evidence for the hypotheses and therefore the conclusion was that there is also a positive relationship between knowledge and study success. Regarding the third purpose, the conclusion was that the knowledge test could work as an instrument for recruitment, selection and prediction, with a few exceptions.

Finally, some limitations of the study were discussed and some suggestions for further research were given.

Studiens syfte var dels att studera effekter av betingelserna övning och instruktion på ett kvantitativt-nume-riskt prov, dels att undersöka om förmåga att tillgodogöra sig respektive betingelse var beroende av intellektuell prestationsförmåga som denna uppskattades utifrån två begåvningsprov. Det kvantitativt-numeriska provet, som betecknades K.VR och bestod av tre delprov, var en förlaga till det nuvarande högskoleprovet. De prov som utnyttjades för att erhålla en uppskattning av intellektuell förmåga utgjordes av ett verbalt och ett ickeverbalt prov från WIT III. Studiens design var en utvidgad variant av Solomon's 4-gruppsdesign och i studien ingick 105 folkhögskoleelever som försökspersoner. Efter det att begåvningsfaktorns inverkan kontrollerats via statistisk metodik blev studiens utfall att ingen av betingelserna övning eller instruktion resulterade i en förhöjd poäng på efterprovet. Tänkbara anledningar till det erhållna resultatet diskuteras.

I föreliggande litteraturgranskning ingår studier som har behandlat övnings- och instruktionseffekter på test. De studier som genomförts inom problemområdet är relativt många och varierar såväl i uppläggning som i innehåll och genomförande och därför har granskningen skett utifrån en speciell systematisering och avser i huvudsak enbart standardiserade test. Utgångspunkten har varit att de redoivsade studierna sorterats och analyserats under huvudrubrikerna övning och instruktion. Inom respektive rubrik har studierna sedan också sorterats enligt ett visst system som inneburit parallellitet och möjligheter till jämförelser inom och mellan betingelserna övning och instruktion. Studiernas design har utgjort den huvudsakliga struktureringsvariabeln och därutöver har struktureringen gjorts utifrån övnings- och instruktions-betingelsens varaktighet, vilken typ av test som utnyttjats, samt individkarakteristika. Det sistnämnda har inneburit att försökspersonernas ålder, kön, prestationsnivå och tidigare erfarenhet av test har innefattats. I den avslutande diskussionen, som innehåller en sammanfattande beskrivning av övnings- och instruktionseffekter på test, behandlas dels olika problem som är förknippade med genomförandet av en litteraturgranskning inom detta område,dels vilka slutsatser som kan dras utifrån existerande forskning inom området.

This report consists of a compilation of a number of studies, which all of them have dealt with the effects of practice and instruction upon a person's test results. A literature review based on a dichotomy of practice and instruction under short-term and long-term conditions respectively, as well as on the denign of the investigations and certain characteristics of the individuals, resulted in general as well as in more specific statements on the arising of practice and instruction effects. In all the empirical investigations quantitative-numerical tests were used (KVR and NOG respectively), where one-of them (NOG) is a part of the so-called Swedish Scholastic Aptitude Test. The basic aim was to investigate if short-term practice (KTÖ) and short-term instruction (KTI) could have an effect upon the score of these tests. In this case practice was defined as taking a pre-test, and in the first three studies the instruction consisted of two phases: a first one concerning general testing strategies, and a second one concerning a specially adapted problem solving strategy. As a basis for the forming of this special problem solving strategy for NOG served a logically constructed and sequential problem solving strategy, which was formed as a direct influence on the individual via instruction in the first three studies, and as an indirect influence on the individual via the item format in the last three studies. From the achieved results can be mentioned that the literature paid attention to the importance of a person's knowledge of and familiarity with tests and examinations in a specific as well as in a more general sense. If a person has no or very little former experience of tests there is a certain probability that there will be a higher score due to KTÖ or KTI. This probability is reduced considerably if the person has some or relatively great former experience in this respect. This was also verified indirectly in the empirical studies, which were mainly carried out on individuals who could be classified as experienced in and familiar with tests. No practice or instruction effects could be found either in the first three studies or indirectly in the last three studies. In these ones a specially constructed item format had been used, which had been formed according to the sequential problem solving strategy. With the intention to obtain an interpretation frame for practice and instruction effects on tests, the basis in this report has been a theoretically constructed model, whereby the achieved effects are first of all related to the individual variable former experience, and secondly to whether a test consists of correctly or incorrectly constructed items.

Syftet med denna studie var att pröva en logiskt uppbyggd med sekvensiell problemlösningsstrategi för ett prov (NOG) avsett att mäta kvantitativ-numerisk förmåga. Provet ingår i den nuvarande versionen av det s k högskoleprovet. Individernas problemlösningsbeteende på NOG formades i enlighet med den sekvensiella problemlösningsstrategin via ett experimentellt uppgiftsformat (NOG^x). Studiens design var en variant av en experiment-kontrollgruppsdesign, där jämförbarheten mellan de två ingående gruppernas initiala prestationsförmåga erhölls utifrån ett verbalt förprov (ORD). I studien ingick 257 elever från samtliga linjer i gymnasiets årskurs 3. Resultatet visade att den sekvensiella problemlösningsstrategin på det hela taget inte hade någon påvisbar effekt. Enbart i de fall där C eller E var korrekt alternativ förelåg en tendens till att uppgiftsformatet eller indirekt den sekvensiella strategin kunde ha viss betydelse. Betydelsen av alternativ C, E relativt A, B, D som korrekt alternativ framkom också vid försökspersonernas säkerhetsskattningar.

I föreliggande studie var syftet att pröva en logiskt uppbyggd och sekvensiell probleml ösningsstrategi för ett kvantitativt-numeriskt prov (NOG). Detta prov ingår som ett av total t sexdel p r ov i det s k högskoleprovet . Den logiskt uppbyggda pr obl emlösni ngsst rat egi n t ogs som u t gångspunkt för ut formandet a v et texperiment e l l t uppgi f t s f o rmat (NOG**) och st udi ens upp läggn ingbaser ades på en j ämf ö r e l s e a v samma pr ov i två versi oner : enkonvent ionel l (NOG) och en exper iment e l l (NOGx x ) . Indi kat i onerpå at t exper iment - och kont rol lgrupp var ini t iai t j ämf ör bar ai pr e s tat ionsavseende erhöl l s ut i f rån et t verbal t f ö rprov (ORD) .I st udi e n ingick 264 el ever f rån samt l i g a f em l injer i gymnas i e t sår skur s 3. St udi ens ut fal l bl e v at t det int e förel å g någon ski l l nadi poängmede l vä rde, baser a t på ant ingen konvent ionel l el lerdi f ferent iel l poäng , mel lan den g rupp som erhål l i t det exper imentell a (NOGx x ) och den g rupp som e rhål l i t det konvent ionel l a(NOG) uppgi f tsf o rmat e t . Det t a i ndi ker ade at t den sekvensi e l l apr obl eml ösni ngsst rat egi n int e hade någon di rekt påvi sbar ef fektel ler indi rekt at t det konvent ionel l a NOG-f o rmat e t int e varkompl ext . Om a nal ysen baser ades på C el ler E som kor rekt al ter nati v bl e v dessutom det exper iment e l l a uppgi f t s f o rmat e t komplexar e än det konvent ionel l a (NOG). Konsekvense r a v det er hållna ut fal let di skut e ras.

Since 1990, the Department of Educational Measurement at Umeå University has been commissioned to study the Swedish drivinglicense test by the Swedish National Road Administration, SNRA. Over the past few years several studies have been conducted in order to develop and improve the Swedish driving-license test. The focus of the majority of the studies has been the theory test.

The aims of this paper were threefold: firstly to describe the development of the driver education and the driving-license test in Sweden during the past century; secondly, to summarize the findings of our research, which is related to important issues in test development; and finally, to make some suggestions for further research.

Studies of subjective driving skill have usually assessed perceived driving skill in relation to the skills of the average driver. In order to examine whether novice drivers are overconfident with respect to their actual skills, a different method was used in the present study, where specific aspects of perceived driver competence were compared with assessments made by a driver examiner. A Finnish (n = 2847) and a Swedish (n = 805) sample of driving test candidates completed self-assessments and took a practical driving test; the instruments differed between the countries. The results indicated that about 50 percent of the Finnish and between 25 and 35 percent of the Swedish candidates made realistic assessments of their competence in the areas Vehicle manoeuvring, Economical driving and Traffic safety. The proportion of those who overestimated their competence was greater among the Swedish candidates than the Finnish candidates. This might be explained by greater possibilities of practicing self-assessment in the Finnish driver education. Furthermore, the results indicate that males are not overconfident to a greater extent than females. In conclusion, when perceived competence is related to actual competence instead of the skills of the average driver, the majority of drivers are no longer found to overestimate their skills.

Introduction. In a standards-based school-system alignment of policy documents with stan-dards and assessment is important. To be able to evaluate whether schools and students have reached the standards, the assessment should focus on the standards. Different models and methods can be used for measuring alignment, i.e. the correspondence between standards and assessment. Based on the assumption that a model must be able to include content and cogni-tive complexity, nine different models are identified and these models are then scrutinized with reference to defined theoretical criteria. The conclusion is that Bloom’s revised taxon-omy and Porter’s taxonomy are the most appropriate models.

Method. Bloom’s revised taxonomy and Porter’s taxonomy are compared based on empirical data from standards and assessment in a chemistry course in upper secondary schools in Swe-den. The comparison is based on five rules and of inter-rater reliability.

Results. Bloom’s revised taxonomy was more inclusive and exclusive than Porter’s taxon-omy. The inter-rater reliability for classification of standards was significantly better for Bloom’s revised taxonomy than for Porter’s taxonomy.

Conclusion. Based on the five rules, the conclusion is that Bloom’s revised taxonomy is the best model.

A new curriculum has been introduced for the Swedish driver education with the intention to (among others) include higher levels of skills and knowledge. The purpose of the present study was to describe and compare the old and the new curriculum for the Swedish driver education by using an instrument specific for this purpose. Bloom’s revised taxonomy and an integrated taxonomy were empirically tested and confirmed reliable and useable as instruments for describing the content in both the old and new curriculum. Three raters classified the objectives in the curriculum with the aid of the instruments and the results showed changes in the required knowledge and skills. One of the major changes was the decreasing demands of the cognitive skill remember and factual knowledge and the increased emphasis on the cognitive skill evaluate and on metacognitive knowledge in the theoretical objectives in the new curriculum. Further, the results showed that the emphasis on the psychomotor skill of mechanism has decreased in the practical objectives. The conclusion of this analysis is that the curriculum of the Swedish driver education has developed as intended. Though, in an education system it is important to have an agreement between goals of education and measurement. In future studies, it is therefore important to examine the alignment between objectives and measurement.

In 2006 a new curriculum for driver education and a new theory test were introduced in Sweden. As the driver education system is a criterion-referenced system, it is important that there is a correspondence between the curriculum, education and the test. The main purpose of the present study was to examine the alignment between the old curriculum for driver education in Sweden and the old theory test, as well as the alignment between new curriculum and the new theory test using Bloom’s revised taxonomy. In addition, the purpose was to examine the interrater reliability when categorizing theory test items into Bloom’s taxonomy. The interrater reliability of judges categorizing the items into Bloom’s taxonomy was moderate to good for the old and new theory test, but somewhat better for the new test. The examination of alignment used four different indicators: content, cognitive complexity, balance and range. Results demonstrated better alignment between the old curriculum and the old test than between the new curriculum and the new test. The conclusion was that the curriculum has changed but that the new theory test still resembles the old test to a great extent. To improve alignment, the new theory test has to be revised in accordance with the curriculum.

The present study concerns an experiment where part of the Swedish Road Administration’s right to the driving-license examination was delegated to the driver education. During the experimental period test takers that failed the practical driving test, due to an isolated shortcoming, were given the opportunity to be retested in a traffic school, with focus mainly on that shortcoming.

The main purpose of this study was to examine the definition of the isolated shortcoming and to obtain indications of whether the judgements of these are as alike as possible. Another purpose was to study the attitudes and experience of participants in the experiment through interviews. Furthermore, the intention was to use available statistics from the experiment to examine to what extent the parties involved, driver examiners and driver instructors, where able to make reliable and relevant judgements regarding the isolated shortcoming.

One conclusion is that the definition of the isolated shortcoming cannot be considered to be reliable since available statistics, as well as results from the interviews, indicate insecurity about the definition. Another conclusion from the interviews is that the overall attitude toward the experiment was positive among driver examiners and driver instructors.

Sammanfattningsvis framstår det som en självklarhet att självvärdering ska vara ett inslag i såväl förarutbildning som förar­prov eftersom det utgör en väsentlig del i den nya kursplanen. Det står också klart att många vinster, både för lärandet och tra­fiksäkerheten, kan göras genom att införa självvärdering i utbildningen. Nästa steg är att diskutera hur detta ska genomföras i praktiken genom att finna sätt för hur självvärdering ska användas både i utbild­ning och i förarprov.

The Swedish Scholastic Aptitude Test (SweSAT) is used in the process of selection to higher education in Sweden. The SweSAT is administered twice a year and consists of 122 items. The total raw score is transformed into a normed score, which is used in the selection procedure. The main purpose of this study was to investigate the effects of repeated test taking regarding the SweSAT. The effects are described in relation to the test taker who chooses to retest. Another purpose was to relate the effects of repeated test taking to the individual subtests of the SweSAT and to analyse the self selection process related to repeated test taking. Multivariate linear models for repeated measurements were used to describe the effects of repeated test taking in relation to the test taker. Calibrated scores were used as a basis for relating the effects to subtests. The summarised conclusions were that test takers with high SweSAT score repeated the test more often than those with low scores. Males and test takers from social group I repeated the test more often than females and social group III, respectively. Repeated test taking had its main effect from the first to the second testing for all social groups and both sexes. This main effect of repeated test taking was related to two subtests, WORD and DTM, for all social groups. For the other subtests, social groups II and III were favoured by subtest DS, social group II was favoured by READ and social group III was favoured by subtest ERC. With reference to total normed test score the results indicated that there was a marginal difference between social groups I and II, when controlled for sex and age. Social group I had a higher mean score than the other social groups. Males had higher mean scores than females. The summarised conclusion was that the differences in mean scores between social groups I and III were higher than between males and females.

The Swedish Scholastic Aptitude Test (SweSAT) is, together with

grade point average (GPA), used in the selection procedure for higher education in Sweden. The SweSAT is administered twice a year, it consists of 122 items and the total raw score is transformed to a normed score which is used in the selection procedure. Certain rules apply to the SweSAT when it is used for selection to higher education and one of these rules is that the test taker can repeat the test. The main purpose of this study was to investigate the effects of repeated test taking for the SweSAT. The effects are described in terms of normed score, partly in relation to the rules for selection, and partly in relation to the test taker who elect to retest. The result indicates that the effects of repeated test taking mainly occur between the first and second testing. This is a tendency that is observed when the effects of repeated test taking are analysed in relation to the rules for selection as well as in relation to the test taker. According to the rules for the SweSAT in the selection system the best score is used if a certain applicant has more than one valid score and the model for description of score gains in this study is based on this fact. Another purpose was to relate the effects of repeated test taking to subtests of the SweSAT. Proportional stratified sampling and multivariate linear models are used when describing the effects of repeated test taking in relation to the test taker, while a reference population and a calibrated raw score is used as a basis when the effects are related to subtests. With regard to subtests, the main effect of repeated test taking is related to three subtests: WORD, DS and ERC. The summarised conclusion that is made in this study is that the effects of repeated test taking between the first and second test occasion, in terms of standard deviation units for total normed score for the SweSAT, is about 0.2 and this finding is in congruence with findings in earlier studies. Plausible hypotheses for

the obtained results are discussed.

The main purpose of this study is to integrate and discuss results from studies that focus on the effects of repeated test taking of the Swe-SAT. Messick’s four-faceted model of validity is used as an integrating and an analytic tool. Another purpose is to use results from supplementary education as a reference in this integration.

The summarized conclusion is that the existing rule concerning repeated test taking for the Swedish Scholastic Assessment Test (Swe-SAT) is relevant. It is relevant as the test taker has a chance to obtain a good estimate of his or her knowledge and ability. The unintended social consequences are reduced if actions are taken to motivate test takers to repeat the test.

The conclusion is quite the opposite for supplementary education. A

change in the rule that eliminates the usage of supplementary grades in the selection procedure will result in higher validity.

The conclusion is also that the Messick model is a very useful tool for validation. When applying the Messick model, the aim and the direction of the process of validation will be systemized and optimized as well as nuanced.

The main purpose of this study was to consider standard setting in a theoretical perspective. The aim was to describe the method categories which are used to set a standard, and to examine which category is the most suitable for a criterion-referenced licensure test. Criterion-referenced tests are used to ascertain an examinee’s status with respect to a well-defined criterion. A licensure test is administrated at the end of an education and covers the whole course. Another purpose of the study was to examine which method, in the most suitable category of methods, should be used to set a standard on a criterion-referenced licensure test like the Swedish driving licence test. The study describes how the standards are set in the old and new Swedish theoretical driving licence tests. The criteria used to decide the most suitable method were technical adequacy and practicability, each defined in several ways, and some specific criteria since it is a licensure test. The theoretical examination showed that it is not possible to find one single method that is best in all possible situations. The conclusion was that either Informed Judgement or Iterative Angoff was the most preferable method for setting a standard on the Swedish driving licence test.