umu.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Validating standard setting: comparing judgmental and statistical linking
Umeå University, Faculty of Social Sciences, Department of applied educational science, Departement of Educational Measurement.ORCID iD: 0000-0003-3028-1299
2017 (English)In: Standard setting in education: the Nordic countries in an international perspective / [ed] Sigrid Blömeke Jan-Eric Gustafsson, Springer, 2017, 1, p. 143-160Chapter in book (Refereed)
Abstract [en]

This study presents a validation of the proposed cut scores for two test forms in mathematics that were developed from the same syllabus and blueprint. The external validity was analyzed by comparing the cut scores set by an Angoff procedure with the results provided by mean and linear observed score equating procedures. A non-equivalent group anchor test (NEAT) design was also used. The results provide evidence that the cut scores obtained through both judgmental and statistical linking are equivalent. However, the equating procedure revealed several methodological and practical challenges.

Place, publisher, year, edition, pages
Springer, 2017, 1. p. 143-160
Series
Methodology of Educational Measurement and Assessment, ISSN 2367-170X, E-ISSN 2367-1718 ; 1
Keywords [en]
Standard setting, Equating, National testing, Equivalence, Fairness
National Category
Educational Sciences
Identifiers
URN: urn:nbn:se:umu:diva-132736DOI: 10.1007/978-3-319-50856-6_9ISBN: 978-3-319-50855-9 (print)ISBN: 978-3-319-50856-6 (electronic)OAI: oai:DiVA.org:umu-132736DiVA, id: diva2:1083669
Available from: 2017-03-22 Created: 2017-03-22 Last updated: 2018-11-05Bibliographically approved
In thesis
1. Dimensions of validity: studies of the Swedish national tests in mathematics
Open this publication in new window or tab >>Dimensions of validity: studies of the Swedish national tests in mathematics
2018 (English)Doctoral thesis, comprehensive summary (Other academic)
Alternative title[sv]
Aspekter av validitet : studier av de Svenska nationella proven i matematik
Abstract [en]

The main purpose for the Swedish national tests was from the beginning to provide exemplary assessments in a subject and support teachers when interpreting the syllabus. Today, their main purpose is to provide an important basis for teachers when grading their students. Although the results from tests do not entirely decides a student’s grade, they are to be taken into special account in the grading process. Given the increasing importance and raise of the stakes, quality issues in terms of validity and reliability is attracting greater attention. The main purpose of this thesis is to examine evidence demonstrating the validity for the Swedish national tests in upper secondary school mathematics and thereby identify potential threats to validity that may affect the interpretations of the test results and lead to invalid conclusions. The validation is made in relation to the purpose that the national tests should support fair and equal assessment and grading. More specifically, the focus was to investigate how differences connected to digital tools, different scorers and the standard setting process affect the results, and also investigate if subscores can be used when interpreting the results. A model visualized as a chain containing links associated with various aspects of validity, ranging from administration and scoring to interpretation and decision-making, is used as a framework for the validation.

The thesis consists of four empirical studies presented in the form of papers and an introduction with summaries of the papers. Different parts of the validation chain are examined in the studies. The focus of the first study is the administration and impact of using advanced calculators when answering test items. These calculators are able to solve equations algebraically and therefore reduce the risk of a student making mistakes. Since the use of such calculators is allowed but not required and since they are quite expensive, there is an obvious threat to validity since the national tests are supposed to be fair and equal for all test takers. The results show that the advanced calculators were not used to a great extent and it was mainly those students who were high-achieving in mathematics that benefited the most. Therefore the conclusion was that the calculators did not affect the results.

The second study was an inter-rater reliability study. In Sweden, teachers are responsible for scoring their own students’ national tests, without any training, monitoring or moderation. Therefore it was interesting to investigate the reliability of the scoring since there is a potential risk of bias against one’s own students. The analyses showed that the agreement between different raters, analyzed with percent-agreement and kappa, is rather high but some items have lower agreement. In general, items with several correct answers or items where different solution strategies are available are more difficult to score reliably.

The cut scores set by a judgmental Angoff standard setting, the method used to define the cut scores for the national tests in mathematics, was in study three compared with a statistical linking procedure using an anchor test design in order to investigate if the cut scores for two test forms were equally demanding. The results indicate that there were no large differences between the test forms. However, one of the test taker groups was rather small which restricts the power of the analysis. The national tests do not include any anchor items and the study highlights the challenges of introducing equating, that is comparing the difficulty of different test forms, on a regular basis.

In study four, the focus was on subscores and whether there was any value in reporting them in addition to the total score. The syllabus in mathematics has been competence-based since 2011 and the items in the national tests are categorized in relation to these competencies. The test grades are only connected to the total score via the cut scores but the result for each student is consolidated in a result profile based on those competencies. The subscore analysis shows that none of the subscores have added value and the tests would have to be up to four times longer in order to achieve any significant value.

In conclusion, the studies indicate that several of the potential threats do not appear to be significant and the evidence suggests that the interpretations made and decisions taken have the potential to be valid. However, there is a need for further studies. In particular, there is a need to develop a procedure for equating that can be implemented on a regular basis.

Place, publisher, year, edition, pages
Umeå: Umeå universitet, 2018. p. 61
Series
Academic dissertations at the department of Educational Measurement, ISSN 1652-9650 ; 11
Keywords
national tests, validity, interrater reliability, standard setting, linking, subscores, test development
National Category
Pedagogical Work
Research subject
didactics of educational measurement
Identifiers
urn:nbn:se:umu:diva-153056 (URN)978-91-7601-936-8 (ISBN)
Public defence
2018-11-30, Hörsal E, Humanisthuset, Umeå, 10:00 (English)
Opponent
Supervisors
Available from: 2018-11-09 Created: 2018-11-05 Last updated: 2018-11-05Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full text

Authority records BETA

Lind Pantzare, Anna

Search in DiVA

By author/editor
Lind Pantzare, Anna
By organisation
Departement of Educational Measurement
Educational Sciences

Search outside of DiVA

GoogleGoogle Scholar

doi
isbn
urn-nbn

Altmetric score

doi
isbn
urn-nbn
Total: 95 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf