umu.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Measurement of alignment between standards and assessment
Umeå University, Faculty of Social Sciences, Department of Educational Measurement.
2008 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Many educational systems of today are standards-based and aim at for alignment, i.e. consistency, among the components of the educational system: standards, teaching and assessment. To conclude whether the alignment is sufficiently high, analyses with a useful model are needed. This thesis investigates the usefulness of models for analyzing alignment between standards and assessments, with emphasis on one method: Bloom’s revised taxonomy. The thesis comprises an introduction and five articles that empirically investigate the usefulness of methods for alignment analyses.

In the first article, the usefulness of different models for analyzing alignment between standards and assessment is theoretically and empirically compared based on a number of criteria. The results show that Bloom’s revised taxonomy is the most useful model. The second article investigates the usefulness of Bloom’s revised taxonomy for interpretation of standards in mathematics with two differently composed panels of judges. One panel consisted of teachers and the other panel of assessment experts. The results show that Bloom’s revised taxonomy is useful for interpretation of standards, but that many standards are multi-categorized (placed in more than one category). The results also show higher levels of intra- and inter-judge consistency for assessment experts than for teachers. The third article further investigates the usefulness of Bloom’s revised taxonomy for analyses of alignment between standards and assessment. The results show that Bloom’s revised taxonomy is useful for analyses of both standards and assessments. The fourth article studies whether vague and general standards can explain the large proportion of multi-categorized standards in mathematics. The strategy was to divide a set of standards into smaller substandards and then compare the usefulness and inter-judge consistency for categorization with Bloom’s revised taxonomy for undivided and divided standards. The results show that vague and general standards do not explain the large proportion of multi-categorized standards. Another explanation is related to the nature of mathematics that often intertwines conceptual and procedural knowledge. This was also studied in the article and the results indicate that this is a probable explanation. The fifth article focuses on another aspect of alignment between standards and assessment, namely the alignment between performance standards and cut-scores for a specific assessment. The validity of two standard-setting methods, the Angoff method and the borderline-group method, was investigated. The results show that both methods derived reasonable and trustworthy cut-scores, but also that there are potential problems with these methods.

In the introductory part of the thesis, the empirical studies are summarized, contextualized and discussed. The discussion relates alignment to validity issues for assessments and relates the obtained empirical results to theoretical assumptions and applied implications. One conclusion of the thesis is that Bloom’s revised taxonomy is useful for analyses of alignment between standards and assessments. Another conclusion is that the two standard setting methods derive reasonable and trustworthy results. It is preferable if an alignment model can be used both for alignment analyses and in ongoing practice for increasing alignment. Bloom’s revised taxonomy has the potential for being such an alignment model. This thesis has found this taxonomy useful for alignment analyses, but its’ usefulness for increasing alignment in ongoing practice has to be investigated.

Place, publisher, year, edition, pages
Umeå: Beteendevetenskapliga mätningar , 2008. , 226 p.
Series
Academic dissertations at the department of Educational Measurement, ISSN 1652-9650 ; 3
Keyword [en]
alignment, standards, assessment, Bloom's revised taxonomy, the Angoff method, the borderline-group method, usefulness, validity
National Category
Manufacturing, Surface and Joining Technology
Identifiers
URN: urn:nbn:se:umu:diva-1865ISBN: 978-91-7264-662-9 (print)OAI: oai:DiVA.org:umu-1865DiVA: diva2:142244
Public defence
2008-10-24, S205, Samhällsvetarhuset, Umeå, 10:15 (English)
Opponent
Supervisors
Available from: 2008-09-30 Created: 2008-09-30 Last updated: 2017-03-28Bibliographically approved
List of papers
1. Alignment of standards and assessment: A theoretical and empirical study of methods for alignment
Open this publication in new window or tab >>Alignment of standards and assessment: A theoretical and empirical study of methods for alignment
2008 (English)In: Electronic Journal of Research in Educational Psychology, ISSN 1696-2095, Vol. 6, no 3, 667-690 p.Article in journal (Refereed) Published
Abstract [en]

Introduction. In a standards-based school-system alignment of policy documents with stan-dards and assessment is important. To be able to evaluate whether schools and students have reached the standards, the assessment should focus on the standards. Different models and methods can be used for measuring alignment, i.e. the correspondence between standards and assessment. Based on the assumption that a model must be able to include content and cogni-tive complexity, nine different models are identified and these models are then scrutinized with reference to defined theoretical criteria. The conclusion is that Bloom’s revised taxon-omy and Porter’s taxonomy are the most appropriate models.

Method. Bloom’s revised taxonomy and Porter’s taxonomy are compared based on empirical data from standards and assessment in a chemistry course in upper secondary schools in Swe-den. The comparison is based on five rules and of inter-rater reliability.

Results. Bloom’s revised taxonomy was more inclusive and exclusive than Porter’s taxon-omy. The inter-rater reliability for classification of standards was significantly better for Bloom’s revised taxonomy than for Porter’s taxonomy.

Conclusion. Based on the five rules, the conclusion is that Bloom’s revised taxonomy is the best model.

Keyword
Alignment, standards, assessment, Bloom's revised taxonomi, Porter's taxonomy
Identifiers
urn:nbn:se:umu:diva-11292 (URN)
Available from: 2008-12-09 Created: 2008-12-09 Last updated: 2011-01-10Bibliographically approved
2. Interpretation of standards with Bloom's revised taxonomy: a comparison of teachers and assessment experts
Open this publication in new window or tab >>Interpretation of standards with Bloom's revised taxonomy: a comparison of teachers and assessment experts
2009 (English)In: International Journal of Research and Method in Education, ISSN 1743-727X, E-ISSN 1743-7288, Vol. 32, no 1, 39-51 p.Article in journal (Refereed) Published
Abstract [en]

In education, standards have to be interpreted, for planning of teaching, for development of assessments and for alignment analysis. In most cases, it is important that there is an agreement between individuals and organizations about how to interpret standards. However, there is a lack of studies of how consistent different group of judges are when interpreting standards. In this study, the usefulness of Bloom’s revised taxonomy for interpreting standards in mathematics is evaluated, using different criteria. The results indicate that the taxonomy is an acceptable tool. The results also indicate that there are differences between the panel composed of teachers and the panel composed of assessment experts. The assessment experts were more consistent in their interpretation of standards. Limitations of the study and requirements for alignment analysis are discussed.

Place, publisher, year, edition, pages
London: Routledge, 2009
Keyword
standards, Bloom’s revised taxonomy, inter-judge consistency, intrajudge
National Category
Pedagogy
Identifiers
urn:nbn:se:umu:diva-31051 (URN)10.1080/17437270902749262 (DOI)
Available from: 2010-01-27 Created: 2010-01-27 Last updated: 2017-12-12Bibliographically approved
3. Alignment between standards and assessment: An evaluation of the usefulness of Bloom's revised taxonomy
Open this publication in new window or tab >>Alignment between standards and assessment: An evaluation of the usefulness of Bloom's revised taxonomy
Article in journal (Refereed) Submitted
Identifiers
urn:nbn:se:umu:diva-3507 (URN)
Available from: 2008-09-30 Created: 2008-09-30Bibliographically approved
4. Interpretation of standards with Bloom's revised taxonomy: Does a division influence its usefulness?
Open this publication in new window or tab >>Interpretation of standards with Bloom's revised taxonomy: Does a division influence its usefulness?
Article in journal (Refereed) Submitted
Identifiers
urn:nbn:se:umu:diva-3508 (URN)
Available from: 2008-09-30 Created: 2008-09-30Bibliographically approved
5. A comparison of two different methods for setting performance standards for a test with constructed-response items
Open this publication in new window or tab >>A comparison of two different methods for setting performance standards for a test with constructed-response items
2008 (English)In: Practical Assessment, Research & Evaluation, ISSN 1531-7714, Vol. 13, no 9, 12- p.Article in journal (Refereed) Published
Abstract [en]

The trustworthiness of performance standards influences the credibility of criterion-referenced large-scale testing. In this paper, two standard-setting methods are evaluated and compared, when applied to a test with polytomously scored constructed-response items. A version of the Angoff method is chosen as representative of the class of test-centred standard-setting procedures and the borderline-group method represents the class of examinee-centred procedures. The evaluation is based on procedural, internal and external evidence. The results indicate that both methods provide reasonable and trustworthy approaches to standard setting, but also confirm some of the potential problems with these methods.

Place, publisher, year, edition, pages
College Park, Md.: ERIC Clearinghouse on Assessment and Evaluation and the Department of Measurement, Statistics, and Evaluation at the University of Maryland, 2008
Identifiers
urn:nbn:se:umu:diva-3509 (URN)
Available from: 2008-09-30 Created: 2008-09-30 Last updated: 2011-03-28Bibliographically approved

Open Access in DiVA

fulltext(883 kB)3519 downloads
File information
File name FULLTEXT01.pdfFile size 883 kBChecksum SHA-1
aeb6e32242b1962d85cffa6b43582fadb5eb84d44c615ef2f7c86ef80cb839de977d4ad8
Type fulltextMimetype application/pdf

Search in DiVA

By author/editor
Näsström, Gunilla
By organisation
Department of Educational Measurement
Manufacturing, Surface and Joining Technology

Search outside of DiVA

GoogleGoogle Scholar
Total: 3519 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 2717 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf