umu.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
An optimal design approach to criterion-referenced computerized testing
Umeå University, Faculty of Social Sciences, Department of Statistics.
2003 (English)In: Journal of educational and behavioral statistics, ISSN 1076-9986, Vol. 28, no 2, 97-110 p.Article in journal (Refereed) Published
Abstract [en]

A criterion-referenced computerized test is expressed as a statistical hypothesis problem. This admits that it can be studied by using the theory of optimal design. The power function of the statistical test is used as a criterion function when designing the test. A formal proof is provided showing that all items should have the same item characteristics, i.e. items that have high discrimination, low guessing and difficulty near the cut-off score give the most powerful statistical test. An efficiency study shows how many times more items are needed if nonoptimal items are used instead of optimal items in order to get the same power in the test.

Place, publisher, year, edition, pages
2003. Vol. 28, no 2, 97-110 p.
Keyword [en]
criterion-referenced tests, efficiency, power maximization
Identifiers
URN: urn:nbn:se:umu:diva-2881DOI: 10.3102/10769986028002097OAI: oai:DiVA.org:umu-2881DiVA: diva2:141198
Available from: 2003-11-10 Created: 2003-11-10 Last updated: 2009-06-11Bibliographically approved
In thesis
1. Computerized achievement tests: sequential and fixed length tests
Open this publication in new window or tab >>Computerized achievement tests: sequential and fixed length tests
2003 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

The aim of this dissertation is to describe how a computerized achivement test can be constructed and used in practice. Throughout this dissertation the focus is on classifying the examinees into masters and non-masters depending on their ability. However, there has been no attempt to estimate their ability.

In paper I, a criterion-referenced computerized test with a fixed number of items is expressed as a statistical inference problem. The theory of optimal design is used to find the test that has the strongest power. A formal proof is provided showing that all items should have the same item characteristics, viz. high discrimination, low guessing and difficulty near the cutoff score, in order to give us the most powerful statistical test. An efficiency study shows how many times more non-optimal items are needed if we do not use optimal items in order to achieve the same power in the test.

In paper II, a computerized mastery sequential test is examined using sequential analysis. The focus is on examining the sequential probability ratio test and to minimize the number of items in a test, i.e. to minimize the average sample number function, abbreviated as the ASN function. Conditions under which the ASN function decreases are examined. Further, it is shown that the optimal values are the same for item discrimination and item guessing, but differ for item difficulty compared with tests with fixed number of items.

Paper III presents three simulation studies of sequential computerized mastery tests. Three cases are considered, viz. the examinees' responses are either identically distributed, not identically distributed, or not identically distributed together with estimation errors in the item characteristics. The simulations indicate that the observed results from the operating characteristic function differ significantly from the theoretical results. The mean number of items in a test, the distribution of test length and the variance depend on whether the true values of the item characteristics are known and whether they are iid or not.

In paper IV computerized tests with both pretested items with known item parameters, and try-out items with unknown item parameters are considered. The aim is to study how the item parameters for try-out items can be estimated in a computerized test. Although the unknown examinees' abilities may act as nuisance parameters, the asymptotic variance of the item parameter estimators can be calculated. Examples show that a more reliable variance estimator yields much larger estimates of the variance than commonly used variance estimators.

Place, publisher, year, edition, pages
Umeå: Statistiska institutionen, Umeå universitet, 2003. 24 p.
Series
Statistical studies, ISSN 1100-8989 ; 29
Keyword
Statistics, item respons theory, sequential analysis, statistical inference, power, optimal design, simulations, Statistik
National Category
Probability Theory and Statistics
Research subject
Statistics
Identifiers
urn:nbn:se:umu:diva-148 (URN)91-7305-539-5 (ISBN)
Distributor:
Statistiska institutionen, 90187, Umeå
Public defence
2003-12-05, Hörsal, Norra beteendevetarhuset, Umeå universitet, Umeå, 10:00 (English)
Opponent
Supervisors
Available from: 2003-11-10 Created: 2003-11-10 Last updated: 2011-06-22Bibliographically approved

Open Access in DiVA

No full text

Other links

Publisher's full text
By organisation
Department of Statistics
In the same journal
Journal of educational and behavioral statistics

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 49 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf