Umeå University's logo

Modifications of current psychometric models for analyzing test data are proposed that produce an additive scale measure of information. This information measure is a one-dimensional space curve or curved surface manifold that is invariant across varying manifold indexing systems. The arc length along a curve manifold is used as it is an additive metric having a defined zero and a version of the bit as a unit. This property, referred to here as the scope of the test or an item, facilitates the evaluation of graphs and numerical summaries. The measurement power of the test is defined by the length of the manifold, and the performance or experiential level of a person by a position along the curve. In this study, we also use all information from the items including the information from the distractors. Test data from a large-scale college admissions test are used to illustrate the test information manifold perspective and to compare it with the well-known item response theory nominal model. It is illustrated that the use of information theory opens a vista of new ways of assessing item performance and inter-item dependency, as well as test takers' knowledge.

Test score equating is used to make scores from different test forms comparable, even when groups differ in ability. In practice, the non-equivalent group with anchor test (NEAT) design is commonly used. The overall aim was to compare the amount of bias under different conditions when using either chained equating or frequency estimation with five different criterion functions: the identity function, linear equating, equipercentile, chained equating and frequency estimation. We used real test data from a multiple-choice binary scored college admissions test to illustrate that the choice of criterion function matter. Further, we simulated data in line with the empirical data to examine difference in ability between groups, difference in item difficulty, difference in anchor test form and regular test form length, difference in correlations between anchor test form and regular test forms, and different sample size. The results indicate that how bias is defined heavily affects the conclusions we draw about which equating method is to be preferred in different scenarios. Practical implications of this in standardized tests are given together with recommendations on how to calculate bias when evaluating equating transformations.

Generalized Kernel Equating is a comprehensive guide for statisticians, psychometricians, and educational researchers aiming to master test score equating. This book introduces the Generalized Kernel Equating (GKE) framework, providing the necessary tools and methodologies for accurate and fair score comparisons.

The book presents test score equating as a statistical problem and covers all commonly used data collection designs. It details the five steps of the GKE framework: presmoothing, estimating score probabilities, continuization, equating transformation, and evaluating the equating transformation. Various presmoothing strategies are explored, including log-linear models, item response theory models, beta4 models, and discrete kernel estimators. The estimation of score probabilities when using IRT models is described and Gaussian kernel continuization is extended to other kernels such as uniform, logistic, epanechnikov and adaptive kernels. Several bandwidth selection methods are described. The kernel equating transformation and variants of it are defined, and both equating-specific and statistical measures for evaluating equating transformations are included. Real data examples, guiding readers through the GKE steps with detailed R code and explanations are provided. Readers are equipped with an advanced knowledge and practical skills for implementing test score equating methods.

In the fields of educational measurement and testing, score distributions are often estimated by the sample relative frequency distribution. As many score distributions are discrete and may have irregularities, it has been common practice to use presmoothing techniques to correct for such irregularities of the score distributions. A common way to conduct presmoothing has been to use log-linear models. In this chapter, we introduce a novel class of discrete kernels that can effectively estimate the probability mass function of scores, providing a presmoothing solution. The chapter includes an empirical illustration demonstrating that the proposed discrete kernel estimates perform as well as or better than the existing methods like log-linear models in presmoothing score distributions. The practical implications of this finding are discussed, highlighting the potential benefits of using discrete kernels in educational measurement contexts. Additionally, the chapter identifies several areas for further research, indicating opportunities for advancing the field’s methodology and practices.

Item response theory (IRT) models the relationship between the possible scores on a test item against a test taker’s attainment of the latent trait that the item is intended to measure. In this study, we compare two models for tests with polytomously scored items: the optimal scoring (OS) model, a nonparametric IRT model based on the principles of information theory, and the generalized partial credit (GPC) model, a widely used parametric alternative. We evaluate these models using both simulated and real test data. In the real data examples, the OS model demonstrates superior model fit compared to the GPC model across all analyzed datasets. In our simulation study, the OS model outperforms the GPC model in terms of bias, but at the cost of larger standard errors for the probabilities along the estimated item response functions. Furthermore, we illustrate how surprisal arc length, an IRT scale invariant measure of ability with metric properties, can be used to put scores from vastly different types of IRT models on a common scale. We also demonstrate how arc length can be a viable alternative to sum scores for scoring test takers.

While well-being does generally constitute a moderate predictor of school achievement, research on the predictive validity of cognitive ability for well-being in school contexts remains scant. The current study analyzed longitudinal relations between cognitive ability measured at age 13 (Grade 6) and well-being measured at age 18 (Grade 12, valid N = 2,705) in a Swedish sample, using several multivariate model techniques. The results indicate that cognitive ability was not a statistically significant predictor when several predictors were entered in a multiple regression model. However, gender was a significant covariate as girls and young women have a substantially lower degree of self-reported well-being. This casts light on the limitations of cognitive ability as a construct for some non-cognitive outcomes, at least in shorter and narrower spatial–temporal contexts.

The aim of the current study is to introduce three assumptions common to psychometric theory and psychometric practice, and to show how alternatives to traditional psychometric approaches can be used to improve psychological measurement. These alternatives are developed by adapting each of these three assumptions. The assumption of structural validity relates to the implementation of mathematical models. The process assumption regards the underlying process generating the observed data. The construct assumption implies that the observed data on its own do not constitute a measurement, but the latent variable that originates the observed data. Nonparametric item response modeling and cognitive psychometric modeling are presented as alternatives for relaxing the first two assumptions, respectively. Network psychometrics is the alternative for relaxing the third assumption. Final remarks sum up the most important conclusions of the study.

The overall aim was to examine effects of differences in group ability and features of the anchor test formon equating bias and the standard error of equating (SEE) using both real and simulated data. Chainedkernel equating, Postratification kernel equating, and Circle-arc equating were studied. A collegeadmissions test with four different anchor test forms administered at three test administrations was used.The simulation study examined the differences in ability of the test groups, and differences in the anchortest form with respect to item difficulty and discrimination. In the empirical study, the equated valuesfrom the three methods only slightly differed. The simulation study indicated that an easier anchor testform and/or an easier regular test form, and anchor items with a wider spread in difficulty, negativelyaffected the SEE and bias. The ability level of groups was also important. Equating with only less or morecapable groups resulted in high SEEs at higher and lower test scores, respectively. The discussion includespractical recommendations to whom an anchor test should be given if there is a choice and how to selectan anchor test form which have equating as primary purpose.

In norm-referenced psychological testing, an individual’s performance is expressed in relation to a reference population using a standardized score, like an intelligence quotient score. The reference population can depend on a continuous variable, like age. Current continuous norming methods transform the raw score into an age-dependent standardized score. Such methods have the shortcoming to solely rely on the raw test scores, ignoring valuable information from individual item responses. Instead of modeling the raw test scores, we propose modeling the item scores with a Bayesian two-parameter logistic (2PL) item response theory model with age-dependent mean and variance of the latent trait distribution, 2PL-norm for short. Norms are then derived using the estimated latent trait score and the age-dependent distribution parameters. Simulations show that 2PL-norms are overall more accurate than those from the most popular raw score-based norming methods cNORM and generalized additive models for location, scale, and shape (GAMLSS). Furthermore, the credible intervals of 2PL-norm exhibit clearly superior coverage over the confidence intervals of the raw score-based methods. The only issue of 2PL-norm is its slightly lower performance at the tails of the norms. Among the raw score-based norming methods, GAMLSS outperforms cNORM. For empirical practice this suggests the use of 2PL-norm, if the model assumptions hold. If not, or the interest is solely in the point estimates of the extreme trait positions, GAMLSS-based norming is a better alternative. The use of the 2PL-norm is illustrated and compared with GAMLSS and cNORM using empirical data, and code is provided, so that users can readily apply 2PL-norm to their normative data.