umu.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Statistical modeling in international large-scale assessments
Umeå University, Faculty of Social Sciences, Umeå School of Business and Economics (USBE), Statistics.ORCID iD: 0000-0001-7282-5384
2016 (English)Doctoral thesis, comprehensive summary (Other academic)Alternative title
Statistisk modellering i internationella komparativa mätningar (Swedish)
Abstract [en]

This thesis contributes to the area of research based on large-scale educational assessments, focusing on the application of multilevel models. The role of sampling weights, plausible values (response variable imputed multiple times) and imputation methods are demonstrated by simulations and applications to TIMSS (Trends in International Mathematics and Science Study) and PISA (Programme for International Student Assessment) data.

The large-scale assessments use multistage sampling design, which means that the units such as schools, classrooms, or students at some or all stages are selected with unequal probabilities. In order to make valid estimates and inferences sampling weights should be used. Thus, in the first paper, we examine different approaches and give recommendations concerning handling sampling weights in multilevel models when analyzing large-scale assessments.

Due to limitations in time and the number of students, the complex surveys use matrix sampling of items. This means that a response variable, i.e. students' performance, contains a large amount of information that is missing by design. Therefore, in order to estimate students' proficiency, TIMSS and PISA use the plausible values approach, which results in a set of five plausible values – proficiencies, computed for each student. In the second paper, different user strategies concerning plausible values for multilevel models as well as means and variances are examined with both real and simulated data. Missing information that is present because of the matrix sampling design for instance like the one used in PISA, can be arranged into a non-monotone missing data pattern, where all variables are incomplete and highly positively correlated. In the third paper, we compare a few imputation methods: a single imputation from a conditional distribution (with and without weights) and multiple imputation, for data with a non-monotone missing pattern (with no complete variables) and high positive correlation between variables.

In several of the recent international large-scale assessments, students in Sweden demonstrate a decreasing performance. Some previous research has shown that changes in performance depend on students’ performance levels. In the fourth paper, we studied the relationship between student performance and the between-school variance and tried to identify factors associated with student performance in mathematics in PISA in low-, medium-, and high- performing schools in the Nordic countries.

Place, publisher, year, edition, pages
Umeå: Umeå universitet , 2016. , 18 p.
Series
Statistical studies, ISSN 1100-8989 ; 51
Keyword [en]
multilevel model, plausible values, sampling weights, missing information, multiple imputation, non-monotone missing pattern, TIMSS, PISA
National Category
Probability Theory and Statistics
Research subject
Statistics; Education
Identifiers
URN: urn:nbn:se:umu:diva-128618ISBN: 978-91-7601-612-1 (print)OAI: oai:DiVA.org:umu-128618DiVA: diva2:1054880
Public defence
2017-01-12, Hörsal E, Humanisthuset, Umeå Universitet, Umeå, 10:00 (English)
Opponent
Supervisors
Available from: 2016-12-16 Created: 2016-12-09 Last updated: 2017-01-24Bibliographically approved
List of papers
1. The Importance of Sampling Weights in Multilevel Modeling of International Large-Scale Assessment Data
Open this publication in new window or tab >>The Importance of Sampling Weights in Multilevel Modeling of International Large-Scale Assessment Data
(English)Manuscript (preprint) (Other academic)
Abstract [en]

Multilevel modeling is an important tool for analyzing large-scale assessment data. However, the standard multilevel modeling will typically give biased results for such complex survey data. This bias can be eliminated by introducing design weights, which must be used carefully as they can affect the results.

The aim of this paper is to examine different approaches and to give recommendations concerning handling design weights in multilevel models when analyzing large-scale assessments such as TIMSS (The Trends in International Mathematics and Science Study). To achieve the goal of the paper we examined real data from two countries and included a simulation study. The analyses in the empirical study showed that using no weights or only student-level weights sometimes could lead to misleading conclusions. The simulation study only showed small differences in estimation of the weighted and unweighted models when informative design weights were used. The use of unscaled or not rescaled weights however caused significant differences in some parameter estimates.

Keyword
informative weights, multilevel models, rescaling weights, simulation study
National Category
Probability Theory and Statistics
Research subject
Statistics; Education
Identifiers
urn:nbn:se:umu:diva-128588 (URN)
Available from: 2016-12-07 Created: 2016-12-07 Last updated: 2016-12-13
2. Using plausible values in secondary analysis in large–scale assessments
Open this publication in new window or tab >>Using plausible values in secondary analysis in large–scale assessments
(English)Manuscript (preprint) (Other academic)
Abstract [en]

Plausible values are typically used in large–scale assessment studies, in particular in the Trends in International Mathematics and Science Study and the Programme for International Student Assessment. Despite its large spread there are still some questions regarding the use of plausible values and how such use affects statistical analyses. The aim of this paper is to demonstrate the role of plausible values in large–scale assessment surveys when multilevel modelling is used. Different user strategies concerning plausible values for multilevel models as well as means and variances are examined. The results show that some commonly used user strategies give incorrect results while others give reasonable estimates but incorrect standard errors. These findings are important for anyone wishing to make secondary analyses of large–scale assessment data, especially those interested in using multilevel models to analyze the data.

Keyword
Achievement, design study, multilevel modelling, simulation studies, testing
National Category
Probability Theory and Statistics
Identifiers
urn:nbn:se:umu:diva-128615 (URN)
Available from: 2016-12-09 Created: 2016-12-09 Last updated: 2016-12-13
3. Single Imputation from a Conditional Distribution vs Multiple Imputation for Data with a Non-monotone Missing Pattern
Open this publication in new window or tab >>Single Imputation from a Conditional Distribution vs Multiple Imputation for Data with a Non-monotone Missing Pattern
(English)Manuscript (preprint) (Other academic)
Abstract [en]

Missing information is common in real data studies. When missingness is large, it should not be ignored and, instead, a missing data imputation method should be considered. The choice of the imputation method depends on the type or the pattern of missing information, as well as the nature of data. For instance, observations in large-scale educational assessments are incomplete by missing some components and based on usually positively correlated results within students. In all types of analysis of such data, the correlation has to be considered in a reliable calculation of properties of estimates. The aim of this paper is to compare a single imputation from a conditional distribution (with or without weights) and multiple imputation for data with a non-monotone missing pattern and high positive correlation between variables. For this purpose, such estimates as mean and variance are compared. The simulation results showed that expectation and variance are estimated more reliably when the imputation from a conditional distribution (without and with weights) or a complete-data set are used, compared to multiple imputation.

Keyword
Item-nonresponse, plausible values, highly correlated measurements, large-scale assessments, planed missingness
National Category
Probability Theory and Statistics
Identifiers
urn:nbn:se:umu:diva-128616 (URN)
Available from: 2016-12-09 Created: 2016-12-09 Last updated: 2016-12-09
4. Low-, Medium-, and High-performing Schools in the Nordic Countries: student Performance at PISA Mathematics 2003-2012
Open this publication in new window or tab >>Low-, Medium-, and High-performing Schools in the Nordic Countries: student Performance at PISA Mathematics 2003-2012
(English)Manuscript (preprint) (Other academic)
Abstract [en]

Decreasing performance among students in Sweden on international comparative studies and increasing segregation of schools, has led to a debate concerning strategies for improving student performance. The aim of this study is to analyse the between school variance and to identify factors associated with student performance in PISA in mathematics at different performance levels in the Nordic countries. In order to separate the effect of school-level variables, from the effect of student’s background factors and to take care of the multistage sampling design used in PISA, multilevel analysis was used. The results show that no evidence regarding the relationship between the average student performance in mathematics and the between-school variance was found which, is in contrast to previous studies conducted on science performance in PISA. Regarding school-level factors, our results overall have shown that few school-level factors (having a positive or a negative effect) seemed to be associated with performance. School-level factors associated with performance have mainly been identified among low- and medium-performing schools, and to a less extent among students at high-performing schools (only in Sweden and Denmark). This is a result which is in line with other studies showing the educational system’s incapacity to provide support for high-performing students and to enhance their learning.

Keyword
Performance levels, Between-school variance, Multilevel modelling, School-level factors, Large-scale comparative studies, School-effectiveness
National Category
Probability Theory and Statistics
Research subject
Education
Identifiers
urn:nbn:se:umu:diva-128617 (URN)
Projects
What can we learn from large-scale assessments about promoting student success?
Funder
Swedish Research Council, 2015-02160
Available from: 2016-12-09 Created: 2016-12-09 Last updated: 2016-12-13

Open Access in DiVA

omslag(132 kB)0 downloads
File information
File name COVER01.pngFile size 132 kBChecksum SHA-512
22e2100360069298b82d228c23454c61bd3b29d50397eecf96c76534b4914d266bea8adf56fe4fd416c281e09a6b837fcf3192d89f5a4c0e0745d7ad6948283f
Type coverMimetype image/png
spikblad(98 kB)17 downloads
File information
File name SPIKBLAD01.pdfFile size 98 kBChecksum SHA-512
f508db568d32708452c09b09af6c678f50e0e5bf27caa1daae428e2dfe645b6471684fc89cf45995bb742b2e70d0de64a3cc0f48839f623812bf37dfba6a957d
Type spikbladMimetype application/pdf
fulltext(535 kB)114 downloads
File information
File name FULLTEXT01.pdfFile size 535 kBChecksum SHA-512
3894f191921fdacd19e53b30c05acaf6998c3b18b48c0042697dccb320e9561a37afaedee299e9918eee47399dcaf2758a436386a0e73e83f0b8c3f79efa89b2
Type fulltextMimetype application/pdf

Search in DiVA

By author/editor
Laukaityte, Inga
By organisation
Statistics
Probability Theory and Statistics

Search outside of DiVA

GoogleGoogle Scholar
Total: 114 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

Total: 882 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf