umu.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
The Importance of Sampling Weights in Multilevel Modeling of International Large-Scale Assessment Data
Umeå University, Faculty of Social Sciences, Umeå School of Business and Economics (USBE), Statistics.ORCID iD: 0000-0001-7282-5384
Umeå University, Faculty of Social Sciences, Umeå School of Business and Economics (USBE), Statistics.
(English)Manuscript (preprint) (Other academic)
Abstract [en]

Multilevel modeling is an important tool for analyzing large-scale assessment data. However, the standard multilevel modeling will typically give biased results for such complex survey data. This bias can be eliminated by introducing design weights, which must be used carefully as they can affect the results.

The aim of this paper is to examine different approaches and to give recommendations concerning handling design weights in multilevel models when analyzing large-scale assessments such as TIMSS (The Trends in International Mathematics and Science Study). To achieve the goal of the paper we examined real data from two countries and included a simulation study. The analyses in the empirical study showed that using no weights or only student-level weights sometimes could lead to misleading conclusions. The simulation study only showed small differences in estimation of the weighted and unweighted models when informative design weights were used. The use of unscaled or not rescaled weights however caused significant differences in some parameter estimates.

Keyword [en]
informative weights, multilevel models, rescaling weights, simulation study
National Category
Probability Theory and Statistics
Research subject
Statistics; Education
Identifiers
URN: urn:nbn:se:umu:diva-128588OAI: oai:DiVA.org:umu-128588DiVA: diva2:1052862
Available from: 2016-12-07 Created: 2016-12-07 Last updated: 2016-12-13
In thesis
1. Statistical modeling in international large-scale assessments
Open this publication in new window or tab >>Statistical modeling in international large-scale assessments
2016 (English)Doctoral thesis, comprehensive summary (Other academic)
Alternative title[sv]
Statistisk modellering i internationella komparativa mätningar
Abstract [en]

This thesis contributes to the area of research based on large-scale educational assessments, focusing on the application of multilevel models. The role of sampling weights, plausible values (response variable imputed multiple times) and imputation methods are demonstrated by simulations and applications to TIMSS (Trends in International Mathematics and Science Study) and PISA (Programme for International Student Assessment) data.

The large-scale assessments use multistage sampling design, which means that the units such as schools, classrooms, or students at some or all stages are selected with unequal probabilities. In order to make valid estimates and inferences sampling weights should be used. Thus, in the first paper, we examine different approaches and give recommendations concerning handling sampling weights in multilevel models when analyzing large-scale assessments.

Due to limitations in time and the number of students, the complex surveys use matrix sampling of items. This means that a response variable, i.e. students' performance, contains a large amount of information that is missing by design. Therefore, in order to estimate students' proficiency, TIMSS and PISA use the plausible values approach, which results in a set of five plausible values – proficiencies, computed for each student. In the second paper, different user strategies concerning plausible values for multilevel models as well as means and variances are examined with both real and simulated data. Missing information that is present because of the matrix sampling design for instance like the one used in PISA, can be arranged into a non-monotone missing data pattern, where all variables are incomplete and highly positively correlated. In the third paper, we compare a few imputation methods: a single imputation from a conditional distribution (with and without weights) and multiple imputation, for data with a non-monotone missing pattern (with no complete variables) and high positive correlation between variables.

In several of the recent international large-scale assessments, students in Sweden demonstrate a decreasing performance. Some previous research has shown that changes in performance depend on students’ performance levels. In the fourth paper, we studied the relationship between student performance and the between-school variance and tried to identify factors associated with student performance in mathematics in PISA in low-, medium-, and high- performing schools in the Nordic countries.

Place, publisher, year, edition, pages
Umeå: Umeå universitet, 2016. 18 p.
Series
Statistical studies, ISSN 1100-8989 ; 51
Keyword
multilevel model, plausible values, sampling weights, missing information, multiple imputation, non-monotone missing pattern, TIMSS, PISA
National Category
Probability Theory and Statistics
Research subject
Statistics; Education
Identifiers
urn:nbn:se:umu:diva-128618 (URN)978-91-7601-612-1 (ISBN)
Public defence
2017-01-12, Hörsal E, Humanisthuset, Umeå Universitet, Umeå, 10:00 (English)
Opponent
Supervisors
Available from: 2016-12-16 Created: 2016-12-09 Last updated: 2017-01-24Bibliographically approved

Open Access in DiVA

No full text

Search in DiVA

By author/editor
Laukaityte, IngaWiberg, Marie
By organisation
Statistics
Probability Theory and Statistics

Search outside of DiVA

GoogleGoogle Scholar

Total: 52 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf