Umeå universitets logga

umu.sePublikationer
Ändra sökning
RefereraExporteraLänk till posten
Permanent länk

Direktlänk
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annat format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annat språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf
Exploring and modeling response process data from PISA: inferences related to motivation and problem-solving
Umeå universitet, Samhällsvetenskapliga fakulteten, Institutionen för tillämpad utbildningsvetenskap, Beteendevetenskapliga mätningar (BVM).ORCID-id: 0000-0002-6998-3397
2023 (Engelska)Doktorsavhandling, sammanläggning (Övrigt vetenskapligt)Alternativ titel
Modellering av responsprocessdata från PISA : inferenser relaterade till motivation och problemlösning (Svenska)
Abstract [en]

This thesis explores and models response process data from large-scale assessments, focusing on test-taking motivation, problem-solving strategies, and questionnaire response validity. It consists of four studies, all using data from PISA (Programme for International Student Assessment) data.

Study I processed and clustered log-file data to create a behavioral evaluation of students' effort applied to a PISA problem-solving item, and examined the relationship between students' behavioral effort, self-reported effort, and test performance. Results show that effort invested before leaving the task unsolved was positively related to performance, while effort invested before solving the tasks was not. Low effort before leaving the task unsolved was further related to lower self-reported effort. The findings suggest that test-taking motivation could only be validly measured from efforts exerted before giving up.

Study II used response process data to infer students' problem-solving strategies on a PISA problem-solving task, and investigated the efficiency of strategies and their relationship to PISA performance. A text classifier trained on data from a generative computational model was used to retrieve different strategies, reaching a classification accuracy of 0.72, which increased to 0.90 with item design changes. The most efficient strategies used information from the task environment to make plans. Test-takers classified as selecting actions randomly performed worse overall. The study concludes that computational modeling can inform score interpretation and item design.

Study III investigated the relationship between motivation to answer the PISA student questionnaire and test performance. Departing from the theory of satisficing in surveys a Bayesian finite mixture model was developed to assess questionnaire-taking motivation. Results showed that overall motivation was high, but decreased toward the end. The questionnaire-taking motivation was positively related to performance, suggesting that it could be a proxy for test-taking motivation, however, reading skills may affect the estimation.

Study IV examines the validity of composite scores assessing reading metacognition, using a Bayesian finite mixture model that jointly considers response times and sequential patterns in subitem responses. The results show that, the relatively high levels of satisficing (up to 30%) negatively biased composite scores. The study highlights the importance of considering response time data and subitem response patterns when the validity of scores from the student questionnaire.

In conclusion, response process data from international large-scale assessments can provide valuable insights into test-takers’ motivation, problem-solving strategies, and questionnaire validity.

Ort, förlag, år, upplaga, sidor
Umeå: Umeå University , 2023. , s. 53
Serie
Academic dissertations at the department of Educational Measurement, ISSN 1652-9650 ; 15
Nyckelord [en]
response processes, large-scale assessments, motivation, problem-solving, computational modeling, Bayesian modeling
Nationell ämneskategori
Övrig annan samhällsvetenskap
Forskningsämne
beteendevetenskapliga mätningar
Identifikatorer
URN: urn:nbn:se:umu:diva-206866ISBN: 978-91-8070-058-0 (digital)ISBN: 978-91-8070-057-3 (tryckt)OAI: oai:DiVA.org:umu-206866DiVA, id: diva2:1751778
Disputation
2023-05-17, Aula Biologica, Umeå, 10:00 (Engelska)
Opponent
Handledare
Tillgänglig från: 2023-04-26 Skapad: 2023-04-19 Senast uppdaterad: 2023-04-19Bibliografiskt granskad
Delarbeten
1. Within-item response processes as indicators of test-taking effort and motivation
Öppna denna publikation i ny flik eller fönster >>Within-item response processes as indicators of test-taking effort and motivation
2020 (Engelska)Ingår i: Educational Research and Evaluation, ISSN 1380-3611, E-ISSN 1744-4187, Vol. 26, nr 5-6Artikel i tidskrift (Refereegranskat) Published
Abstract [en]

The present study used process data from a computer-based problem-solving task as indications of behavioural level of test-taking effort, and explored how behavioural item-level effort related to overall test performance and self-reported effort. Variables were extracted from raw process data and clustered. Four distinct clusters were obtained and characterised as high effort, medium effort, low effort, and planner. Regression modelling indicated that among students that failed to solve the task, level of effort invested before giving up positively predicted overall test performance. Among students that solved the task, level of effort was instead weakly negatively related to test performance. A low level of behavioural effort before giving up the task was also related to lower self-reported effort. Results suggest that effort invested before giving up provides information about test-takers’ motivation to spend effort on the test. We conclude that process data could augment existing methods of assessing test-taking effort.

Ort, förlag, år, upplaga, sidor
Taylor & Francis Group, 2020
Nyckelord
computer-based assessment, PISA 2012, problem solving, Process data, test-taking effort, test-taking motivation
Nationell ämneskategori
Lärande
Identifikatorer
urn:nbn:se:umu:diva-186835 (URN)10.1080/13803611.2021.1963940 (DOI)000684127100001 ()2-s2.0-85112337645 (Scopus ID)
Tillgänglig från: 2021-09-03 Skapad: 2021-09-03 Senast uppdaterad: 2023-04-19Bibliografiskt granskad
2. Latent program modeling: Inferring latent problem-solving strategies from a PISA problem-solving task
Öppna denna publikation i ny flik eller fönster >>Latent program modeling: Inferring latent problem-solving strategies from a PISA problem-solving task
2022 (Engelska)Ingår i: Journal of Educational Data Mining, E-ISSN 2157-2100, Vol. 14, nr 1, s. 46-80Artikel i tidskrift (Refereegranskat) Published
Abstract [en]

Response process data have the potential to provide a rich description of test-takers’ thinking processes. However, retrieving insights from these data presents a challenge for educational assessments and educational data mining as they are complex and not well annotated. The present study addresses this challenge by developing a computational model that simulates how different problem-solving strategies would behave while searching for a solution to a Program for International Student Assessment (PISA) 2012 problem-solving item, and uses n-gram processing of data together with a naïve Bayesian classifier to infer latent problem-solving strategies from the test-takers’ response process data. The retrieval of simulated strategies improved with increased n-gram length, reaching an accuracy of 0.72 on the original PISA task. Applying the model to generalized versions of the task showed that classification accuracy increased with problem size and the mean number of actions, reaching a classification accuracy of 0.90 for certain task versions. The strategy that was most efficient and effective in the PISA Traffic task evaluated paths based on the labeled travel time. However, in generalized versions of the task, a straight line strategy was more effective. When applying the classifier to empirical data, most test-takers were classified as using a random path strategy (46%). Test-takers classified as using the travel time strategy had the highest probability of solving the task (p̂ ≈ 1). The test-takers classified as using the random actions strategy had the lowest probability of solving the task (p̂ ≈ 0.11). The effect of (classified) strategy on general PISA problem-solving performance was overall weak, except for a negative effect for the random actions strategy (β ≈ −65, CI95% ≈ [−96, −36]). The study contributes with a novel approach to inferring latent problem-solving strategies from action sequences. The study also illustrates how simulations can provide valuable information about item design by exploring how changing item properties could affect the accuracy of inferences about unobserved problem-solving strategies.

Ort, förlag, år, upplaga, sidor
International Educational Data Mining Society, 2022
Nyckelord
computational cognitive modeling, educational assessment, PISA, problem-solving, process data
Nationell ämneskategori
Tillämpad psykologi
Identifikatorer
urn:nbn:se:umu:diva-203575 (URN)10.5281/zenodo.6686443 (DOI)2-s2.0-85145815648 (Scopus ID)
Tillgänglig från: 2023-01-19 Skapad: 2023-01-19 Senast uppdaterad: 2023-04-19Bibliografiskt granskad
3. Questionnaire-Taking Motivation: Using response times to assess motivation to optimize in the PISA student questionnaire
Öppna denna publikation i ny flik eller fönster >>Questionnaire-Taking Motivation: Using response times to assess motivation to optimize in the PISA student questionnaire
(Engelska)Manuskript (preprint) (Övrigt vetenskapligt)
Nationell ämneskategori
Övrig annan samhällsvetenskap
Forskningsämne
beteendevetenskapliga mätningar
Identifikatorer
urn:nbn:se:umu:diva-206863 (URN)
Tillgänglig från: 2023-04-19 Skapad: 2023-04-19 Senast uppdaterad: 2023-04-19
4. Investigating satisficing in PISA 2018 questionnaire items by jointly modeling response times and subitem responses
Öppna denna publikation i ny flik eller fönster >>Investigating satisficing in PISA 2018 questionnaire items by jointly modeling response times and subitem responses
(Engelska)Manuskript (preprint) (Övrigt vetenskapligt)
Nationell ämneskategori
Övrig annan samhällsvetenskap
Forskningsämne
beteendevetenskapliga mätningar
Identifikatorer
urn:nbn:se:umu:diva-206865 (URN)
Tillgänglig från: 2023-04-19 Skapad: 2023-04-19 Senast uppdaterad: 2023-04-19

Open Access i DiVA

fulltext(1097 kB)318 nedladdningar
Filinformation
Filnamn FULLTEXT01.pdfFilstorlek 1097 kBChecksumma SHA-512
62e398bedb080b294ebb48cc58f8e51ac9c6423478187f64601834dca17d0e81457c7fa5bf0b30edff1b279badb6257f1ae594fc8883028a3a57bda2d06ce81a
Typ fulltextMimetyp application/pdf
spikblad(104 kB)49 nedladdningar
Filinformation
Filnamn FULLTEXT02.pdfFilstorlek 104 kBChecksumma SHA-512
6c20fda6d0a6f229719020c892cc43cba4878d1c54aee790dbcc2b71750b4330a64c2c2f012c01ce69652280246d19a84cd4c5d6ed82c420807e4319726b855b
Typ spikbladMimetyp application/pdf

Person

Lundgren, Erik

Sök vidare i DiVA

Av författaren/redaktören
Lundgren, Erik
Av organisationen
Beteendevetenskapliga mätningar (BVM)
Övrig annan samhällsvetenskap

Sök vidare utanför DiVA

GoogleGoogle Scholar
Totalt: 368 nedladdningar
Antalet nedladdningar är summan av nedladdningar för alla fulltexter. Det kan inkludera t.ex tidigare versioner som nu inte längre är tillgängliga.

isbn
urn-nbn

Altmetricpoäng

isbn
urn-nbn
Totalt: 956 träffar
RefereraExporteraLänk till posten
Permanent länk

Direktlänk
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annat format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annat språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf