Umeå University's logo

umu.sePublikasjoner
Endre søk
RefereraExporteraLink to record
Permanent link

Direct link
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annet format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annet språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf
Within-item response processes as indicators of test-taking effort and motivation
Umeå universitet, Samhällsvetenskapliga fakulteten, Institutionen för tillämpad utbildningsvetenskap.
Umeå universitet, Samhällsvetenskapliga fakulteten, Institutionen för tillämpad utbildningsvetenskap.ORCID-id: 0000-0002-4630-6123
2020 (engelsk)Inngår i: Educational Research and Evaluation, ISSN 1380-3611, E-ISSN 1744-4187, Vol. 26, nr 5-6Artikkel i tidsskrift (Fagfellevurdert) Published
Abstract [en]

The present study used process data from a computer-based problem-solving task as indications of behavioural level of test-taking effort, and explored how behavioural item-level effort related to overall test performance and self-reported effort. Variables were extracted from raw process data and clustered. Four distinct clusters were obtained and characterised as high effort, medium effort, low effort, and planner. Regression modelling indicated that among students that failed to solve the task, level of effort invested before giving up positively predicted overall test performance. Among students that solved the task, level of effort was instead weakly negatively related to test performance. A low level of behavioural effort before giving up the task was also related to lower self-reported effort. Results suggest that effort invested before giving up provides information about test-takers’ motivation to spend effort on the test. We conclude that process data could augment existing methods of assessing test-taking effort.

sted, utgiver, år, opplag, sider
Taylor & Francis Group, 2020. Vol. 26, nr 5-6
Emneord [en]
computer-based assessment, PISA 2012, problem solving, Process data, test-taking effort, test-taking motivation
HSV kategori
Identifikatorer
URN: urn:nbn:se:umu:diva-186835DOI: 10.1080/13803611.2021.1963940ISI: 000684127100001Scopus ID: 2-s2.0-85112337645OAI: oai:DiVA.org:umu-186835DiVA, id: diva2:1590752
Tilgjengelig fra: 2021-09-03 Laget: 2021-09-03 Sist oppdatert: 2023-04-19bibliografisk kontrollert
Inngår i avhandling
1. Exploring and modeling response process data from PISA: inferences related to motivation and problem-solving
Åpne denne publikasjonen i ny fane eller vindu >>Exploring and modeling response process data from PISA: inferences related to motivation and problem-solving
2023 (engelsk)Doktoravhandling, med artikler (Annet vitenskapelig)
Alternativ tittel[sv]
Modellering av responsprocessdata från PISA : inferenser relaterade till motivation och problemlösning
Abstract [en]

This thesis explores and models response process data from large-scale assessments, focusing on test-taking motivation, problem-solving strategies, and questionnaire response validity. It consists of four studies, all using data from PISA (Programme for International Student Assessment) data.

Study I processed and clustered log-file data to create a behavioral evaluation of students' effort applied to a PISA problem-solving item, and examined the relationship between students' behavioral effort, self-reported effort, and test performance. Results show that effort invested before leaving the task unsolved was positively related to performance, while effort invested before solving the tasks was not. Low effort before leaving the task unsolved was further related to lower self-reported effort. The findings suggest that test-taking motivation could only be validly measured from efforts exerted before giving up.

Study II used response process data to infer students' problem-solving strategies on a PISA problem-solving task, and investigated the efficiency of strategies and their relationship to PISA performance. A text classifier trained on data from a generative computational model was used to retrieve different strategies, reaching a classification accuracy of 0.72, which increased to 0.90 with item design changes. The most efficient strategies used information from the task environment to make plans. Test-takers classified as selecting actions randomly performed worse overall. The study concludes that computational modeling can inform score interpretation and item design.

Study III investigated the relationship between motivation to answer the PISA student questionnaire and test performance. Departing from the theory of satisficing in surveys a Bayesian finite mixture model was developed to assess questionnaire-taking motivation. Results showed that overall motivation was high, but decreased toward the end. The questionnaire-taking motivation was positively related to performance, suggesting that it could be a proxy for test-taking motivation, however, reading skills may affect the estimation.

Study IV examines the validity of composite scores assessing reading metacognition, using a Bayesian finite mixture model that jointly considers response times and sequential patterns in subitem responses. The results show that, the relatively high levels of satisficing (up to 30%) negatively biased composite scores. The study highlights the importance of considering response time data and subitem response patterns when the validity of scores from the student questionnaire.

In conclusion, response process data from international large-scale assessments can provide valuable insights into test-takers’ motivation, problem-solving strategies, and questionnaire validity.

sted, utgiver, år, opplag, sider
Umeå: Umeå University, 2023. s. 53
Serie
Academic dissertations at the department of Educational Measurement, ISSN 1652-9650 ; 15
Emneord
response processes, large-scale assessments, motivation, problem-solving, computational modeling, Bayesian modeling
HSV kategori
Forskningsprogram
beteendevetenskapliga mätningar
Identifikatorer
urn:nbn:se:umu:diva-206866 (URN)978-91-8070-058-0 (ISBN)978-91-8070-057-3 (ISBN)
Disputas
2023-05-17, Aula Biologica, Umeå, 10:00 (engelsk)
Opponent
Veileder
Tilgjengelig fra: 2023-04-26 Laget: 2023-04-19 Sist oppdatert: 2023-04-19bibliografisk kontrollert

Open Access i DiVA

fulltext(3594 kB)206 nedlastinger
Filinformasjon
Fil FULLTEXT02.pdfFilstørrelse 3594 kBChecksum SHA-512
25f4c4a04c472c5573775fbc9f16318caf1156db9755e902f1e910a9e8d84666bf12a007a64d0e599e14e4754410c466e4c726131bac43882019b019e520d625
Type fulltextMimetype application/pdf

Andre lenker

Forlagets fulltekstScopus

Person

Lundgren, ErikEklöf, Hanna

Søk i DiVA

Av forfatter/redaktør
Lundgren, ErikEklöf, Hanna
Av organisasjonen
I samme tidsskrift
Educational Research and Evaluation

Søk utenfor DiVA

GoogleGoogle Scholar
Totalt: 269 nedlastinger
Antall nedlastinger er summen av alle nedlastinger av alle fulltekster. Det kan for eksempel være tidligere versjoner som er ikke lenger tilgjengelige

doi
urn-nbn

Altmetric

doi
urn-nbn
Totalt: 344 treff
RefereraExporteraLink to record
Permanent link

Direct link
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annet format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annet språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf