Umeå University's logo

When mathematics tasks are used in multilanguage assessments, it is necessary that the task versions in the different languages are equivalent. The purpose of this study is to deepen the knowledge on different aspects of equivalence for mathematics tasks in multilanguage assessment. We analyze mathematics tasks from PISA 2012 given to students in English, German and Swedish. To measure formal equivalence, we examine three linguistic features of the task texts and compare between language versions. To measure functional equivalence, a Differential item functioning (DIF) analysis is conducted. In addition, we examine statistically if there is a relation between DIF and the differences regarding linguistic features. The results show that there is both DIF and differences regarding the linguistic features between different language versions for several PISA tasks. However, we found no statistical relation between the two phenomena.

The purpose of this study is to deepen the understanding of the relation between features of the text of mathematics tasks and the tasks’ demand of reading ability. Variations in students’ reading processes when they work with PISA mathematics tasks with high demand of reading ability are identified and analyzed. These variations can be related to linguistic features of words, phrases, or sentences in the tasks, which in turn can be possible sources for the high demand of reading ability.

Practicing mathematics is not possible without the use of language. To communicate mathematical content, not only words in natural language are used but also non-verbal forms of communication such as mathematical symbols, graphs, and diagrams. All these forms of communication can be seen as part of the language used when doing mathematics. When mathematics tasks are used to assess mathematical competence, it is important to know how language can affect students’ possibility to solve the task. In this thesis, two different but related aspects of the relation between language and mathematics tasks are investigated. The first aspect concerns linguistic features of written mathematics task that can make the task more difficult to read and/or to solve. These features may result in unnecessary and unwanted reading demands, that is, the task then partially assesses students’ reading ability instead of their mathematical ability. The second aspect concerns differences between different language versions of mathematics tasks used in multilanguage assessments. These differences may cause inequivalence between the language versions, that is, the task may be more difficult to solve for students of one language group than students of another. Therefore, the purpose of this thesis is to investigate some of the effects that language can have on written mathematics tasks, in particular, on the validity of mathematics assessments. The thesis focuses on unnecessary reading demands and inequivalence in multilanguage assessments. The data in this thesis are obtained from tasks of the Programme for International Student Assessment (PISA) 2012. The task texts and the student results on these tasks are analyzed quantitatively to identify the occurrence and possible sources of unnecessary reading demands and inequivalence. Think-aloud-protocols and task-based interviews of students who had worked with some of the tasks, serve to qualitatively identify possible sources of reading demands and inequivalence, respectively.

The results showed both unnecessary reading demands and inequivalence in some of the tasks. Some linguistic features were identified as possible sources of these reading demands, while others were not related to them. For example, sentence length was not related to reading demands of tasks in Swedish, whereas sentence structure was identified as a possible source of unnecessary reading demands. Some linguistic differences between different language versions of mathematics tasks were also identified as possible sources of inequivalence, and in addition there were curricular differences that were such potential sources. The findings of this thesis have implications for designing mathematics tasks both in one language and in multilingual settings. They may help to ensure validity of mathematics assessments, but also to make mathematics texts easier to understand for students in general.

When mathematics tasks are translated to different languages, there is a risk that the different language versions are not equivalent and display differential item functioning (DIF). In this study, we aimed to identify possible sources of DIF. We investigated whether differences in some linguistic features are related to DIF between the English (USA), German, and Swedish versions of mathematics tasks of the PISA 2012 assessment. The linguistic features chosen in this study are grammatical person, voice (active/passive), and sentence structure. We analyzed the three different language versions of 83 mathematics PISA tasks in three steps. First, we calculated the amount of differences in the three linguistic features between the language versions. Then, we calculated DIF, using the Mantel-Haenszel procedure pairwise for two language versions at a time. Finally, we searched for correlations between the amount of linguistic differences and DIF between the versions. The analysis showed that differences in linguistic features occurred between the language versions – differences in voice were most common – and that several items displayed intermediate or large level of DIF. Still, there were no statistical significant correlations between differences in linguistic features and DIF between the language versions, that is, there must be other sources of DIF.

The purpose of this study is to deepen the understanding of the relation between the language used in mathematics tasks and the difficulty in reading and solving the tasks. We examine issues of language both through linguistic features of tasks (word length, sentence length, task length, and information density) and through different natural languages used to formulate the tasks (English, German, and Swedish). Analyses of 83 PISA mathematics tasks reveal that tasks in German, when compared with English and Swedish, show stronger connections between the examined linguistic features of tasks and difficulty in reading and solving the tasks. We discuss if and how this result can be explained by general differences between the three languages.

When the results of international comparative studies such as PISA or TIMSS get published, they are discussed broadly in media and are used to influence politics and public opinion. To solve mathematics PISA tasks, students have to read and understand the task text. Still, since the mathematics tasks are primarily supposed to measure mathematical ability and not reading ability, it is important to avoid unnecessary demands of reading ability in the tasks. In addition, the different language versions of a task used in PISA might vary in reading difficulty. Such differences can result in differential item functioning (DIF), that is, that students with the same mathematical ability but from different countries have a different probability of answering the item correctly. One reason for DIF between language versions is that linguistic features can differ between language versions. In this study we focus on four different linguistic features that in earlier studies have shown connections to the difficulty of solving mathematics tasks (e.g., Abedi, Lord, & Plummer, 1997).

• Grammatical person, that is, if the text is written in first, second, or third person.
• Voice, that is, if active or passive voice is used in the text.
• Sentence structure, that is, how the sentences are built of main and subordinate clauses.
• Word order, that is, the order of subject, finite verb, and object in the sentence.

This study is part of a larger project examining the relation between the language used in mathematics tasks and both the tasks’ difficulty and demand of reading ability. The research questions in this study are: Which differences in the four linguistic features investigated occur between PISA tasks in English, German, and Swedish? Which of these differences are related to DIF between the task versions? The English (USA), German, and Swedish language versions of 83 mathematics tasks of the PISA 2012 assessment are analyzed. The first step of the analysis was to search for differences in the four linguistic features between the different language versions of the tasks. The next steps will be quantitative analyses of the differences, a statistical analysis to detect DIF between the versions, and then statistical analyses to investigate possible relations between the differences and DIF. The first step showed that some differences occur sporadically, for example, the use of third person (he/she/it) in one language version and second person (you) in another language version. Other differences occur much more frequently. For example, differences in word order are quite common, in particular since the finite verb always is at the last position in subordinate clauses in German but not in English and Swedish. The next steps of the analysis are at present (January 2017) ongoing.

The mathematics PISA tasks are primarily supposed to measure mathematical ability and not reading ability, so it is important to avoid unnecessary demands of reading ability in the tasks. Many readability formulas are using both word length and sentence length as indicators of text difficulty. In this study, we examine differences and similarities between English, German, and Swedish mathematics PISA tasks regarding word length and sentence length. We analyze 146 mathematics PISA tasks from 2000–2013, in English, German, and Swedish. For each task we create measures of mean word and sentence length. To analyze if there are any differences between the three language versions of the tasks, we use t-tests to compare the three languages pairwise. We found that in average, the German versions have the longest words, followed by Swedish and then English. Average sentence length was highest for English, followed by German and then Swedish.