Background International medical graduates need to show their proficiencies to obtain a medical license in a new country. In 2016 the Swedish National Board of Health and Welfare redesigned the proficiency test for Doctors of Medicine. The theoretical test consists of around 180 MCQ/SBA. All tests are made public after being used in accordance with the Swedish Public Access to Information and Secrecy Act. Thus, new tests have to be continuously developed. In addition, as a public institution, the test giver is required to reply to all challenges of the items on the test.
Summary of Work To involve the stakeholders and to improve the quality of the test, the items and corresponding answer keys were made public immediately after the test was finished. The test takers were invited to challenge any item and correct answer via a web form and support challenge with literature references. The item authors then reviewed and replied to each individual challenge. Based on these challenges and the associated review, definitive answer keys were decided. Three weeks after test admission, the test takers received their results, a copy of their test together with correct answers, and the response to their challenges.
Summary of Results So far, 13 different tests have been given to totally 2,328 test takers. 3,047 challenges were submitted and resulted in 87 changes in answer keys or deletions of an item from the test (3.7% of the 2,360 items across all 13 tests). The number of changes/deletions has decreased significantly with the number of developed tests.
Discussion & Conclusion We consider the number of test items that have been rejected or adjusted answer keys to be low. Nevertheless, it was necessary to build a system where the challenges from the test takers were built into the test process. In the Swedish educational system appealing a grade is not possible, but if any item in a test is incorrect, the 1992 administrative law requires test givers to promptly adjust test scores. While the implemented process is resource demanding, we believe that it is crucial both for test acceptance among the examinees and to improve the skills of the item authors.
Take-home Message Involving the test takers in quality assurance of a national licensing test might improve item quality.
The European Board of Medical Assessors (EBMA) , 2020.