Performance of doctors in the UK GP licensing Applied Knowledge Test for different question lengths

Problem

The phrasing and length of questions in written examinations may affect candidate performance. It is not known whether this affects performance of candidates that have difficulties processing written language including to a greater extent those with a specific learning difficulty (SpLD) or who have English as a second language. The Applied Knowledge Test (AKT) consists of computer administered multiple-choice questions with stems of varying length. The aim of this study was to investigate how candidates with a declared SpLD and candidates of different ethnicity or those graduating outside the UK performed on longer questions.

Approach

We used a retrospective study design examining data comprising candidates’ results starting with AKT23 up to AKT44. Questions were divided into seven categories (no scenario, 1 line, 2 lines, 3 lines, 4 lines, 5 lines, and 6 or over 6 lines). The proportion of correct responses was calculated for each question length category and for each candidate. Multivariable predictive models were run using question length as the outcome and a declared SpLD, ethnicity and country of primary medical qualification (PMQ) as the main predictors.

Findings

We included 18,925 candidates who undertook the AKT23 to AKT44 of different ethnicity (White British 54.12%, minority ethnic 42.95% or mixed 2.93%), country of qualification (UK 73.99%, non-UK 26.01%), and declared disability (disability declared 4.86%, no disability declared 95.14%). All candidates had a better performance on longer (6 or more lines) questions with an average of 82% of correct responses compared to an average score varying between 74% and 76% for other question length categories (p<0.001). Multivariable regression models revealed that candidates of non-UK PMQ, White ethnicity and with a SpLD performed significantly less well on all question length categories (p<0.001) except for candidates with a SpLD on questions of 6-lines or more which was comparable to that of candidates without a SpLD (B= -0.01, 95%CI -0.03, 0.01, p=0.247).

Consequences

These findings indicate that performance on different question lengths reflects the general performance of candidates with a declared SpLD, or of different ethnicity and country of qualification. The longest questions seem to favour performance for all candidates especially for candidates with SpLD, suggesting a greater ease of answering (higher facility) for these questions. Further research is needed to understand candidates’ perceptions of differences in question facility related to question length.