Should item review be allowed in a CAT?
As Computer Adaptive Testing (CAT) is being adopted, the controversial discussion relating to whether candidates should be allowed to review and change their responses under exam conditions has increased.
In comparable paper-and-pencil testing conditions, it has been argued that candidates would be able to alter previously selected answers at their leisure (time-permitting of course), and as such, this should also be accommodated for in CAT conditions, to ensure candidates are not disadvantaged through a change in the mode of assessment.
Conversely, it has been suggested that since CAT is based on selecting the most appropriate question for each individual candidate from an item-bank based on their previous response/s (also assuming that a continuous best effort is being made to answer each item), a review would be difficult to incorporate to test delivery since it could, theoretically, alter the path of questions the candidate could have been presented with.
When reviewing the literature, it is evident that candidates in a testing situation often experience greater satisfaction with the testing experience when given the opportunity to review their answers (Vispoel, Hendrickson and Bleiler, 2000). Thus, in some high-stakes testing situations, review of answers may be required without drastically altering the ability estimate the candidate would have attained.
EPEC has investigated, and enabled the review function for high-stakes testing conditions with various clients. Candidates have been able to review items, change responses and review again in the time allowed to sit the exam.
After administering over 10 000 CATs with the possibility of review, the correlation of 0.996 between initial ability estimates and final ability estimates was attained. So, although there may be a more significant difference in changing the ability estimates for some individual candidates it seems overall, the difference is negligible.
As such, it seems if candidate review is allowed in CATs, although there may not be a statistically significant change in their overall ability estimate, candidates may have a more positive experience of the overall testing process, and may experience greater sense of security in being able to review their answers.
The next question may be to investigate whether a 'skip' function should be allowed under testing conditions, and what effect this may have on the overall ability estimate, test length, and time required for test completion.
