Are you ready to implement CAT?
If you have heard of the underlying principles of computer-adaptive testing (CAT), you will know the concept is relatively straightforward (avoiding the obvious discussion of the complex algorithms underlying this concept). That however does not necessarily mean it is appropriate for you or your organisation.
CAT is a form of computer-based testing which can identify at which level a person is performing, i.e. what their 'ability level' is. This is achieved through tailoring the questions presented to a test taker based on their response pattern. CAT involves administering questions to individuals from an item bank with calibrated difficulties, pin-pointing the pathway an individual takes to finally attain their ability estimate.
In the pathway diagram below, the cut-score, or 'pass mark' of the test / exam is represented by the dotted horizontal line, the ability estimate of the individual is represented by the circles, and the standard error of measurement (SEM; not to be confused with the concept of standard error of the mean) is shown by the vertical lines associated with the ability estimates.
You can see that the individual was initially administered a question with a difficulty just above the cut score in this instance. They answered the question correctly, and thus, this question is coloured in black. The following three questions were answered incorrectly, and as such are represented by red lines. You will see how the ability estimate also decreases accordingly.
As the test continues, you will note that the candidate's ability estimate converges, and that the SEM range also decreases. At the point where the candidate has answered approximately 100 questions, it is evident that they are consistently performing below the cut-score, resulting in a fail in this test / exam. (You may also note the spaces between the questions; these are non-scored items in the test / exam).
So simply, CAT is magic! But, whether you or your organisation are ready to implement a CAT program, and whether you can maintain such a program is another issue entirely.
There are a few things which need to happen before you can implement and run a CAT. Some of the practical considerations include;
whether you have a calibrated item bank - and how many questions are in your item bank
whether you are in a position to maintain and update your item bank moving forward
what test administration procedure will suit your organisation and needs and whether this is in your budget
how to ensure the test and item bank is secure
whether you will be able to manage examinee / candidate communication appropriately
Once such issues have been investigated, planned for and resolved, you may be able to consider applying CAT to your testing programs in order to achieve efficient, effective and precise results upon which to make decisions. If you want to have a discussion on any of these issues, we would be happy to assist. Please contact us at admin@epecat.com for more information.
