How should we measure performance when scoring questionnaires? An EPEC presentation at the 2015 IOP
Many of us conduct research by collecting data to report on areas such as performance, satisfaction, safety, health and well-being. One of the easiest ways to collect this data, can be through questionnaires due to their versatility, efficiency and often, generalisability.
However, if we interpret the results incorrectly due to poor analyses, what is the point of investing time, money and expert opinion on developing interventions to change behaviours or procedures?
Some of the common issues in scoring data once collected, is that we assume that all the questions contribute equally to the construct we are measuring, or we assume there are equal intervals between the categories in the scale.
The fundamental mistakes here are obvious. We often have ordinal level data, and are making assumptions based on interval level data, and use Classical Test Theory to analyse the data. Thus, any conclusions we draw are most likely going to be based on incorrect interpretations of the data.
If, for example, you are using a questionnaire to assess something such as employee satisfaction, and the overall result according to Classical Test Theory indicated perhaps lack of professional development opportunities was the greatest concern, it may be worth considering whether the appropriate analyses were undertaken before investing money into new initiatives to establish PD activities in the organisation in an attempt to improve satisfaction. If intervals and contributions to the construct were taken into account, it may have indicated that an entirely different issue was of concern.
We thoroughly enjoyed presenting on this topic at the APS - 11th Industrial and Organisational Psychology Conference in Melbourne on 2 July, 2015. If you missed it, we will be happy to discuss some of our experiences with you; just contact us at admin@epecat.com.
