An Alternative Scoring Approach for Best-Worst Scaling (BWS)

Abstract

The paper proposes an alternative scoring approach for the Best-Worst scaling (BWS) Object Case to capture both preference heterogeneity and experimental design differences to improve the prediction accuracy at the individual level. The unduplicated and highly unique scores across individuals also provide helpful input for further analysis, such as hybrid models to help understand people’s preferences in other tasks. Whilst the existing BWS scoring methods, including the most commonly used best-minus-worst and the best-over-worst ratio scores, have been applied primarily to elicit preference and ranking at both aggregate and individual levels, there are limitations such as equally scored items when we predict choices and order. We propose an alternative approach to target several limitations of existing methods. The proposed scoring approach can make several contributions: 1) it breaks equality in scores; 2) it introduces instruments to minimise design-induced effects such as different item co-occurrences in different balanced incomplete block designs (BIBD); and 3) it introduces a risk-averse instrument to lower the impact of incorrect predictions. We used seven empirical BWS Case I data sets with respondents completing full BIBD designs varied to test the new scoring against the current scoring. Results show a universal improvement in prediction accuracy. Compared to the present method, generating a limited set of discrete scores, the new approach generates almost unduplicated scores for object items across individuals with continuous distributions.