Investigating the construct validity of an electronic in-basket exercise using bias-corrected bootstrapping and Monte Carlo re-sampling techniques

Abstract

Orientation: Technology-based simulation exercises are popular assessment measures for the selection and development of human resources.

Research purpose: The primary goal of this study was to investigate the construct validity of an electronic in-basket exercise using computer-based simulation technology. The secondary goal of the study was to investigate how re-sampling techniques can be used to recover model parameters using small samples.

Motivation for the study: Although computer-based simulations are becoming more popular in the applied context, relatively little is known about the construct validity of these measures.

Research approach/design and method: A quantitative ex post facto correlational design was used in the current study with a convenience sample (N = 89). The internal structure of the simulation exercise was assessed using a confirmatory factor analytical approach. In addition, bias-corrected bootstrapping and Monte Carlo simulation strategies were used to assess the confidence intervals around model parameters.

Main findings: Support was not found for the entire model, but only for one of the dimensions, namely, the Interaction dimension. Multicollinearity was found between most of the dimensions that were problematic for factor analyses.

Practical/managerial implications: This study holds important implications for assessment practitioners who hope to develop unproctored simulation exercises.

Contribution/value-add: This study aims to contribute to the existing debate regarding the validity and utility of assessment centres (ACs), as well as to the literature concerning the use of technology-driven ACs. In addition, the study aims to make a methodological contribution by demonstrating how re-sampling techniques can be used in small AC samples.