A PSYCHOMETRIC APPROACH TO SUPERVISORY COMPETENCY ASSESSMENT

The primary purpose of this study was to evaluate the possibility of using a psychometric approach for assessing supervisory competencies relevant to the mining and refining environment. The competency questionnaire was developed using supervisory roles and registered supervisory unit standards from the United Kingdom (UK), as no registered unit standards exist in South Africa. Twenty-four supervisors from three departments (Production, Engineering and Laboratory) were evaluated by 125 raters; besides by themselves, also by their managers, peers, customers and their sub-ordinates. Based on difference scores derived from the Importance and Performance scales, a single factor was extracted with an internal reliability of 0,965. No statistical significant differences were obtained (ANOVA’s, t-test and F-statistics) between groups based on biographical variables or between rater groups. The findings and their implications are further discussed. OPSOMMING Die primêre doel van die studie was om die moontlikheid vir die gebruik van ’n psigometriese benadering tot toesighouerbevoegdheidsbeoordeling, te evalueer. Die bevoegdheidsvraelys is ontwikkel deur gebruik te maak van toesighouersrolle en geregistreerde toesighouerseenheidstandaarde van die Verenigde Koningkryk, as gevolg van ‘n gebrek aan bestaande eenheidstandaarde in Suid-Afrika. Vier-en-twintig toesighouers van drie departemente (Produksie, Ingenieurswese en Laboratorium) is deur 125 beoordelaars geëvalueer; buiten deur hulself, ook deur hul bestuurders, kollegas, kliënte en hul ondergeskiktes. ’n Enkele faktor, met ’n betroubaarheid van 0,965, gebaseer op die verskiltellings van die Prestasieen Belangrikheidskaal, is onttrek. Geen beduidende verskille (ANOVA’s, t-toetse en F-statistiek) kon tussen groepe gebaseer op biografiese veranderlikes en die onderskeie beoordelaarsgroepe gevind word nie. Hierdie bevindinge en die implikasies daarvan word verder bespreek. A PSYCHOMETRIC APPROACH TO SUPERVISORY COMPETENCY ASSESSMENT Requests for copies should be addressed to: A Vorster, Department of Human Resource Management, RAU University, PO Box 524, Auckland Park, 2006 38 SA Journal of Industrial Psychology, 2003, 29 (3), 38-44 SA Tydskrif vir Bedryfsielkunde, 2003, 29 (3), 38-44

From the summary of objectives in Table 1, it is clear that developing people, as well as giving credit for development, is to be promoted. This brings to mind a question about the relevancy and meaning of competency-based processes in managing human resources. According to Weightman (1994, p. 11) "using competencies as the basis for National qualification, and then equivalences across borders, alleviates the difficulties of differences between job titles and cultures, as competencies concentrate on the ability to do the job". It is therefore necessary that any competency-based assessment approach for training and development purposes in particular, will have to meet the objectives as described in the said legislation. The world of vocational education and training uses two distinct categories of competency models, namely 1) Competency models that are inputs based, and 2) Competency models that are outcomes based (Buckley & Caple, 1995;Mansfield, 1989). The first category constitutes descriptions of individual knowledge and skills. It is based on the assumption that competencies are individual attributes. The second category is based on descriptions of work role outcomes.
For the purpose of this study, competencies are defined as "skills or clusters of skills performed in a particular context or environment to specific standards" (Bellis, 1999, p. 19). This definition supports the notion of competencies being outcomes based, thus establishing a strong relationship with national standards, which are also outcomes based (Hallendorff, Richardson & Wood, 2000). It is therefore proposed that competencies as standards, and national unit standards should be seen as being the same. The underlying principle of this competency approach offers the organisation access to a framework (i.e. National Qualifications Framework) of assessing and developing staff, which is formal, structured, reliable and recognised at national and industry level. The individual benefits from this competency approach through its transferable accreditation of competencies acquired.
To be useful in any applied assessment context, competency standards have to be defined, described and assessed in terms of the behaviour they consist of. This will constitute the content validity of the assessment. It should thus be done in such a way that it accommodates most or all of the uses of competency standards, as mentioned before. Buckley and Caple (1995) and Mitchell (1989) identified the components of a competency standard as performance criteria, range statements, related knowledge and assessment requirements, which are similar to the key components thereof as described by Hallendorff et al (2000). These key components are: competency standard title (an outcome large enough to be recognised and credited), specific outcomes (which are smaller outcomes of the unit standard title), assessment criteria (which explains how well the specific outcomes should be performed) and range statements (used to describe the context or environment). These components underpin an approach of providing evidence as proof of competency during the standard-based assessment process.
In the vocational training process, specified learning outcomes that cover the aims of the curriculum, could be taught and assessed in unambiguous ways, which would overcome the problems of reliability, content validity and fairness in assessments. The objective therefore, is to base learning on outcomes-based competency standards (Gonczi, 1999).
It would therefore be beneficial for organisations to apply the outcomes-based assessment and development of, specifically, key employees such as supervisors, cost-effectively. The roles of supervisors will therefore be discussed in more detail below.
On the role of supervision Managers, as supervisors do, have many roles to play in the organisation. These roles are: Figurehead (performs ceremonial and symbolic duties such as greeting visitors or signing legal documents); Leader (direct and motivate subordinates, communicating with subordinates, training and counseling them); Liaison (maintains information links both inside and outside organisation); Monitor (seeks and receives information, scans reports and maintains personal contacts); Disseminator (forward information to other organisation members, e.g. by sending memos and reports); Spokesperson (transmits information to outsiders through speeches, reports and memos); Entrepreneur (initiates improvement projects and identifies new ideas); Disturbance Handler (takes corrective action during disputes or crises and resolves conflicts among subordinates); Resource Allocator (decides who gets resources through scheduling, budgeting and setting priorities); Negotiator (represents department's interests in meetings and negotiations).
The functions that managers fulfil, are contained in their roles. This theory brings a different dimension to analysing management through focusing on the role instead of the function. The first line supervisor performs all the managerial roles of top management, but it is done within the context of his or her own section (Mintzberg, 1973).
Organisational co-ordinating mechanisms in this supervisory context are: "the glue holding organisational structure together" (Mintzberg, 1979, pp. 3-7). These mechanisms involve: i) mutual adjustment, ii) direct supervision, (own emphasis), iii)standardisation of work processes, iv) standardisation of skills, and v) standardisation of outputs/results.
As organisational work becomes more complex, co-ordination seems to shift from mutual adjustment to direct supervision, to standardisation and can finally revert back to mutual adjustment. He postulated that: "direct supervision achieves coordination by having one individual taking responsibility for the work of others" (Mintzberg, 1979, pp. 3-4). This is evident with large groups that are less able to co-ordinate informally, thus with further complexity, supervision becomes necessary. Of these five elements, direct supervision and standardisation of skills support the main focus of the current study.
The necessity of supervision in the modern organisation can be attributed to new technology and a change in worker values (Hodgetts, 1987). Direct supervision, according to Mintzberg (1979), included planning, scheduling, allocating, instructing and monitoring actions, and closely resembles what Hilgert and Leonard (1998,) (planning, organizing, staffing, leading and controlling), as well as Hodgetts (1987) (planning, organising, directing and controlling) described as supervisory functions. These supervisory functions could be grouped in the same groupings as Mintzberg (1973) has done with managerial activities and roles as illustrated in Table 2. The supervisory competencies, according to UK standards, are also listed in the last column to indicate the overlap with supervisory functions and roles. Companies are often encountering difficulties in the area of supervision. These difficulties refer to problems with communications, controlling internal mobility, managing change, managing subordinates, performance management and measurement, and managing discipline and grievances (Calhoon, 1971). These difficulties can be related to the roles of supervisors as was described above. It is argued that the problems as mentioned, can be addressed through ensuring the competence (standardisation of skills) of supervisors in the roles that they have to play. Multi-rater performance evaluations play a valuable role in this regard.
On 360° competency assessments As mentioned before, there are different approaches towards assessing competencies. Standards-based competency assessment can be a long and tedious process due to its nature of seeking proof or evidence of competence. It is highly likely that this process will be biased when performed by a single individual. Different sources of bias in performance ratings were identified by Roodt (1999, 2000) who listed different working relationships; ego-centric bias; differences in organisational level; rater-leniency; the halo effect; purpose of ratings; and differences in rating criteria as examples. Theron and Roodt (2000) also added different mental models of raters to this list, based on their own research. Mental models in this instance refer to "unitary, spatial models where distance has functional consequence, and which are used to give meaning and understanding to complex systems or phenomena" (Theron & Roodt, 2000, p. 15). Quantitative evidence of the validity and reliability of such single-sourced, standards-based competency assessments are therefore difficult to determine. Following a multi-rater approach on the other hand, such as 360° competency assessments, may address most of these concerns by generating quantitative, psychometric information on reliability and validity coefficients. In 360° competency assessments, according to Jones and Bearly (1996), an individual receives feedback from a multi-rater assessment process, providing a more objective, holistic view of the individual. Overall, 360° assessments are more accurate, credible, fair and motivational than single-source assessments (Edwards, 1998) when applied effectively.
The success of 360° assessments (multi-rater assessment) is largely dependent on effective integration with human resource systems and business strategy, valid content, accurate responses, clear and specific results, integration with career development and planning support, and measurable performance improvement (Morical, 1999). In order to achieve this, specific attention should be given to design, implementation and evaluation factors (Theron & Roodt, 2001). More specifically, users of the system should be allowed to participate in the development of the system, to ensure that the system will be perceived as fair. Users of the system should also be trained in the proper use of the system (Edwards & Ewen, 1996), and the requirements of the Employment Equity Act (No 55 of 1998) that requires all psychological tests and other similar assessments to be valid and reliable, fair as well as not biased against any employee or any group of employees, should be adhered to. This necessitates that any 360° assessment rating system should be evaluated and validated.
In this study, the 360° competency assessment approach is seen as supplementary to the formal standards-based competency approach, as it is based on registered UK standards. The statistical reliability and validity of the 360° competency assessment process can be defended and could possibly be included in the standards-based assessment process.
In view of the discussion in the section above, the research objectives are stated as follows:

Research Objectives
The primary objective of the study was to determine whether a 360° degree competency assessment questionnaire could reliably and validly assess supervisory competencies.
The secondary objectives of the study were: To determine if there are any statistically significant differences in the mean difference scores between the Importance and Performance scales, with regard to different work departments, educational qualification levels and language groups. This comparison would assess whether different comparison groups have different mental models regarding supervisory competencies.
To determine if there are any statistically significant differences in the mean difference scores between the Importance and Performance scales, with regard to different rater groups (manager, self, customer, sub-ordinate and peer/colleague). This comparison would assess possible differences in mental models on supervisory competencies between different rater groups.

METHOD The research participants
The study was conducted in the mining and refining industry during 2002. A group of twenty-four first line supervisors from three departments (Production, Laboratory and Engineering) was assessed using the 360° Supervisory Competency Inventory (360° SCI). The total population of participants (N = 125) in this study were from 5 different rater groups: managers, self, customers, peers/colleagues, and subordinates.
Participating employees were briefed about how to complete the rating form, the aim of the assessment and the purpose of 360° assessment. Table 3 provides a brief overview of the biographical background of the rater sample in this study. The level of education of the participants ranges from standard 8 to tertiary qualifications. Four language groupings were included, namely Afrikaans, English, Zulu and Other. It should be noted that 129 respondents (more than 125) are listed under language groups, which can be attributed to multi-lingual individuals who listed more than one home language.
The measuring instrument Accepted United Kingdom standards for management and supervision were analysed and compared with in-house supervisory roles. The obtained results were further validated by subject matter experts, such as line management and individuals drafting standards for the Standards Generating Body for management and supervision. The nine supervisory competency dimensions (refer to "The person I am assessing: -Motivates team members to do a good job.
-Seeks to clarify conflicting information." Participants responded to the items of the 360° SCI on two scales, namely a six-point scale in respect of actual Performance and a six-point scale in respect of Importance (in terms of overall performance), where a 1 on the response scale was 'strongly disagree' (meaning poor performance), and a 6 'strongly agree' (meaning good performance) or 'low importance' and 'high importance' respectively.

The research procedure
The Business Manager of the Business Unit provided a letter of support to the project to explain the objective of the project and to urge participants to be honest and co-operative. This letter was attached to each form (the assessment was paperbased) and distributed to the participating employees with instructions on how to complete the 360° SCI and whom they were to assess. A deadline date for submitting the completed questionnaire was also communicated.
Senior staff members from each section or department were identified as contact people for the study, overseeing the distribution, completion, collection and control of the 360° SCI forms. This ensured the integrity of the assessment tool and the data collected for the study.

First level factor analysis on the item inter-correlation matrix
The initial 34 items were inter-correlated. Owing to a lack of space, the inter-correlation matrix cannot be reproduced here. A Kaiser-Meyer-Olkin measure of sampling adequacy (MSA) of 0,906 was obtained which indicated that the unreduced matrix was suitable for a first level factor analysis. Item 23 was omitted from the inter-correlation at this point. Eigenvalues were calculated on the inter-correlation of the remaining 33 items and six factors were postulated according to Kaiser's (1961) criterion (eigenvalues larger than unity). These six factors explained 71,738% of the variance in the factor space, as reflected in Table 4.
Six factors were extracted using principal axis factoring. Meaningful item loadings were obtained on all six postulated factors. Subsequently, the sub-scores were calculated for these six factors. Second level factor analysis on the sub-score intercorrelation matrix The six sub-scores (obtained from the first level factor analysis) were inter-correlated and the results of the inter-correlation of the sub-scores are displayed in Table 5. A Kaiser-Meyer-Olkin measure of sampling adequacy of 0,904 was obtained on this inter-correlation matrix, which indicated that this matrix was also suitable for the second level factor analysis.  Table 6 provides the eigenvalues of the unreduced intercorrelation matrix of sub-scores. A single factor was postulated based on Kaiser's (1961) criterion. This factor explained 71,606% of the variance of the factor space. The factor solution converged after six iterations. As only one factor was extracted, no further rotation took place. Table 7 provides the factor matrix of the 360° SCI. Iterative item analyses were performed in respect of the single factor for both the Performance and the Importance measures of the 360° SCI and high Cronbach alpha coefficients of 0,968 and 0,964 respectively were obtained. The results will not be presented separately for each scale, owing to limited space. Difference scores were used to present the scale properties. A high Cronbach alpha coefficient of 0,965 was obtained in respect of the difference scores of these two scales. Table 8 provides the item statistics of the difference scores of the two scales in respect of the 360° SCI.
It is clear from Table 8 that the corrected item-total reliabilities vary between 0,330 and 0,815 with only five items having values below 0,50 that contributed to the high internal consistency of the scale. The above results suggest that supervisory competencies could be reliably (high Cronbach alpha coefficient of 0,965) and validly [based on factorial validity -a component of construct validity (Allen & Yen, 1979)] measured which support the primary objective of this study.
The first secondary objective of the study was to determine whether there were any statistically significant differences in the mean difference of competency scores of groups created in terms of the different biographical variables. ANOVA'S (more than two groups), t-tests (two groups only) and F-statistics were used for these analyses. As a large majority of the respondents were male and very few females participated in the study, gender was not included in this analysis. No statistically significant differences were found in respect of the mean scores of different departments, qualification levels or home languages. Tables 9, 10 and 11 reflect the results in respect of the mentioned biographical variables. As the obtained value of F = 0,427 does not exceed F (p=0,05; df=2, 122) = 3,07 it is concluded that there are no statistically significant differences in respect of the mean difference scores of the departments. As the obtained value of t = 0,061 does not exceed t (p=0,05; df=119) = 1,960 and the obtained value of F = 0,047 does not exceed F (p=0,05; df=2, 119) = 3,07 it is concluded that there are no statistically significant differences in respect of the mean difference scores of the educational qualification levels of the sample. As the obtained value of F = 1,988 does not exceed F (p=0,05; df=3, 123) = 2,68 it is concluded that there are no statistically significant differences in respect of the mean difference scores of the different language groups.
The last secondary objective of the study was to determine if there were any significant differences between the mean difference scores of the different rater groups. ANOVA's and F-statistics yielded the following results in Table 12: As the obtained value of F = 1,311 does not exceed F (p=0,05; df=4, 120) = 2,45 it is concluded that there are no statistically significant differences in respect of the mean difference scores of the rater groups. This finding supports the last of the secondary objectives.

DISCUSSION
The primary purpose of this study was to evaluate the possibility of using a psychometric approach for assessing supervisory competencies relevant to the mining and refining environment. The results obtained from the factor analyses and the item analyses indicate that the construction of the 360° SCI was based on sound psychometric principles. The factor analysis resulted in a single factor that indicates a solid theoretical base and sound procedure in the construction of the 360° SCI. A high level of reliability and a limited amount of error variance in the measurement of the construct of "supervisory competency" was obtained, indicated by the high internal consistency of 0,968 and 0,964 for the Performance and Importance scales respectively. The difference scores between these two scales yielded an Alpha coefficient of 0,965.
These results indicate that supervisory competencies can be assessed in a reliable and consistent manner with the 360° SCI. The content validity (based on the content of unit standards) supports the face validity of the instrument that was based on the comparison of supervisory competencies and Mintzberg's managerial roles (1973) as indicated in Table 2.
It is clear that the 360° SCI can be used for supervisors in different functional contexts within in the mining and refining industry, as no significant differences were found between the mean difference scores for the three departments that participated in this study. The 360° SCI also seems to be fair and equitable, as no significant differences were found in the mean difference scores with regard to the qualification level or language groups. As only six females participated in the study, gender groups were not included in the analysis. It appears that the 360° SCI is not biased in terms of the biographical differences analysed, which supports the first of the secondary objectives of this study.
No significant differences were obtained on the ratings of the different rater groups (self, manager, peer, customer/supplier and sub-ordinate). This is often not the case for first time (or even second time) ratings as pointed out by Theron and Roodt (1999) and Theron and Roodt (2000). This implies that the different rater groups often apply different mental models for interpreting the questionnaire, resulting in significantly different ratings.
The implications of these findings are far-reaching for organisations, as well as at a national level. The 360° SCI provides a cost-efficient and time-efficient way of assessing the supervisory competencies of individuals in organisations. The time consuming process of declaring a person competent to supervise people through evidence-seeking, standards-based assessments, can be made more effective and efficient. The 360°a ssessment approach has added advantage in that it seems to be a valid and reliable way of measuring supervisory competencies.
One of the shortcomings of the study was that there were very few female participants in the study. Another shortcoming was that the rater sample was relatively small, even though sufficient sampling adequacy was achieved.
In future studies it might be useful to consider the following: Including more female participants in all the rater groups. The results may be very different when this level of diversity is introduced.
Considering the mental models that the different rater groups use as frames of reference, even though no significant differences were found between the rater groups in this study.
Researching the possibilities of using the instrument for developmental and promotional purposes or remuneration increases.
Investigating the possibility of measuring other competencies with a multi-rater assessment approach.
Testing the research against current legislation and practical or operational requirements. Using a larger sample of raters and ratees. Including other units within the organisation in the study to test its applicability, reliability and validity for the larger organisation and not just one unit. Correlating the ratings on the 360° SCI with ratings obtained from supervisory unit standards assessment.
The study was aimed at evaluating the possibility of using a psychometric approach for assessing supervisory competencies. The results reported in this article support the use of psychometric assessment (in the form of a 360° supervisory competency questionnaire) in this regard and could be considered as a sound alternative to single-sourced, standardsbased competency assessment.