THE CONCURRENT VALIDITY OF THE TECHNICAL TEST BATTERY AS AN INDICATOR OF WORK PERFORMANCE IN A TELECOMMUNICATIONS COMPANY

The purpose of this study was to assess the concurrent validity of the Technical Test Battery (TTB) in a South African telecommunications institution. The Technical Test Battery (TTB) was administered to a sample of 107 technical officers. Their test scores were compared to the scores obtained from a job performance rating scale specifically designed for this position on the basis of a thorough job analysis. The TTB demonstrated high concurrent validity as an indicator of work performance for technical posts in the telecommunications environment. These results suggest that the TTB may have a high predictive validity for performance in technical positions. The findings and implications of the study are discussed.

The fact that reliability, validity, bias and fairness are highlighted in Section 8 of the Employment Equity Act and the need for being able to show "scientifically" that these criteria have been met pose specific dilemmas in all the contexts in which assessment is used (Lopes, Roodt & Mauer, 2001).The cost associated with selection errors also warrants studies of this nature: selection errors are not only costly to employers, but also affect individual employees and South Africa's economy as a whole.
The business environment in which organisations have to operate has become increasingly complex.Organisations are confronted with ever-increasing uncertainty, turbulence and change in their environment (La Grange & Roodt, 2001).The telecommunications market in particular has had to face large changes and challenges during the past ten years, and will continue to do so.Twenty years ago, stand-alone services dominated this environment.Conventional voice telephony accounted for the overwhelming majority of all communication traffic.However, these services have now developed into telecommunications markets in a highly competitive global environment.This requires them to focus on mass-market business products to which leading edge expertise must be applied (Szlichcinski, 2001).
A prominent element in maintaining leading edge expertise is the selection and development of people who are competent to perform particular jobs.High selection validity translates into considerable financial savings for most organisations.Hunter and Schmidt (1982) conducted a study in the Pennsylvania Police Department to determine the importance of ability testing at entry-level jobs.They estimated that using tests of cognitive ability as opposed to using no tests prevents a loss of a minimum of eighty billion dollars per year.Even replacing cognitive ability tests with any instrument of lower validity would incur very high costs (Hunter & Hunter, 1984).In a study conducted in South Africa, Muchinsky, Kriek and Schreuder (2002) reported savings of R1,7 million in the first year for 80 insurance brokers who used a selection battery with a validity of r = 0,43.
A crucial element in maintaining productivity in organisations is the selection of people who have the ability they require to do the jobs they are hired to do (La Grange & Roodt, 2001).According to Hunter and Hunter (1984), for most jobs, two predictive instruments with high validity are cognitive and psychomotor ability tests.Various studies conducted by Hunter and Schmidt (1982) in governmental and in private organisations suggest that ability tests are good predictors of performance in the job and work success.Studies have also shown that the ability tests that were reviewed were valid in predicting job proficiency across all jobs (Hunter, 1980(Hunter, , 1981;;Schmidt, Hunter & Pearlman, 1981).In a more recent metaanalysis, Schmidt and Hunter (1998) concluded that cognitive ability is one of the best predictors of job performance and should form part of all personnel selection batteries.They found a mean validity coefficient of r = 0.51 when cognitive ability was correlated with work success across various job families.Hunter and Hunter (1984) emphasise that ability is probably the best predictor for entry-level hiring.Furthermore, they state that for some jobs, specific ability assessment has high validity, especially in positions where certain technical abilities are necessary to perform the job.Thus, it is vital that organisations conduct cognitive ability assessments in order to maintain their competitive edge and to continue to provide quality services.
It terms of the requirements of the Employment Equity Act (Republic of South Africa, 1998), it is important that employers should provide proof of the validity of the cognitive assessment instruments they use for specific contexts or jobs.Test users and employers can investigate the validity of a cognitive assessment by means of concurrent and predictive validity studies, depending on the situation and the availability of data.Cascio (1998) distinguishes clearly between concurrent and predictive studies.He uses time as a basis for this distinction.He states that predictive studies are oriented toward the future and involve a time interval during which events occur before the criterion data are obtained.By contrast, concurrent studies are oriented toward the current situation and there is no significant time lapse between events and the time when the criteria data are obtained.In the employment context, the difference can be illustrated by asking two questions.In a concurrent design, the question is whether the employee can do the job now, whereas in a predictive design, the question is whether the employee will be able to do the job in future (Cascio, 1998).
According to Cascio (1998), concurrent studies are oriented toward the present and reflect only the status quo.However, both designs involve a similar paradigm in which a relationship is established between the predictor and the criterion performance.Cascio (1998) adds that, with cognitive ability tests, concurrent studies are often used as substitutes for predictive studies.Owen and Taljaard (1996) point out that their review of empirical comparisons of the validity estimates of cognitive ability tests (both predictive and concurrent designs) indicated that, at least for these measures, the two types of designs do not yield significantly different estimates.They state that the passage of time between the measurement of the predictor and the measurement of the criterion does not significantly influence the validity coefficient.By contrast, Guion and Cranny (1982) caution that, on both conceptual and practical grounds, these different validity designs should not be considered equivalent or interchangeable across all situations.They state that without explicit consideration of the influence of uncontrolled variables in a given situation, one cannot simply substitute a concurrent design for a predictive one.In the light of this limitation, Cascio (1998) emphasises the importance of conducting cross-validation studies within the predictive context before predictive validity can finally be assumed.
It can be concluded that concurrent validity studies do provide valuable information on the relationship between predictor and criterion variables, but that concurrent validity studies should not be used as a substitute for determining predictive validity.In addition, concurrent validity studies provide valuable information on the relationship between test performance and current work performance, thereby contributing to the diagnostic value of a test.

Aim of study
The aim of this study was to determine the concurrent validity of the Technical Test Battery (TTB) as an indicator of work performance in a large South African telecommunications company.

METHOD Participants
This research, using a concurrent validity design, was conducted in a large telecommunications company in South Africa.The population consisted of 140 technical officers who were geographically distributed throughout South Africa.The following five regions were represented: Bloemfontein, Western Cape, Durban, Port Elizabeth and Gauteng.The sample consisted of 107 technical officers who were available for testing.The sample was divided more or less equally in terms of the five regions, but there were fewer officers from the Bloemfontein group.The regional composition of the sample is set out in Table 1.

Measuring instruments
Measures of criterion performance are necessary in almost all personnel application fields.Criteria are needed to assess the effectiveness of personnel selection procedures, organisational training programmes, job design efforts and many other personnel-related actions and interventions (Cascio, 1998).
In the next section, the use of criterion measures, as well as predictor variables, is discussed to indicate their relevance in this study.

The criterion measure
Saville & Holdsworth (Pty) Ltd (SHL) developed a criterion questionnaire for the position of technical officer in a telecommunications environment.The questionnaire contained questions to elicit background information (the personal information of the person completing the questionnaire), instructions (the aim of the study and instructions on how to rate each employee) and the 37 items on which the employee should be rated.The questionnaire was based on information from a job analysis conducted using the Work Profiling System (WPS).The WPS is an integrated job analysis system in which information is collected about a job in a structured way, using questionnaires.
The information is then analysed by means of a computer to meet a number of information objectives, including a profile of the required job tasks and a profile of the human attributes required to complete the specified tasks.Respondents, typically job holders, are required to identify eight to ten key categories of job tasks that are important for achieving their objectives from a selection presented on cards.For each category chosen, a set of specific task statements is then rated on two scales to define precisely the amount of time devoted to the activity, and its importance to achieving the job objectives.Further information is collected by means of specific questions relating to the job environment, including questions about responsibilities and accountabilities.The information obtained is analysed via computer to reveal the critical job elements required to achieve the objectives, as well as the human attributes required for effective job performance (Saville & Holdsworth (Pty) Ltd, 2001).
The criterion questionnaire contained the following dimensions: managing tasks (activities such as planning, implementing, co-ordinating and evaluating); managing people (activities such as supervising or directing, motivating others, disciplining and liaising); receiving information (activities such as collecting information by active listening, gathering information from devices and using instruments, as well as using one's senses); working with information (activities such as assessing and evaluating, analysing and interpreting, as well as calculating); communicating (activities such as influencing and advising, presenting, briefing, representing and selling); and physical activities (physical activities such as responding physically by movements with the hands or feet, performing physical tasks by climbing and working with equipment and machinery) .
The reliability coefficient for the criterion questionnaire was calculated using Cronbach's Coefficient Alpha.The reliability of the criterion ranged from r = 0,80 to r = 0,96.The criterion questionnaires were given to the human resources consultants involved in this study in all five regions.These human resources consultants then distributed the questionnaires to the line managers who supervised the employees in the sample.The supervisors were given clear instructions, provided by SHL, about the completion of the questionnaires, as well as background information on the reasons for and significance of the study.Supervisors were required to rate their employees using a number of statements relating to the above criteria.They rated the employees on a five-point scale, ranging from 1 (Unsatisfactory performance) to 5 (Outstanding performance).
The supervisors had to complete the rating of the employees within a month after the questionnaires had been handed to them.The completed questionnaires were then collected by the human resources consultants and sent back to SHL for processing.

The predictor variable
The The internal consistency reliability coefficients for all the TTB tests were calculated using Cronbach's Coefficient Alpha.The reliability coefficients of the tests ranged from r = 0,78 to r = 0,92 (Saville & Holdsworth (Pty) Ltd, 1991).
Various validation studies have been conducted by Saville & Holdsworth (Pty) Ltd (1991) to obtain the inter-correlations between the sub-tests of the technical test battery.One particular study was conducted on a sample of engineering apprentices (N = 89).The results have shown high correlations between all the sub-tests.
Another study was conducted in the telecommunications environment was conducted to assess the effectiveness of the TTB in the selection for the position of future Demand Forecasters.Demand Forecasters are employees who investigate the future need for telecommunications in various organisations.Consistently high correlation coefficients were found between the predictors.The inter-correlations ranged from r = 0,38 to r = 0,54.The correlations between the predictors and the criteria ranged from r = 0,23 to r = 0,42 (Saville & Holdsworth (Pty) Ltd, 2000).

Procedure
The TTB was administered to job incumbents in the position of technical officer at a large telecommunications organisation.
There was no specific prescribed order for the battery administration.A qualified psychometrist employed by the telecommunications organisation administered the assessment battery as per standardised method suggested by SHL.This process entailed starting with the administration of the first verbal (VT1.1)sub-test, thereafter administering the first numerical sub-test (NT6.1),followed by the visual estimation sub-test (ET3.1),followed by the second verbal sub-test (VT5.1) and concluding with the fault-finding sub-test (FT7.1).
According to Saville & Holdsworth (Pty) Ltd ( 2002), the best practice is to assess a maximum of 20 candidates per assessment session.Therefore approximately 15 to 20 candidates were assessed per session.The instructions for the subtests were read from the instruction cards per sub-test in a standard fashion, emphasising the strictness of the test conditions and what the candidates should expect from the tests.There were also example questions which were presented before each sub-test for candidates to practise on to ensure that they had understood the instructions for the test.This standardised instructional routine was followed for the entire test battery and for every group that was assessed (Saville & Holdsworth (Pty) Ltd, 1991).
All the raw data from the tests were collected and organised into a workable format.To enable comparisons to be drawn, biographical details such as age, educational level, race and language were also recorded.
Raw data from the five subtests were available for a final sample of 107 subjects.

Data analysis
The statistical techniques used included descriptive statistics, where the means, standard deviations, skewness and kurtosis coefficients were calculated.Product-moment correlation coefficients were calculated to provide a detailed analysis of the relationship between the various predictors and the criterion scores.A canonical correlation was calculated to analyse the simultaneous relationship between the multiple independent and dependent variables that formed part of the study.Multiple regression analysis was used to determine the significance of the relationship between the multiple independent variables and the composite dependent variable which was used.
The main difference between multiple regression and canonical correlation is that in multiple regression there is only one combination of variables, because there is only a single variable to predict on the other side of the equation.In canonical correlation, however, there may be several variables on both sides and there may be several ways to recombine the variables on both sides to relate them to each other (Tabachnick & Fidell, 1989).

RESULTS
The descriptive statistics on the variables that were included in the research are set out in Table 3.All the variables comply reasonably well with the basic requirements for conducting parametric statistical analyses.The zero-order correlation between the criteria and the predictors is set out in Table 4. Overall, the correlations between the tests and the criterion dimensions were moderate to high, and positive.The zero-order correlations on the test total ranged from r = 0,17 to r = 0,48.High statistically significant correlations were reported on "Receiving Information" and the predictors.The overall moderate to high positive correlations between the criterion dimensions and the test dimensions indicate that the test battery will be valuable in the prediction of work performance and can be used in a diagnostic context.

TABLE 4 CORRELATION OF THE
Principal component analysis was performed to determine the underlying structure of the test dimensions and the criterion dimensions with regard to the research sample.
Table 5 sets out the results of the principal component analysis of the TTB.The sample size was adequate, according to the Kaiser-Meyer-Olkin measure of sampling adequacy (0,82).One component was greater than unity, indicating a uni-dimensional structure that accounts for 66,458 % of the total variance.The component matrix provides the loadings of each test dimension on the uni-dimensional component solution.The results of the principal component analysis suggest that a general factor underlies the TTB, namely general technical ability (GTA).One component was greater than unity, indicating a unidimensional structure that accounts for 80,105 % of the total variance.The component matrix provides the loadings of each criterion dimension on the one-component solution.The results of the principal component analysis suggest that there is a general factor that underlies the criterion measures.Thus, the supervisor's ratings of behavioural performance appeared to be supported by a general underlying performance component that explained most of the variance in respect of these scores.Theoretically, this is in line with what can be expected when behavioural performance is assessed in the workplace (Cascio, 1998).However, the strong interrelationships that were obtained between the criterion dimension scores may signify bias in the ratings in terms of the "halo" effect (Cascio, 1998).
From the above, it is clear that there is a strong relationship between all the independent and the dependent variables.The multiple regression technique was used to investigate the relationship between the independent and dependent variables.
In a multiple regression analysis, several independent variables (instead of just one) are combined to predict a value on a dependent variable for each subject (Tabachnick & Fidell, 1989.For the purposes of this study, this technique was used to analyse the relationship between the TTB and the composite criteria. Table 8 shows that the multiple correlation between the predictor (the independent variable) and the criteria (the dependent variables) was r = 0,417.This correlation was statistically significant (F = 19,382; p £ 0,000) and explained 17,4% of the variance of the total performance score.
Table 8 suggests that Verbal Comprehension is the only unique predictor of the criterion variable and that none of the other predictors were statistically significant in terms of the uniqueness of the contribution they made toward predicting the criterion variable.These results confirmed the high interrelatedness and uni-dimensionality of the predictor variables, as depicted in the results of the factor analysis and canonical correlation analysis.Thus, there was evidence of a strong unidimensional relationship between the predictors and the criterion variables.

DISCUSSION
The aim of this study was to determine the concurrent validity of the TTB in a telecommunications environment.Although a rather small sample of 107 respondents was used, statistically significant results were reported that point toward concurrent validity.
The results of this study indicate that the TTB would be valuable in predicting work performance.However, this finding needs to be confirmed with a separate study on the predictive validity of the TTB.
The factor analysis and canonical correlation analysis results indicated that both the independent and the dependent variables can be considered one-dimensional in terms of what is measured.
The first variant from the canonical correlation analysis results loaded high on both the independent and the dependent variables, pointing toward a strong relationship between the one-dimensional independent and the dependent variables.Thus one can conclude that general technical ability, as measured by the TTB and a behaviourally based general work performance measure for a technical environment, show strong positive interrelationships.The multiple regression analysis also indicated that there is a significant relationship between the predictors and the composite criteria.
The above results indicate that the TTB is a good indicator of work performance in the telecommunications environment and could be used for diagnostic and development purposes and probably for selection purposes.The high correlations obtained between the TTB and the selected work performance criteria are consistent with previous findings such as those of Saville & Holdsworth (Pty) Ltd (2000).
It is of the utmost importance that individual potential and human capital should be the main focus of South African organisations if these organisations want to develop and prosper economically.Especially when one takes into account new legislation, it is evident that when high potential individuals are identified, then any time, effort and resources expended on them will have a much higher return on investment than when organisations appoint employees in a haphazard manner (Muchinsky et al, 2002;Hunter & Schmidt, 1982;Lopes et al, 2001).

Limitations
One of the limitations of the study is the extent to which work performance ratings by supervisors are biased.The results point toward the possibility that the ratings in this study were biased in terms of the halo-effect.This would have a negative impact on the validity of the criterion measure and would inhibit the strength of the relationship between the TTB and the criterion measure.
The research is limited to the position of a technical officer and the results may not apply to other technical positions.Although the findings could, to a certain extent, be generalised to similar jobs that have more or less the same content, more validity evidence is required before the TTB can be used as a diagnostic, development and selection tool for other technical jobs in the telecommunications environment.

Recommendations
It is recommended that further research be conducted to determine the concurrent and predictive validity of the TTB for technical positions in more than one context.An important factor in criterion-related validity research is not only the validity of the measure in question, but also the validity of the criterion measure.The validity of the criterion measure could be promoted considerably if the people who are responsible for conducting performance ratings are sensitised to the effect of bias on ratings.According to Cascio (1998), the stability and validity of a criterion measure can be enhanced if more than one rating is conducted per person.This could be achieved by conducting multiple ratings over time and/or by using data from multiple sources (e.g.supervisors, peers, subordinates).

Table 2
than Grade 12.Of the group, 20% had obtained a qualification higher than a Grade 12.The average job experience of the respondents was eight years.
TTB is one of a range of ability tests developed by SHL specifically for use in assessment in a technical environment.It consists of a series of sub-tests particularly suitable for the selection and development of individuals in technically or practically oriented jobs.The emphasis in the TTB is the assessment of relevant work-related skills.The TTB tests have been designed to measure the core abilities required in a wide range of technical occupations.Job analysis of a variety of such posts has shown that verbal, numerical and visual skills are often key attributes associated with successful job performance(Saville & Holdsworth (Pty) Ltd, 1991).The VT1.1 measures vocabulary and basic word skills using language that reflects the requirements of technical occupations.It is appropriate for use with apprentices in engineering, building and other industries.The question types include sentence completion, word meanings and likes and opposites.The language used in the test has been chosen to reflect that which respondents may encounter in a technical working environment.The test consists of 36 items, which are presented in a multiple-choice format with five alternative answers.The time limit for completing the test is 12 minutes(Saville & Holdsworth (Pty) Ltd, 1991).
The FT7.1 assesses the ability to identify faults in logical systems.No specialised knowledge is required.The candidate is given a number of sequences representing various processes and must locate which particular process in each sequence is not working as specified.The test consists of 36 items presented in a multiple-choice format with five alternative answers.The test has a time limit of 18 minutes(Saville & Holdsworth (Pty)  Ltd, 1991).

Table 6
sets out the results of the component analysis of the criterion measure.The sample size was adequate, according to the Kaiser-Meyer-Olkin measure of sampling adequacy (0,90).