More than three decades ago, Cameron (1986, p. 539) posed the question: What makes an organisation ‘excellent, of high quality, productive, efficient, healthy, or possessing vitality?’ Individually and jointly, all of these aspects serve as proxies for the concept organisational effectiveness, and several contributors to effectiveness in organisations have over the years evolved via numerous inquiries (cf. Ashraf & Kadir, 2012; Cameron, 1978, 1986; Kwan & Walker, 2003; Quinn & Rohrbaugh, 1981; Roland, 2011; Shilbury & Moore, 2006). The term ‘organisation’ has for many years not been associated with the higher education (HE) sector, with institution the preferred label. Tucker and Bryan’s (1988, p. 4) view of academic management as ‘an art; not a science, because a college or a university is very unlike the standard corporation or business’ had and still has wide backing (Drew, 2006; Mintzberg 2004; Schmidtlein & Milton, 1989; Scott, Bell, Coates, & Grebennikov, 2010). However, the upswing towards managerialism and accountability has led many scholars to an opposing point of view, namely that a higher education institution (HEI) is and should increasingly be managed like a business (Davis, Jansen van Rensburg, & Venter, 2016; Deem & Brehony, 2005; Deem, Hillyard, & Reed, 2007; Kolsaker, 2008). These contrasting opinions affirm Cameron’s (1986, p. 540) argument that ‘As the metaphor describing an organization changes, so does the definition or appropriate model of organizational effectiveness’. The concept institutional effectiveness (IE) gradually became HE’s take on organisational effectiveness.

Institutional effectiveness typically asks (Leimer, 2011; Roland, 2011; Volkwein, 2007) the following question: how well is a higher education institution (HEI) fulfilling its purpose? Answers to three questions are typically sought when monitoring whether a HEI is indeed making strides towards accomplishing its purpose (Boehmer, 2006; Welsh & Metcalf, 2003), namely: (1) to what extent the institutional mission is being conquered, (2) whether progress is made in attaining the goals and objectives of the institution’s strategic plan and (3) whether the stated educational outcomes of an institution (at programme, departmental and school or faculty levels) are being achieved. Volkwein (2011, p. 5) highlights a pertinent development of the post-2010 era in HE, namely that: ‘accountability and accreditation have shifted institutions’ emphases to goal attainment, program evaluation, and institutional effectiveness – with especially heightened emphasis on student learning outcomes’ (authors’ emphasis).

The assessment of student learning outcomes seems to have become a key indicator of IE in the current HE context. The US Middle States Commission on Higher Education (2005, p. 3) aptly captures the essence of the IE challenge facing HE institutions via two questions, namely: ‘As an institutional community, how well are we collectively doing what we say we are doing?’ and, in particular, ‘How do we support student learning, a fundamental aspect of institutional effectiveness?’

There is no single best way of measuring the effectiveness of a HEI. This fundamental reality stems from the recognition that, like most social systems, HEIs have multiple stakeholders, who might hold different, sometimes competing conceptions of what constitutes effectiveness. A paradox exists in the role of students as stakeholders in education. While many argue that educational institutions exist primarily to serve the interests of students (Ashraf & Kadir, 2012; Bitzer, 2003; Ronco & Brown, 2000), evidence of the vulnerability of students in the power relationship with regard to other primary stakeholders within HEIs persists (Glasser & Powers, 2011; Seale, Gibson, Haynes, & Potter, 2015).

Although there has been a recent surge in measurement research relevant to student perspectives of IE, the resultant instruments are either too narrow in focus (often limited to assessing one or two educational outcomes) or remain untested in the South African context. Some useful outputs in this regard are found in Díaz and De León (2016) for measuring dropout intentions, an indicator of student retention (SR), Maddox and Nicholson (2014) on student satisfaction and Soldner, Smither, Parsons and Peek (2016) for measuring student persistence, a proxy for SR. To attain a fuller understanding of how students value the HEIs in which they study, an instrument that can capture multiple dimensions of IE and that is developed and tested in the local context, is required. Although this article cannot claim to fill this gap, it strives to make a contribution in that regard.

By upholding the primacy of student interests as the focus of IE as a point of departure, this study has, as primary purpose (or aim), to explore the reliability and construct validity of an instrument used to measure and report on students’ perceptions of IE at a HEI. In support of this aim, the article pursues the following objectives:

One way of framing IE is the strategic constituency approach (SCA), which originated as the participant satisfaction approach (Cameron, 1978; Quinn & Rohrbaugh, 1983). Rooted in stakeholder theory (Donaldson & Preston, 1995; Freeman, 1984), SCA raises the question about whose interests should be privileged in defining effectiveness criteria. At the core of this approach is the recognition that the various stakeholders of an institution might value different outcomes, and therefore, diverse views must be considered in defining relevant IE criteria (Balduck & Buelens, 2008). As Ashraf and Kadir (2012) appropriately argue, the stakeholder approach to effectiveness integrates the concept of social responsibility and might be readily attractive for application in HE environments where effectiveness often has to be adjudged through the eyes of multiple constituents, both internal and external to the institution.

In the last three decades, the idea of the student voice, which explores the importance of students’ perspectives in diverse aspects of educational provision, has gained prominence in HE research (Leckey & Neill, 2001; Levin, 1998; Subramanian, Anderson, Morgaine, & Thomson, 2013). Subramanian et al. (2013, p. 137) identify four ways in which students’ insights might improve the educational process, encapsulating ideas that have been articulated by several other authors. These include (1) providing formative feedback on pedagogical practices (Harvey, 2003; Seale, 2010), (2) being a key element of the quality assurance strategy (Estes, 2004; Leckey & Neill, 2001; Moufahim & Lim, 2015), (3) promoting engagement and reflection on teaching and learning activities (Cook-Sather, 2006) and (4) advancing democratic participation and empowerment (Cohn, 2012; Del Prato, 2013; Levin, 1998).

The student voice must, however, be considered with care. One major critique is that the student voice is in reality not a lone or unified voice but one that is constructed from a mosaic of diverse, sometimes conflicting voices (McLeod, 2011). Another critique is the question whether students are adequately equipped to know and articulate what is best for their own learning or related outcomes (Slade & McConville, 2006).

In the end, these critiques in the authors’ views do not necessarily devalue the importance of the student voice, but rather, urge that it is used more habitually, although responsibly.

Almost five decades ago, Kohlberg and Mayer (1972) elegantly captured the importance of clearly identifying the most salient goals of education thus:

These sentiments continue to be echoed by contemporary HE scholars (Etzel & Nagy, 2016; Judson & Taylor, 2014). Inspired by Volkwein’s (2011) vital observation regarding the primacy of student learning outcomes in defining IE, a search of the relevant literature for what might constitute such outcomes revealed four key themes. This review privileges a sociological perspective (Díaz & De León, 2016), focusing on the institutional determinants of each outcome as they are more likely to signify IE and therefore highlight critical areas for intervention (Hilton & Gray, 2017), as opposed to other confounding or mediating factors. Student educational outcomes (SEOs) are used in this study to represent a set of four distinct but related outcomes, which have been reported by other scholars to various degrees. For instance, Etzel and Nagy’s (2016) notion of academic success encompasses three components namely, academic performance, academic satisfaction and major change intention (a proxy for SR), while DeShields, Kara and Kaynak (2005) focus on student satisfaction and retention. Additionally, there is a growing body of literature on the importance of graduate employability (EM) (Archer & Chetty, 2013; Dacre Pool & Sewell, 2007; O’Leary, 2013; Yorke, 2010) as an indicator of IE. A brief review of each outcome now follows.

The overarching paradigm of pragmatism (Onwuegbuzie & Leech, 2005; Creswell & Plano Clark, 2011) was deemed to be most appropriate in addressing a current real-life issue like IE of HEIs. The empirical component, which constitutes the crux of the article, is post-positivist in nature (Jacobs & Jacobs, 2014, p. 37), assuming that an exterior reality exists and that this reality cannot be known fully, but that it could be measured (at least partially).

The suitability tests yielded an index of 0.939 for the KMO test (Kaiser, 1960) and a significant result (p = 0) for Bartlett’s test of sphericity (Bartlett, 1954), both bettering the minimum requirements for conducting factor analysis, namely an index of 0.6 (for KMO) and p < 0.5 for Bartlett’s test (Pallant, 2007, p. 181; Tabachnick & Fidell, 2007). The data were therefore deemed suitable for factor analysis.

As shown in Table 3, an application of PAF on the initial data set (based on the 26-item questionnaire) suggested that four factors met Kaiser’s (1960) criterion of eigenvalues greater than 1 and may therefore be retained for further analysis (Velicer, 1976, p. 322).

Items 7–25 have been omitted from Table 3 for the sake of brevity. The results show that together the four principal factors (eigenvalue > 1) collectively explain just below 60% of total variability in student perceptions of the quality of the educational outcomes they derive from the institution.

Upon inspection of the initial pattern and structure matrices, items that did not load substantially (i.e. factor coefficient less than 0.4) (Pallant, 2007, p. 192) were removed, and EFA was rerun. The removal of low loading items improved the total variance explained by almost 2% to 61.78% (see Table 4), which exceeds the required threshold of 60% for acceptable construct validity (Field, 2013, p. 677) and more than the mean value of just over 52% reported by Henson and Roberts (2006, p. 402) from a review of 43 articles in applied psychology research.

To further interrogate the number of permissible factors, parallel analysis (Horn, 1965) was performed which also yielded four factors with eigenvalues greater than the corresponding criterion values for a randomly generated data matrix of comparable size (Choi, Fuqua, & Griffin, 2001; Horn, 1965), thus further supporting the appropriateness of a four-factor solution, as shown in Table 5.

To facilitate the interpretation of the four factors, Oblimin rotation (with Kaiser normalisation) was performed. The results (see Table 6) show a clear four-factor structure solution, with each item loading strongly on only one factor and each factor having the recommended minimum of three variables (Pallant, 2007, p. 192). Table 6 shows the combined pattern and structure matrix after low loading items were removed. As all but two items loaded strongly on the corresponding factors as had been conceptualised in the design of the instrument, the factors were intuitively labelled according to the constructs as initially theorised.

Two items (aa5 and aa6) initially thought to be indicative of AA were instead found to be more indicative of ES. Further reflection on the formulation and meanings of these two items suggests that they are indeed more indicative of the general quality of academic provision and therefore more likely to influence satisfaction than AA.

Weak to moderate positive correlations were observed between the four factors (see Table 7), thus signalling that although the four constructs may be regarded as distinct scales, they could be mutually reinforcing. For example, the strongest correlation is observed between ES and SR (r = 0.591), which supports the intuition that when students are happy with the quality of educational provision, they would be more loyal to the institution and strive to retain their place at the institution.

To summarise, the results highlight the possibility that each of the four constructs represents a unique indicator of IE in the students’ minds, thus validating the initial students’ educational outcomes dimensions as were conceptualised at the onset of the study.

Table 8 shows that in all three phases, Cronbach’s alpha values exceeded the admissible threshold of 0.7 (DeVellis, 2003), thus signifying that items within each cluster reasonably measured the same construct, that is, they are unidimensional. No significant changes in alpha were observed over the three phases. Although this might suggest that dropping the three items with factor pattern coefficients below 0.4 was redundant, it yielded an increase in the total variance explained (see Table 4). All four constructs had the recommended minimum of three items per construct (Pallant, 2007, p. 192).

In line with its aim, the article focuses on demonstrating the reliability and validity of the proposed instrument as a tool for estimating the IE of a HEI in terms of students’ perceptions of ES, AA, EM and SR. Firstly, the research results demonstrate the reliability of the instrument, with Cronbach’s alpha coefficients ranging between 0.77 and 0.89 for all four constructs. Secondly, the results from EFA support the four components of IE as initially conceptualised. Furthermore, EFA results support the preliminary expert opinions regarding the face validity and content validity of the instrument. Thus, in essence, the results signify the potential value of SEEQ as a useful instrument for measuring the four indicators of IE explored in this study. It is, however, imperative to review each key element of the results in relation to findings from prior research.

Firstly, the instrument was found to reliably and validly measure ES in the context of the participating HEI. The results support the value of importing the construct of consumer satisfaction from consumer behaviour (cf. Kotler, Keller, Koshy, & Jha, 2009; Sojkin, Bartkowiak, & Skuza, 2012) into the HE domain. Further, the constituent items that make up the ES construct (see Table 7) support the view of many others and that student ES is a function of both the content (i.e. quality of learning outcomes, quality of advising, quality of administrative and technical support) and process (i.e. the efficiency of administrative and technical operations as well as the school climate) of the learning environment (Maddox & Nicholson, 2014; Wilkins & Balakrishnan, 2013).

Secondly, the results show that SEEQ can be used as a reliable and valid instrument to measure AA, a construct recognised in the SA HE policy environment as arguably the most potent indicator of teaching and learning effectiveness (DHET, 2013). The results support the view that students’ perceptions of AA can be a credible proxy (in lieu of assessment scores, for instance) for determining levels of AA (Guay et al., 2010; Stadem et al., 2017).

Thirdly, the empirical results from this study demonstrate the usefulness of SEEQ to reliably and validly measure EM as an indicator of IE from a student’s perspective. This study conceptualises EM as a dynamic phenomenon (rather than a static outcome), focusing on an institution’s capacity to equip students with competences and attributes required to access (or create) and sustain gainful employment. The six items that signify EM in SEEQ draw heavily from (and, in turn, by virtue of the current results lend empirical credence to) Knight and Yorke’s (2002) Understanding, Skills, Efficacy beliefs and Metacognition (USEM) and to a lesser extent, Dacre Pool and Sewell’s (2007) CareerEDGE model of graduate EM. Also, importantly, the items focus on attributes that aim to build the individual into a potentially useful resource, regardless of context, thus advancing Schwartzman’s (2013) idea that the value of EM must extend beyond mere vocational needs.

Fourthly, the results demonstrate that SEEQ can be used as a reliable and valid instrument for measuring SR as an indicator of IE. In interrogating the construct of SR, SEEQ incorporates and affirms both components of retention, namely persistence (driven by factors such as academic and social engagement) and loyalty (driven by factors such as institutional image) as argued for many years by many scholars, notably Tinto (1982, 1998) and more recently, Angulo-Ruiz and Pergelova (2013), as well as Von Treuer and Marr (2013).

These results are to be understood against the backdrop of the demographic and contextual characteristics of the research sample, as these factors might influence reliability and validity scores (Cizek, 2016; Schaap & Kekana, 2016). Accordingly, it is important to keep in mind that the research sample was drawn from four schools (cf. faculties) of a SA HEI. The gender split of the sample was approximately 60:40 in favour of females. About 35% of the sample were aged 24 or younger and a further 34% were between 25 and 34 years old. The sample included students studying via various modes of delivery, including distance learning (26%), full-time contact learning (19%) and part-time contact learning (52%). Most (86%) of the participants were either currently employed or have had some work experience.

Analyses of the potential influence of the selected student demographic variables (see Table 1) on the SEOs indicated, in summary, that only two of these factors, namely, mode of delivery and work experience, appeared to materially influence the outcomes. More specifically, it was observed that full-time contact learning students (who also constituted the vast majority of students with no work experience) reported significantly lower levels of ES, EM and SR. No demographic variable was found to influence AA. A detail exposition of the results lies outside the scope of this article and is reported elsewhere (see Ayuk, 2016; 2017)

This article offers researchers and academic managers who might be interested in evaluating the performance of HEIs a coherent set of constructs against which such assessment may be conducted. More pertinently, it offers a simple, yet multidimensional instrument that might be considered for collecting credible data on the selected educational outcomes. By detailing the process of instrument design as well as demonstrating its reliability and credibility in a specific context, the article enables the user to appropriately judge the suitability of the instrument for the intended context.

A credible data collection instrument is imperative for the generation of quality information on which academic leaders and managers may make evidence-based decisions on matters such as instructional design, design of academic support programmes as well as work readiness and work-integrated learning programmes.

The proposed instrument might also be a starting point for other interested researchers to test the appropriateness and credibility of its use in different contexts and by so doing, contribute towards a more comparable body of knowledge on what we know about the ways and extent to which HEIs serve the educational needs of their students.

As noted earlier, the findings of this article are based on data from one institution, whose contextual factors might distinctly influence the test reliability and validity (Cizek, 2016; Schaap & Kekana, 2016). Hence, the application of the proposed instrument in multiple institutional (private and public HE) contexts would enhance opportunities for more generalisable claims regarding its reliability and validity.

Furthermore, the items included in the proposed instrument are by no means exhaustive. It can be expected that further interrogation and application of the instrument might result in the addition and/or removal of items in ways that further enhance its reliability and validity.

Further, findings from EFA can at best only provide preliminary support of the structural validity of an instrument. Further research requiring a new (and comparable) data set and using analytical techniques such as CFA or SEM is required to refine the theory resulting from this initial effort, thus strengthening the credibility (Lahey et al., 2012) of the proposed instrument.