Abstract
Orientation: Work-related research from the perspective of psychology journal publications is reviewed, indicating research topic trends and research method use.
Research purpose: What psychology journals are publishing about work-related topics as well as how these topics are being investigated was indicated. The specific objectives of this study were to analyse what research methods are being used, how these methods are being used and for what topics in work-related research.
Motivation for the study: The lack of studies that investigate the use of research methods and work-related study themes from the perspective of miscellaneous psychology journals prompted this study.
Research approach/design and method: A systematised review design was followed based on data collected by a previous study. Work-related research articles (n = 73) from five top-tier international miscellaneous or general psychology journals (published between 2013 and 2017) were collected and categorised.
Main findings: Quantitative methods, convenience samples, cross-sectional designs and questionnaires for data collection as well as analysis of variance were the most frequently used methodologies. Workplace relationships, job search quality and re-employment and work stress were the most frequently investigated topics.
Practical/managerial implications: Researchers should pay attention to the areas that lack methodological transparency in their studies to address the replication crisis in psychology. Method use should be expanded beyond quantitative methods where applicable. Industrial and organisational psychologists are reminded of their identity as applied psychologists.
Contribution/value-add: The use of research methods in work-related research published by psychology journals is presented. Trends for this sample as well as areas for improving the replication crisis in psychology were identified.
Keywords: applied psychology; industrial and organisational psychology; psychology publication; research methodology; research trends; replication crisis; quantitative methods; convenience sampling.
Introduction
Orientation
The early concept of industrial and organisational (I-O) psychology was investigated by industrial engineers Frederick Winslow Taylor and Frank Gilbert (Tiemann, 2019). However, it was the ‘people experts’ (Tiemann, 2019, par. 4) from the field of psychology that finally created the applied psychology discipline (Cunningham, 2010; Kazi, 2012). It is speculated that the first book written about I-O psychology was written by a psychologist who had experienced work conflict with his Harvard colleagues, namely, Hugo Munsterberg (Landy & Conte, 2004). Munsterberg is seen as the ‘first man to break the ice’ (Moore & Hartmann, 1931, p. 4) between the world of work and psychology. He promoted the concept of addressing problems in industry through psychological methods (Schreuder, 2001). Then, the use of psychology in personnel recruitment further increased interest in the field (Schreuder, 2001). What followed was years of collaboration between the two fields to form what we now know as I-O psychology (Tiemann, 2019).
Industrial and organisational psychology typically applies theories and methods from the field of psychology to address issues in an organisational or work context (Tiemann, 2019), thereby making a key responsibility of industrial psychologists to conduct research through applying theories, knowledge and methods (Health Professions Council of South Africa [HPCSA], 2019). However, the field of psychology as a whole is experiencing a crisis (‘the replication crisis’) of no confidence because of failure and a lack of replication studies (see Earp & Trafimow, 2015; Efendic & Van Zyl, 2019; Everett & Earp, 2015; Makel, Plucker, & Hegarty, 2012; Martin & Clarke, 2017; Open Science Collaboration, 2015). Concurrently, I-O psychology has also been scrutinised for its applied research practices and validity (Grand et al., 2018). The abilities to replicate and reproduce findings are pillars in psychology (Wiggins & Chrisopherson, 2019), supporting truth or validity in the claims of studies (Neuliep, 1991). However, continuous failures to replicate esteemed experiments (Open Science Collaboration, 2012, 2015) and an outbreak of fraudulent studies (e.g. Bem, 2011) throughout the last decade have led to unsettled scientific ground in the psychology community, with many deeming the situation a replication crisis (Wiggins & Chrisopherson, 2019). To counter the replication crisis, psychologists have focused on statistical and methodological reform and other problems in the field, namely, research and publication practices, statistical power, measurement, null hypothesis testing, replication and transparency (Wiggins & Chrisopherson, 2019).
Transparency with regard to methodology and data used go hand-in-hand with the reproducibility of psychology studies (Derksen, 2019), and the application of sound research methodology is critical for supporting the field of psychology as a scientific endeavour, as ‘the method really matters for the science and therefore psychology’ (Haig, 2018, p. 1). Scholtz, De Klerk and De Beer (2020) found that the broad field of psychology tends to utilise quantitative research methods above all other research methods (as defined by Nieuwenhuis, 2016) in a sample of five international journals. Furthermore, a lack of rigour by means of transparency of the used research method in articles was also indicated. The broad topic trends investigated by these five journals resulted in 10 areas of research in psychology (Weiten, 2012) of which the most popular research topic, namely, social psychology, included work-related articles.
Literature review
Topics investigated in I-O psychology specifically range from investigating industrial issues to how employees behave and function in their work environment (Tiemann, 2019). Topic trends in I-O psychology research typically concur and evolve with the current state of work, organisations, society and developments in the broader field of psychology (Kozlowski, 2017). In today’s modern world, the area of work-related research is at a critical point of investigation as it is currently experiencing the Fourth Industrial Revolution through technology (Schwab, 2016). This revolution is considered the most important economic and societal development globally as it will change the fundamental nature of society, work and business (Arntz, Gregory, & Zierahn, 2016; Brynjolfsson & McAfee, 2014; Ford, 2015). Concurrently, the Society of Industrial and Organisational Psychology predicts that the research topic trends for 2020 will include data visualisation and communication, virtual working spaces, meaningful work, health and well-being at work, change of the nature of work and artificial intelligence, amongst others (see Haynes, 2020).
Previous academic studies have also investigated the occurrence of research trends in I-O psychology from various perspectives. A systematic review by Schreuder and Coetzee (2010) focused on topic trends in accordance with the sub-disciplines of industrial psychology as identified by the Health Professions Council of South Africa from 1950 to 2008. The main trends found during this period were that research in personnel psychology has declined over the years, whereas employee well-being and organisational psychology has increased proportionally (Schreuder & Coetzee, 2010). Coetzee (2019), on the other hand, provided a more recent overview of scholarly research trends in the South African Journal of Industrial Psychology specifically. Seven core domains of investigation were identified, which included scale development and measurement of individual and organisational behaviour as well as themes regarding diversity and themes about the societal context of people’s behaviour (see Coetzee, 2019).
Research purpose and objectives
Despite I-O psychology’s long history in the field of psychology research, a synthesis of work-related research from the perspective of broad psychology journals is lacking. Reviews tend to focus on synthesising articles concerned with work from specific I-O psychology journals or domains for the purpose of collecting discipline-specific articles. Reviews of work-related research can indicate where the field currently is, as well as provide recommendations for advancing the field (Casper, Eby, Bordeaux, Lockwood, & Lambert, 2007). In addition, the field of I-O psychology is experiencing new developments (because of the Fourth Industrial Revolution) and critique of its methods of application. Thus, an exploration into the current trends with regard to topics investigated and methodologies applied in work-related research is warranted. Review articles are further encouraged as a way to ‘examine emergent methods as up-and-coming research tools or techniques (both quantitative and qualitative) that innovatively address complex organisational research problems and develop new knowledge’ (Reio, 2009 as cited in Nimon, 2016, p. 457). Furthermore, Nimon (2016) identified the publication of articles, such as the current article, as a natural step in the development of a research field and that it may enhance the scientific rigour in the field. Thus, the current research study aims to contribute to the academic domain of I-O psychology by indicating work-related research trends through a systematised review of psychology journals. Therefore, the research question is, ‘what are the work-related research trends investigated by miscellaneous international psychology journals?’
Work-related articles from data collected by Scholtz et al. (2020) were to reach the aim of this research study and the included sample is available at https://methodgarden.xtrapolate.io/. They analysed a sample of 999 psychology articles across five miscellaneous top-tier international psychology journals published between 2013 and 2017 to reach the following objectives: what research methods are being used, how these methods are being used and for what topics in practice (i.e. journal publications). Data from their study were vast (included all topics in psychology, e.g., developmental psychology); however, results were only presented superficially. Work-related articles (n = 73) merely formed one code as part of a larger theme.
The previous study (Scholtz et al. 2020) discussed this theme in relation to the broad field of psychology and paid little attention to the articles these codes consisted of. The detailed content of these work-related articles, as well as their potential to provide I-O psychology insight, was therefore unused – thus allowing this research study the opportunity to provide unique insights through an in-depth presentation of work-related research articles, specifically. The specific objectives of this research study were the following: what research methods are being used, how these methods are being used and for what topics in work-related research articles from miscellaneous psychology journals. These objectives will make the contribution of this research study threefold. Firstly, the data will allow insight into the rigour applied in these articles. Secondly, methodological trends will be made apparent and, lastly, the work-related topics that are of interest to the broad field of psychology will be made visible.
This article is unique as it aims to explore research trends in the field of I-O psychology from the perspective of the broad field of psychology, and not only from I-O psychology-oriented journals, which allows for a synthesis of topics from a different perspective than previous reviews. Another possible unique contribution is that this investigation into work-related articles published in psychology journals can also indicate the interdisciplinary relationship between psychology fields. Interdisciplinary research, especially between scholars, is seen as imperative for societal well-being as various branches of knowledge are combined (Van Kerkhoff, 2014). In addition, the current research study has taken an exploratory approach to code and creating themes (categories) of the topic trends instead of using predefined categories. Lastly, this research study utilised a systematised review design, a design that is growing in popularity, thereby contributing to the use of this methodology in the field of psychology by increasing its exposure.
Research design
Review approach
A systematised review design was followed to collect data in this study, which consisted of the following: select sample (journals identified through SCImago journal country rank), apply inclusion criteria to journals and review sample, start data analysis (Title > Abstract > Full text) and cluster artciles/codes under relevant sections (topics, methods and methodology). Lastly, produce findings in table format and write report (Scholtz et al., 2020). Systematised reviews are used to code and categorise data in tabular form in a systematic manner (Grant & Booth, 2009).
Sample
The journal sample for this research study was selected in 2018 through purposive sampling (Ritchie, Lewis, & Elam, 2009) and included journal articles published between 2013 and 2017 to present the most recent indication of research trends at the time of the research study (Lee, 2015). Journals were included in the sample if they formed part of the top five miscellaneous psychology journals ranked on the SCImago Country Journal Ranking website (SCImago Journal & Country Rank, 2017). SCImago presents a yearly list of journals based on Scopus® database, the largest database of peer-reviewed citations and abstracts globally (Scopus, 2017).
Journals were excluded if they provided no full-text access, focused on a specific discipline in psychology or indicated a preference for certain research methods. In addition, only empirical articles were included from these five journals. Thus, opinion pieces, book reviews and interview reports were excluded. These inclusion and exclusion criteria were deemed appropriate for the current research study because they allow the researchers to address the research question from the perspective of a broad recent sample of psychology journals.
The included journals were British Journal of Psychology, Europe’s Journal of Psychology, Journal of Psychology: Applied and Interdisciplinary, Australian Journal of Psychology and the International Journal of Psychology. From these journals, 999 articles were collected of which only 73 articles focused on work-related research and therefore formed the sample for the current study.
Research procedure
The research procedure for this systematised review design was adapted from the work of Ferreira, Bessa, Drezett and De Abreu (2016), and aspects of rigour as identified by Johnston, Kelly, Hsieh, Skidmore and Wells (2019) were applied throughout the process.
Firstly, the systematised review procedure was approved by a research and ethics committee (Johnston et al., 2019) to increase the rigour of the research study.
After obtaining ethical approval in 2017, data collection commenced in 2018 and ended in 2019. A systematic process was followed to screen for eligible journals and articles by the first author and reviewed by the second and third authors (Johnston et al., 2019). Codes were derived from these articles manually (Grant & Booth, 2009) and an audit trail was created on a Microsoft Excel spreadsheet for this process (Bandara, Furtmueller, Gorbacheva, Miskon, & Beekhuyzen, 2015; Johnston et al., 2019). During the coding process, an independent person acted as a co-coder. The coding process was guided by the research question (Johnston et al., 2019) and included the following: the design used, research topic (Bittermann & Fischer, 2018), sampling method (Ritchie et al., 2009), methods used (Nieuwenhuis, 2016) and methodology (data collection and data analysis [Maree, 2016]). Codes were assigned to articles based on the wording in each article. Then themes were created that resembled the codes. Themes and codes were also checked by the co-coder. Lastly, as per the systematised review design, the themes were categorised into a table format.
Ethical consideration
The original research study obtained from the Health Research Ethics Committee, Faculty of Health Sciences Ethics Office for Research Training and Support, North-West University, Potchefstroom Campus (clearance number: NWU-00115-17-A1) before the research commenced. The current research study still falls under this ethical clearance as it has merely provided in-depth information on the same objectives as the original research study using data that were collected and analysed by the original research study.
Results
Only 4% of the 73 articles included in this work-related research sample explicitly stated about the methods used in their studies, whereas 64% stated the used design. Furthermore, it is important to note that the frequency amount totals in these results refer to the occurrence of the concerned methodology in this sample and will therefore not amount to the article total as articles would often include more than one sampling or data collection or analysis or design. Lastly, the listed methodology has been identified in accordance with the wording used in the articles.
Overall, the Journal of Psychology: Applied and Interdisciplinary published the most work-related research, having almost double the number of articles as that published in Europe’s Journal of Psychology, which had the second highest amount (see Table 1).
| TABLE 1: Work code frequency in journals. |
With regard to specific research objectives, the results indicated the following: Firstly, the results showed that the quantitative research method was employed in 97% of work-related research articles published in the selected psychology journals, whilst only 1% of articles utilised a qualitative, review and mixed-method design each. There were no multi-method studies in this sample (see Table 2).
| TABLE 2: Method use in sample frequency. |
Secondly, the methodology utilised in the included sample of work-related research articles consisted of the following: design, sampling, data collection and data analysis.
Design
The designs applied in this research study concurred with the frequency of methods used, namely, that 59% of studies applied a cross-sectional design, 20% used a non-experimental design and 9% used experimental designs. The mixed-method study included in this research study used an ethnographic and cross-sectional design. Whilst the review merely indicated using a literature review, the qualitative method did not state its design (see Table 3).
| TABLE 3: Design use in sample frequency. |
Sampling
Most studies did not indicate their sampling method; however, some forms of inclusion and exclusion criteria were mentioned. Of those that had indicated, the convenience sampling method was the most popular method (used by 11% of studies), whereas stratified, snowball and random sampling methods each had a frequency of 3%. Sample sizes in the studies varied, ranging from 23 to 13 853 participants, with a mean of 749 and median of 310–312 participants. The modal samples were 242, 250, 266 and 290, each occurring twice in this sample (see Table 4).
Data collection
Data collection followed questionnaires for 85% and experimental tasks for 6% of the collection methods, concurring with the high frequency of quantitative methods. Interviews were also popular in this sample, amounting to 6% of the data collection methods (see Table 5).
| TABLE 5: Data collection method frequency. |
Data analysis
Analysis of variance was one of the most frequently used methods of data analysis (21.8%), followed by descriptive statistics (17.20%), confirmatory factor analysis (17.0%) and correlational analysis (8.9%). The remainder of the data analysis techniques is presented in Table 6.
| TABLE 6: Data analysis in sample frequency. |
Lastly, the topics that were investigated by the included work-related articles focused on combining work-related aspects with that from psychology. For example, the article by Helmes and Fudge (2017) investigated a combination of psychological distress and unemployment. Coding of articles therefore firstly focused on work-related aspects and themes were formulated based on these work codes. After creating themes, the psychological aspects addressed within these themes were identified, resulting in 13 work themes and their included psychological aspects (see Table 7).
| TABLE 7: Work-related themes and the psychology concepts as investigated by sampled articles. |
Discussion
Outline of results
This research study aimed to gain insights into the work-related research trends investigated by miscellaneous international psychology journals. Research method trends that were followed in these articles were mainly quantitative research methods, whilst qualitative, mixed methods and reviews only made up 3% of the articles.
Despite some growth in qualitative research in psychology, Pratt and Bonaccio (2016) found little evidence of this growth in I-O psychology, supporting the low frequency of qualitative methods found in this study.
O’Neil and Koekemoer (2016) and Coetzee and Van Zyl (2014) also found quantitative methods to be dominant in South African I-O psychology research. The reason for I-O psychologists’ preference for quantitative methods over qualitative methods could be because of qualitative method’s strength in realism but low precision and generalisability of results (Pratt & Bonaccio, 2016). However, despite the shortcomings of qualitative research, Pratt and Bonaccio (2016) highlighted various reasons that I-O psychology should apply this method, for example, creating new theories that are adapted to the changes in organisations or team science (Solis, Aristomene, Feitosa, & Smith, 2016). In contrast, Burlacu (2016) supported the continuation of quantitative methods and reminded that I-O psychologists are in charge of measuring performance and variables that influence performance as this impacts organisation and that measurement should be conducted in a common systematic numerical manner.
The manner in which these methods were employed in the sampled articles highlights a critical aspect in psychology today, namely, that of transparency, rigour and replication. Only 1% of studies in psychology are replication studies (Makel et al., 2012), and the trustworthiness of I-O psychology research is still of critical concern (see Kepes, Bennett, & McDaniel, 2013). Hardwicke et al. (2018) further supported this result and stated that articles often lack the basic methodological steps followed in I-O psychology research. According to Grand et al. (2018), robust science in I-O psychology is fostered through six principles, of which transparency, rigour and replication are three. These principles are not only the sole responsibility of the authors but also the reviewers of these articles (Grand et al., 2018). Schmidt and Landers (2013) added that despite the importance placed on replication studies, these studies are not incentivised by journals or rationally desirable for I-O psychology academics. Makel et al. (2012, p. 537) stated that researchers in psychology may be hesitant to conduct replication studies as they are often looked down for not contributing knowledge to the field and therefore labelled as ‘bricklayers’.
Furthermore, researchers also experience pressure to reach certain outcomes, whereas getting published may trump research accuracy (Derksen, 2019; Nosek, Spies, & Motyl, 2012). This could possibly hinder transparency (Derksen, 2019) and explain the lack of methodological rigour in articles. Hoole (2019) also presented a more holistic perspective on the replication crisis and listed research methodology, ethics, policies and systemic or institutional aspects as root causes. However, stating the research methodology along with its research question, amongst other methodological steps, still forms the cornerstones of transparent research, and a lack thereof challenges the robust science of I-O psychology (Open Science Collaboration, 2015). Therefore, McAbee, Grubbs and Zickar (2018) identified open science and transparency as critical elements in the applied research process to increase the scientific robustness of I-O psychology.
Following the lack of transparency overall, the applied sampling method, in the included articles, indicated minimal information about the used sample and identified no sampling strategy. The lack of identifying the population along with the appropriate justified sampling strategy to draw a sample representatively has long needed attention in I-O psychology (Fisher & Sandell, 2015). Despite this lack of transparency, the most widely reported sampling method found in this sample of articles was convenience sampling. The use of convenience samples is habitual in psychology and requires a clear statement about the limitations of its use in studies (Nielsen, Haun, Kärtner, & Legare, 2017). The results indicated that sample sizes varied in this sample, and its applicability is generally determined by the hypothesis or aim of the specific study (Michel, Hartman, O’Neill, Lorys, & Chen, 2016). With regard to the designs used in this sample, the results concurred with the high frequency of quantitative research in that cross-sectional design was the most frequently applied design. Despite the view that longitudinal designs provide more causal results, Spector (2019) stated that it has a limited advantage over cross-sectional designs. Its popularity can be seen throughout organisational psychology history (Spector, 2019) and the current results and will most likely continue.
Questionnaires, whether online or paper-based, are a popular choice in cross-sectional designs for data collection (Spector, 2019). The high frequency of questionnaires is also a testament to the lack of qualitative methods in this sample. Data analysis consisted of a wide variety of techniques, as can be seen in Table 6. The analysis of variance method was used in most of the studies.
Lastly, the data indicated 13 work-related research topics that are of interest to the broad field of psychology based on the sampled journals. As can be seen from Table 7, the integration of psychological concepts |
Hide
Show all
Email this article (Login required)
