ARTICLE

The Reliability of the Manchester Triage System (MTS): A Meta-analysis

Amir Mirhaghi1, Reza Mazlom3, Abbas Heydari 2, Mohsen Ebrahimi4

1. Department of Medical-Surgical Nursing, School of Nursing and Midwifery, Mashhad University of

Medical Sciences, Mashhad, Iran
2. Evidence-Based Caring Research Center, Department of Medical-Surgical Nursing, School of
Nursing and Midwifery, Mashhad University of Medical Sciences, Mashhad, Iran.

2. Department of Medical-Surgical Nursing, Faculty of Nursing and Midwifery, Chahrrah-e-Doktorha,

Mashhad, Razavi Khorasan, Iran

3. Department of Emergency Medicine, Imam Reza Hospital, Imam Reza Circle, Mashhad, Razavi
Khorasan

Correspondence: Amir Mirhaghi, Department of Medical-Surgical Nursing, School of Nursing and

Midwifery, Mashhad University of Medical Sciences, Mashhad, Iran. Tel: +98 511 8591511; Fax: +98
511 8539775; Email: mirhaghia@mums.ac.ir.

Conflict of Interest Statement: None

Accepted on August 29th, 2016

Keywords: Triage; Manchester Triage System; Emergency Treatment; Reliability and

Validity; Meta-Analysis.

Abstract

Objective
Although the Manchester Triage System (MTS) was first developed two decades ago, the
reliability of the MTS has not been questioned through comparison with a moderating
variable; therefore, the aim of this study is to determine the extent of the reliability of MTS
using a meta-analytic review.
Method
Electronic databases were searched up to March 1st, 2014. Studies were only included if they
had reported sample sizes, reliability coefficients, and adequate description of the Emergency
Severity Index (ESI) reliability assessment. The Guidelines for Reporting Reliability and
Agreement Studies (GRRAS) was used. Two reviewers independently examined abstracts
and extracted data. The effect size was obtained by the z-transformation of reliability

This article has been accepted for publication and undergone full peer review but has not been
through the copyediting, typesetting, pagination and proofreading process, which may lead to
differences between this version and the Version of Record. Please cite this article as doi:
10.1111/jebm.12231.

This article is protected by copyright. All rights reserved.

coefficients. Data were pooled with random-effects models, and meta-regression was
performed based on the method of moments estimator.
Results
Seven studies were included. The pooled coefficient for the MTS was substantial at 0.751 (CI
95%: 0.677-0.810); the incidence of mistriage is greater than fifty percent. Agreement is
higher for the latest version of MTS (for adults) among nurse-experts and in countries in
closer proximity to the country of MTS origin (the UK, in Manchester) than for the oldest
(pediatric) version, nurse-nurse raters, and countries at a greater distance from the UK.
Conclusion
The MTS showed an acceptable level of overall reliability in the emergency department, but
more development is required to attain almost perfect agreement.

Introduction

Patients are categorized based on clinical acuity when admitted to the emergency department;
the more critically ill a patient is, the more immediate his or her treatment and care needs (1).
The Manchester Triage System (MTS) is a five-level emergency department triage algorithm
that has been continuously developed in the United Kingdom and adopted by several
countries. A 5-point triage scale, the MTS has been endorsed by the Accident and Emergency
Nurses Association (2-8). The MTS is based on an algorithmic approach in which the
patient’s complaints are compared with one of 52 flow chart diagrams as well as with key
discriminators for each of these diagrams (9,10).
Several studies have investigated the validity and reliability of the MTS scale in adult and
pediatric populations (2-8); however, it remains unclear to what extent the MTS would
support consistency in triage nurses’ decision making in the United Kingdom in comparison
to other countries, considering the wide variety of health care systems around the world.
Some studies (11,12) have addressed contextual influences on the triage decision making
process, and defining these variables’ impact on the reliability of the triage scale is important.

This article is protected by copyright. All rights reserved.

2
To date, some studies have reported moderate consistency for MTS; however, the MTS
requires more exploration in terms of participants, statistics, instruments, and other
influencing criteria as well as with respect to mistriage.
The reliability of triage scales should be assessed by internal consistency, repeatability, and
inter-rater agreement (13). However, kappa statistics have been most commonly used to
measure inter-rater agreement, so it’s worth mentioning that kappa statistics could be
influenced by incidence, bias, and levels of scale, culminating in misleading results (14-16).
Weighted kappa statistics have been reported to reveal high and deceiving reliability
coefficients (13). Therefore, computing a pooled estimate of a reliability coefficient could
help us identify significant differences among reliability methods.
Meta-analysis is a systematic approach for introduction, evaluation, synthesis, and unified
results in relation to studying research questions. It also produces the strongest evidence for
intervention (17). Therefore, it is an appropriate method to gain comprehensive and deep
insight into the reliability of a triage scale, especially in regard to kappa statistics.
In one review of the reliability of the MTS, the kappa values ranged from 0.31 (moderate) to
0.62 (almost perfect) (18). The considerable variation in the kappa statistics indicates a real
gap in the reliability of triage scale. In view of the methodological limitations of the triage-
scale reliability, the prevalence (by necessity) of context-based triage decision making, and
the need for comprehensive insight into scale reliability during the triage process in
emergency departments, the aim of this study was to provide a meta-analytic review to
determine the extent of reliability of the MTS.

Methods

Literature search
In the first phase of the study, a literature search was conducted using the Cinahl, Scopus,
Medline, Pubmed, Google Scholar, and Cochrane Library databases up to March 1st, 2014.
Searching databases were not limited to time periods. The search terms included
“Reliability”, “Triage”, “System”, “Scale”, “Agreement”, “Emergency”, and “Manchester
Triage System” (Appendix A). The University Research ethics committee approved the
study. Relevant citations in the reference lists of final studies were hand-searched to identify
additional articles regarding the reliability of MTS. Two researchers independently examined
the search results to recover potentially eligible articles. Authors of research articles were
contacted to retrieve supplementary information if needed.
Eligibility criteria
Irrelevant and duplicated results were eliminated. Only English-language publications were
reviewed. Articles have been chosen according to the Guidelines for Reporting Reliability
and Agreement Studies (GRRAS) (19). According to the guidelines, only those studies that
had reported descriptions for sample size, number of raters and subjects, sampling method,
rating process, statistical analysis, and reliability coefficients were included in the analysis.
Each item was graded qualified if described in sufficient detail in the paper. A qualified paper

This article is protected by copyright. All rights reserved.

3
was defined as a paper with a qualifying score of > 6 out of the 8 criteria. Disagreements
were resolved by consensus. Articles that did not report the type of reliability were excluded
from the analyses. Researchers also recorded moderator variables such as participants, raters,
origin, and publication year of study. Studies were also noted that were conducted based on
the latest version of the triage scale.

Data extraction
In the next phase, participants (age-group, size), raters (profession, size), instruments (live,
scenario), origin and publication year of study, reliability coefficient, and method were
retrieved. Reliability coefficients were extracted from articles as below:
- Inter-rater reliability: Kappa coefficient (weighted and un-weighted), intraclass
correlation coefficient, Pearson correlation coefficient, and Spearman rank correlation
coefficient.
- Intra-rater reliability: Articles which contained reliability statistics including Pearson
correlation coefficient, intraclass correlation coefficient, and Spearman rank correlation
coefficient were included.
- Internal consistency: Articles that reported alpha coefficients were included in the
analyses.
In meta-regression, each sample was considered a unit of analysis. If the same sample was
reported in two or more articles, it was included once. In contrast, if several samples
regarding different populations were reported in one study, each sample was separately
included as a unit of analysis.
Data analysis
Pooling data was performed for all three types of reliability. The most qualified articles
reported reliability coefficients using kappa statistics, which could be considered as an r type
of coefficient ranging from -1.00 to +1.00. Standard agreement definition was used as poor
(κ=0.00–0.20), fair (κ = 0.21–0.40), moderate (κ=0.41–0.60), substantial (κ=0.61–0.80), and
almost perfect (κ=0.81–1.00) (20). Kappa can be treated as a correlation coefficient in meta-
analysis (21). In order to obtain the correct interpretation, back-transformation (z to r
transformation) of pooled effect sizes to the level of primary coefficients was performed
(22,23). Fixed effects and random effects models were applied. Data were analyzed using
Comprehensive Meta-Analysis software (Version 2.2.050; Biostat, Inc, Englewood, New
Jersey, USA).
Simple meta-regression analysis was performed according to the method of moments
estimator (24). In the meta-regression model, the effect size was considered as a dependent
variable and studies and subject characteristics as an independent variable to discover
potential predictors of reliability coefficients. z-transformed reliability coefficients are
regressed on the following variables: origin and publication year of study and studies based
on the latest version of the MTS scale versus on the prior version. Distance was defined as
the distance from the origin of each study to the place of origin for the MTS (Manchester,
UK). Meta-regression was performed using a random effects model because of the presence
of significant between-study variation (25).

This article is protected by copyright. All rights reserved.

4
Results
Study selection and characteristics
The search strategy introduced 81 primary citations relevant to the reliability of MTS(Figure
1). Finally, 7 unique citations emerged (8.6% out of the original 81) that met the inclusion
criteria (Figure 1). Subgroups were organized for participants (Adult/pediatric), raters (nurse,
physician, expert) and method of reliability (intra/inter rater), reliability statistics
(weighted/un-weighted kappa), and origin and publication year of study. The level of
agreement among reviewers through the final selection of articles was almost perfect (κ=1.0).
19,439 cases were included in analysis. The reliability of MTS has been assessed in 4
different countries. The publication year of the studies ranged from 1999 to 2014 (median,
2009). Twenty-eight percent of all studies were conducted using the latest version of the
MTS scale. Inter-rater reliability had been used in all studies except for two studies that used
intra-rater reliability. No study in our analysis used alpha coefficients to report internal
consistency in the reliability analysis. The weighted kappa coefficient was the most
commonly used statistic (Table 1). At 0.751, the overall pooled coefficient for the MTS was
substantial (CI 95%: 0.677-0.810).
Subgroup analyses
Participants’ pooled coefficients ranged from substantial at 0.768 (CI 95%: 0.694-0.826) for
nurse-nurse agreement to almost perfect at 0.863 (CI 95%: 0.262-0.982) for nurse-expert
agreement (Figure 2).
Agreement regarding the adult and pediatric versions of the MTS was substantial at 0.802
(CI 95%: 0.718-0.864) for adults and 0.775 (CI 95%: 0.522-0.883) for pediatrics (Figure 3).
Agreement regarding paper-based scenario assessment was substantial at 0.768 (CI 95%:
0.694-0.826), while it was almost perfect at 0.863 (CI 95%: 0.262-0.982) for assessment of
real live cases.
Agreement regarding inter-rater and intra-rater reliability was substantial at 0.793 (CI 95%:
0.719-0.849) and 0.805 (CI 95%: 0.683-0.883), respectively. Agreement relating to weighted
kappa was substantial at 0.781 (CI 95%: 0.716-0.832) and also for the unweighted kappa at
0.656 (CI 95%: 0.478-0.782). Agreement regarding the most updated version was almost
perfect at 0.852 (CI 95%: 0.630-0.946); for the prior version, agreement was substantial at
0.756 (CI 95%: 0.675-0.819).
Only three studies (3,6,7) have presented a (5x5) contingency table to show the frequency
distribution of triage decisions upon each MTS level between two raters (Table 2). Overall
agreement was 37.68%. Agreement for each MTS level was MTS L-1 (0.44%), MTS L-2
(2.92%), MTS L-3 (13.25%), MTS L-4 (20.70%), and MTS L-5 (0.37%); disagreement was
(1.61%), (9.85%), (21.26%), (20.23%), and (9.36%) respectively. Decisions leading to
mistriage occurred in 59.51% of cases; overtriage occurred for most of these decisions
(46.65%), while undertriage accounted for the remaining 12.86% of decisions (Table 2).
Moderator effect
Meta-regression analysis was performed based on the method of moments for moderators
(distance, publication year, and studies which were conducted based on the latest version)
(Table 3). Studies according to the latest version of scale, publication year of study and the

This article is protected by copyright. All rights reserved.

5
distance from the origin of the MTS significantly showed higher pooled coefficients. Studies
reported higher pooled coefficients for locations closer to the UK as opposed to further away
(Table 3).

Discussion

The overall reliability of the MTS is substantial in the emergency department. The MTS
showed an acceptable level of reliability to guarantee decisions were made consistently
regarding allocating patients to appropriate categories. Furthermore, the MTS supports
evidence-based practice in the emergency department (26,27). In spite of these positive
findings, it is worth-mentioning that there is a considerable gap between research and clinical
practice even at the best of times (28). In addition, most studies used weighted kappa
statistics to report reliability coefficients (Table 1), and the fact that weighted kappa statistics
overestimate the reliability of the triage scale (29) makes it necessary to interpret the results
with extreme caution. So, it’s probably important to keep in mind that the degree of reliability
for MTS is actually at the moderate level, which is congruent for several studies (2,3).
Almost 3 in 5 (59.51%) triage decisions were recognized as mistriage. Although it`s
remarkable and alarming, 46.65% (or more than three-fourths of the total) of these incidents
were overtriage, and it could extenuate disagreement among raters in favor of patients (7). In
addition, 3.96% of the triage decisions were related to undertriage in levels I and II (Table 2).
Compared to other triage scales, mistriage in the Emergency Severity Index (10.93%) is
notably lower than for MTS, with considerably higher agreement among raters (78.56%).
Improbably, storm-Versloot et al indicated that MTS has higher agreement than ESI (6); this
finding is perhaps justified by the fact that hospitals in the Netherlands are more familiar with
the MTS triage system.
However, the ESI has a strong tendency towards categorizing patients as level 2 (23.39% in
all), while the MTS has distributed patients among all of the different triage levels except for
level V, preventing an influx of patients that are all in the same category. Such an influx
could create significant disturbances in patient flow in the EDs and causes other parts of the
ED to remain unusable (12). In this way, the MTS behaves similarly to the Canadian Triage
and Acuity Scale (CTAS) and the Australasian Triage Scale (ATS) (30,31).
The MTS showed diverse pooled reliability coefficients regarding participants, patients,
raters, reliability method, and statistics. The results demonstrated that agreement was higher
for the latest (adult) version of MTS and among nurse-experts, and lower for the previous
(pediatric) version and among the other groups of raters. This result is congruent with ESI
moderators (12). All of these moderator variables could be more exclusively explored in
further studies.
The MTS has been documented and supported by scientific evidence in different countries.
However, only four countries have reported reliability studies on MTS; there are no studies
from its country of origin to our knowledge (the UK) on this topic (Table 1). In this way,
meta-regression showed that a significant difference exists in terms of distance from origin of
MTS. Among other items, meta-regression demonstrated that MTS has lower reliability
coefficients in countries that are far from the country of origin (UK) of the scale (Table 3).

This article is protected by copyright. All rights reserved.

6
One possible reason for this is the complaint-based nature (i.e. it is determined by how
patients present their situations when admitted in many instances) of MTS, which could be
translated in different fashions in routine practice in countries outside the UK, with different
health systems and cultures of health care practice. These results differentiate the MTS from
the ESI triage scale, which is highly generalizable and could be adopted in other countries in
spite of cultural diversities (12).
The third edition of MTS has been released (9) and the reliability of triage scales has been
improved through the years, indicating revisions had been effective and supporting the idea
that it’s necessary for emergency departments to update triage systems according to the latest
version of triage scale. Additionally, the MTS could be enhanced through the years and
showed improved reliability (Figure 4).
In general, intra-rater reliability is more satisfactory than inter-rater reliability (32), so it
revealed almost perfect agreement comparing to substantial agreement for inter-rater
reliability. While intra- and inter-rater reliability are intended to report the degree to which
measurements taken by the same and different observers are similar, respectively, other
methods of examining reliability have remained uncommon in studies regarding triage
reliability (33).
The weighted kappa coefficient showed substantial agreement, demonstrating higher
reliability than did the unweighted kappa coefficient because it put more emphasis on the
large differences between ratings than on small differences (34,35). It’s important to note that
even one category difference in allocating patients into the appropriate category could
endanger clinical outcomes of critically ill patients. Un-weighted kappa statistics provide
more a realistic estimation of the reliability of triage scales (29).
A number limitation of this study must be noted. In our analysis, none of these studies have
reported raw agreement for each individual MTS-level; only a few studies presented a
contingency table for inter-rater agreement between raters. Since this study is limited to
overall reliability, some inconsistencies may exist across each MTS level; therefore, the
results should be interpreted with caution.

Conclusion
Overall, the MTS triage scale showed an acceptable level of reliability in the emergency
department, and (except for level V) appropriately distributed patients into triage categories.
Therefore, the MTS triage scale at this point just needs more development to reach almost
perfect agreement and decrease disagreement. The reliability of triage scales requires a more
comprehensive approach that includes all aspects of reliability assessment, so further studies
are needed that concentrate on the reliability of triage scales, especially in different countries.

Table 1 Studies on reliability of MTS

Studies Patient Participants* Instrument Methods** Statistics*** Country

Goodacre 1999(2) Adult NE Scenario Inter Κuw Australia

Adult NN Scenario Inter Κuw
van der Wulp 2008(3) Adult NN Scenario Intra ICC Netherlands
Adult NN Scenario Inter Kw

This article is protected by copyright. All rights reserved.

7
 Grouse 2009(4) Adult NN Scenario Inter Κw Australia
Adult NN Scenario Inter Κuw
Olofsson 2009(5) Sweden
Adult NN Scenario Inter Κw
Adult NN Scenario Intra Κuw
Adult NN Scenario Inter Κuw
Storm-Versloot 2009(6) Netherlands
Adult NN Scenario Intra Κw
Adult NN Scenario Inter Κw
Pediatrics NE Live cases Inter Κw
van Veen 2010(7) Netherlands
Pediatrics NN Scenario Inter Κw
Graf 2014(8) Adult NE Live cases Inter Κw Germany
*
EE: expert-expert, NE: nurse-expert, NN: nurse-nurse, NP: nurse-physician, PE: physician-expert, PP: physician-physician.
**
Inter: inter-rater reliability, Intra: intra-rater reliability.
***
Kw: Weighted kappa, Kuw: Un-weighted kappa, r: correlation coefficient, ICC: Intraclass coefficient .
Table 2 The Contingency table of triage decision distribution relating to each MTS category among ED raters

ED Raters decisions by MTS category

ED Raters decisions
1 2 3 4 5 Total
*
1 71 (0.44) 38 84 26 7 226

2 234 468 (2.92) 1135 954 524 3314

3 79 202 2124 (13.25) 2347 735 5936

4 48 73 1321 3319 (20.70) 1629 6390

5 0 0 16 90 60 (0.37) 165

Total 431 781 5129 6735 2955 16032 (100.00)

* No (% of total)

Table 3 Meta-regression of Fisher`s z-transformed kappa coefficients on predictor variables (studies with weighted
kappa coefficient)

Independent variable B SEb P

Updated Version 0.357 0.026 0.00

Distance from Origin -0.00001 0.000 0.00

Publication date 0.18 0.008 0.00

References
1. Ebrahimi M, Mirhaghi A, Heydari A, Mazlom R, Nassehi A, Jafari M. The Role Descriptions of Triage Nurse in
Emergency Department: A Delphi Study. Scientifica 2016; 5269815.
2. Goodacre SW, Gillett M, Harris RD, Houlihan KP. Consistency of retrospective triage decisions as a standardised
instrument for audit. J Accid Emerg Med 1999; 16(5): 322-4.
3. van der Wulp I, van Baar ME, Schrijvers AJ. Reliability and validity of the Manchester Triage System in a general
emergency department patient population in the Netherlands: results of a simulation study. Emerg Med J 2008 Jul; 25(7):
431-4.
4. Grouse AI, Bishop RO, Bannon AM. The Manchester Triage System provides good reliability in an Australian
emergency department. Emerg Med J 2009; 26(7): 484-6.
5. Olofsson P, Gellerstedt M, Carlström ED. Manchester Triage in Sweden - interrater reliability and accuracy. Int
Emerg Nurs 2009; 17(3): 143-8.
6. Storm-Versloot MN, Ubbink DT, Chin a Choi V, Luitse JS. Observer agreement of the Manchester Triage System
and the Emergency Severity Index: a simulation study. Emerg Med J 2009; 26(8): 556-60.
7. van Veen M, Steyerberg EW, Ruige M, van Meurs AH, Roukema J, van der Lei J, Moll HA. Manchester triage
system in paediatric emergency care: prospective observational study. BMJ 2008; 337: a1501.

This article is protected by copyright. All rights reserved.

8
8. Gräff I, Goldschmidt B, Glien P, Bogdanow M, Fimmers R, Hoeft A, Kim SC, Grigutsch D. The German Version
of the Manchester Triage System and its quality criteria--first assessment of validity and reliability. PLoS One 2014; 9(2):
e88995.
9. Mackway-Jones K, Marsden J, Windle J. (eds) Emergency Triage: Manchester Triage Group. 3rd ed. London:
John Wiley & Sons; 2013. doi: 10.1002/9781118299029
10. Christ M, Grossmann F, Winter D, Bingisser R, Platz E. Modern triage in the emergency department. Dtsch
Arztebl Int 2010; 107(50): 892-8.
11. Andersson A, Omberg M, Svedlund M. Triage in the emergency department--a qualitative study of the factors
which nurses consider when making decisions. Nurs Crit Care 2006; 11(3): 136-45.
12. Mirhaghi A, Heydari A, Mazlom R, Hasanzadeh F. Reliability of the Emergency Severity Index Meta-analysis.
Sultan Qaboos University Med J 2015; 15(1): 67-73
13. Goransson K, Ehrenberg A, Marklund B, Ehnfors M. Accuracy and concordance of nurses in emergency
department triage. Scand J Caring Sci 2005; 19(4): 432-8.
14. Julius S, Wright CC. The kappa statistic in reliability studies: use, interpretation, and sample size requirements.
Phys Ther 2005; 85(3): 257-68.
15. van der Wulp I, van Stel HF. Calculating kappas from adjusted data improved the comparability of the reliability
of triage systems: a comparative study. J Clin Epidemiol 2010; 63(11): 1256-63.
16. Viera A, Garrett J. Understanding interobserver agreement: the kappa statistic. Fam Med 2005; 37(5): 360-3.
17. Petitti D. Meta-Analysis, Decision Analysis, and Cost Effectiveness Analysis. New York, NY: Oxford University
Press; 1994. P. 69.
18. Christ M, Grossmann F, Winter D, Bingisser R, Platz E. Modern triage in the emergency department. Dtsch
Arztebl Int 2010; 107(50): 892-8.
19. Kottner J, Audige L, Brorson S, Donner A, Gajewski BJ, Hróbjartsson A, Roberts C, Shoukri M, Streiner DL.
Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. Int J Nurs Stud 2011; 48(6): 661-71.
20. Julius S, Wright CC. The kappa statistic in reliability studies: use, interpretation, and sample size requirements.
Phys Ther 2005; 85(3): 257-68.
21. Rettew DC, Lynch AD, Achenbach TM, Dumenci L, Ivanova MY. Meta-analyses of agreement between
diagnoses made from clinical evaluations and standardized diagnostic interviews. Int J Methods Psychiatr Res 2009; 18(3):
169-84.
22. Hedges LV, Olkin I. Statistical Methods for Meta-analysis. Academic Press; 1985: 76-81.
23. Rosenthal R. Meta-Analytic Procedures for Social Research. SAGE Publications; 1991: 43-89.
24. Chen H, Manning AK, Dupuis J. A Method of Moments Estimator for Random Effect Multivariate Meta-
Analysis. Biometrics 2012; 68(4): 1278-84.
25. Riley RD, Higgins JPT, Deeks JJ. Interpretation of random effects meta-analyses. BMJ 2011; 342: d549.
26. Parenti N, Reggiani MLB, Iannone P, Percudani D, Dowding D. A systematic review on the validity and
reliability of an emergency department triage scale, the Manchester Triage System. Int J Nurs Stud 2014; pii: S0020-
7489(14)00020-0.
27. Farrohknia N, Castrén M, Ehrenberg A, Lind L, Oredsson S, Jonsson H, Asplund K, Göransson KE. Emergency
department triage scales and their components: a systematic review of the scientific evidence. Scand J Trauma Resusc
Emerg Med 2011; 19: 42.
28. Le May A, Mulhall A, Alexander C. Bridging the research–practice gap: exploring the research cultures of
practitioners and managers. Journal of Advanced Nursing 1998; 28(2): 428-37.
29. Goransson K, Ehrenberg A, Marklund B, Ehnfors M. Accuracy and concordance of nurses in emergency
department triage. Scand J Caring Sci 2005; 19(4): 432-8.
30. Mirhaghi A, Heydari A, Mazlom R, Ebrahimi M. The reliability of the Canadian Triage and Acuity Scale: Meta-
analysis. N Am J Med Sci 2015; 7(7): 299-305.
31. Ebrahimi M, Heydari A, Mazlom R, Mirhaghi A. Reliability of the Australasian Triage Scale: Meta-analysis.
World J Emerg Med 2015; 6(2): 94-99.
32. Eliasziw M, Young SL, Woodbury MG, Fryday-Field K. Statistical methodology for the concurrent assessment of
interrater and intrarater reliability: using goniometric measurements as an example. Phys Ther 1994; 74(8): 777-88.
33. Hogan TP, Benjamin A, Brezinski KL. Reliability methods: A note on the frequency of use of various types.
Educational and psychological measurement 2000; 60(4): 523-31.
34. Ebrahimi M, Mirhaghi A. Re: Inter-rater reliability and validity of the Ministry of Health of Turkey`s mandatory
emergency triage instrument. Emerg Med Australas. 2015; 27(5): 496-7.
35. Pieper D, Mathes T, Eikermann M. Impact of choice of quality appraisal tool for systematic reviews in overviews.
J Evid Based Med 2014; 7(2): 72-8.

Figure legends

This article is protected by copyright. All rights reserved.

9
Figure 1 Flow chart of literature search and selection process

Figure 2 Fisher`s Z transformed pooled estimates of participants` reliability (Random effects model) (NE:
nurse-expert, NN: nurse-nurse) Weighted Kappa

This article is protected by copyright. All rights reserved.

10
Figure 3 Fisher`s Z transformed pooled estimates of patients` reliability based on age group (Random
effects model) Weighted Kappa

Figure 4 Fisher`s Z-transformed kappa coefficients in relation to the Publication year of study.

This article is protected by copyright. All rights reserved.

11
This article is protected by copyright. All rights reserved.
12