J Med Syst (2017) 41: 199
DOI 10.1007/s10916-017-0826-0
SYSTEMS-LEVEL QUALITY IMPROVEMENT
Auditing The Completeness and Legibility
of Computerized Radiological Request Forms
Yahya Al Muallem 1 & Majed Al Dogether 1 & Mowafa Househ 1
Received: 23 January 2016 / Accepted: 25 September 2017 / Published online: 4 November 2017
# Springer Science+Business Media, LLC 2017
Abstract Certain Saudi healthcare organizations transfer out-
patients to medical imaging departments for radiological ex-
aminations in a manual process that relies on the use of paper-
based forms. With the increased implementation of electronic
medical records in Saudi Hospitals, little is known about the
completeness and legibility of information captured in
electronic-based medical imaging forms. The purpose of this
study is to audit the completeness and legibility of medical
imaging paper-based forms in comparison with electronic-
based medical imaging forms. As a secondary objective, we
also examined the number of errors found on the forms.An
observational retrospective cross-sectional study was uti-
lized to audit the completeness and legibility of both paper
and electronic forms collected between March 1 and
May 15, 2015. The study measured the association among
categorical variables using Chi-Square analysis. The results
of this investigation show a significant association between
form completion and type of record (i.e., paper vs. electronic)
where electronic-based systems were found to be more com-
plete than paper-based records. Electrnoic based records were
also found to improve form legibility, promote user adherence
to complete the forms and minimize entry errors. In conclu-
This article is part of the Topical Collection on Systems-Level Quality
Improvement
* Mowafa Househ
mowafah@gmail.com
1
College of Public Health and Health Informatics (CPHHI), King
Saud Bin Abdul Aziz University for Health Sciences (KSAU-HS),
King Abdullah International Medical Research Center (KAIMRC),
Riyadh, Kingdom of Saudi Arabia
2
College of Medicine, The University of Sharjah, Sharjah, United
Arab Emirates
& Basema Saddik 1,2
sion, electronic-based medical imaging forms are more com-
plete and legible than paper based forms. Future studies
should evaluate other hospitals and compare both legibility
and completeness of electronic-based medical imaging forms
and conduct usability evaluation studies with users to explore
the impacts of system design on both completeness and legi-
bility of electronic forms, in general, but more specifically,
electronic-based medical imaging forms.
Keywords Electronic medical record . Radiology .
Medical-imaging . Health informatics . Public health . Saudi
Arabia
Introduction
In Saudi Arabia, outpatients are transferred to radiology labs
to complete further diagnostics procedures such as x-rays,
Computed Tomography (CT) Scans and Magnetic
Resonance Imaging (MRI). In many Saudi hospitals, the pro-
cess occurs manually through a paper form that is given to the
patient by a nurse or physician which is then taken by the
patient to the radiology lab to obtain and schedule an appoint-
ment for the relevant diagnostic test. Although many hospitals
in Saudi Arabia have moved from paper to electronic-based
record systems, no studies have compared the impacts of mov-
ing from paper to electronic referral forms in general, and
specifically, as it relates to medical imaging.
The major challenge with paper-based medical imaging
referral forms involves their incompleteness and possible mis-
interpretation, due to illegible handwriting, of patient informa-
tion which can lead to medical errors. Standardizing how in-
formation is filled in medical imaging request forms can have
a positive impact on patient treatment as a result of complete
and legible information which can lead to reduced medical
199 Page 2 of 7
errors and improved clinical workflow [3, 4]. Tang, Lim, and
How addressed the importance of completing medical imag-
ing request forms by providing the necessary patient infor-
mation and completing fields guided by clinical prac-
tice guidelines, irrespective if it was a paper or an electronic-
based form [1]. Similarly, Alfolabi, Fadare, and Essien have
highlighted the importance of applying auditing measures for
completing the medical imaging request forms to force
users to complete the information on the referral form, irre-
spective if it was paper or an electronic-based form [2]. Not
having auditing measures can lead to medical errors due to
possible misinterpretations of medical imaging reports, espe-
cially, with paper forms [5, 6]. Incomplete radiological re-
quests are an obstacle to patient care which can impact the
communication among clinical staff leading to medical errors.
Having electronic-based medical imaging forms can force cli-
nicians to complete the forms making them more complete
and legible [7].
Saudi Arabia is going through a major healthcare
transformation[9]. The ministry of health, the private health
sector, as well as military related institutions are moving from
paper to electronic-based systems. One of the largest and most
well-known hospitals in Saudi Arabia is administered by the
Ministry of National Guard- Health Affairs (MNGHA), which
has six medical cities within the country of Saudi Arabia[10].
In 2016, MNGHA successfully implemented its BestCare
electronic health record system with the goal of making all
of its hospitals paperless[11]. As a result of the BestCare im-
plementation, paper-based medical imaging referral forms
were now being captured electronically. Little is known about
the impacts of moving from paper-based forms to electronic
based forms for medical imaging referrals. The purpose of this
work is to evaluate the completeness and legibility of outpa-
tient medical imaging paper-based request forms in compari-
son to electronic-based medical imaging request forms. The
purpose of making this comparison is to determine which
format, paper vs. electronic, provides a more complete and
legible medical imaging referral form. Although the literature
would argue that electronic forms are a better option for pa-
tients than paper-based forms, the literature does not show
how the electronic format is better than the paper format.
For example, the literature does not address whether a paper-
based signature is more authentic than a signature on an
electronic-based form. It’s expected that in some paper forms
clinicians will either forget or neglect to fill-in their informa-
tion (such as name, signature… etc.) where in the electronic
system, this information is filled-in automatically by the sys-
tem as they log-in with their usernames and passwords.We
anticipate that the work carried out in the study will be used
to inform further studies especially around issues relating to
the usability of electronic based systems, and especially, as it
relates to referral forms.
J Med Syst (2017) 41: 199
Methods
Study setting
The study was conducted at the Radiology department located
at the Ministry of National Guard Health Affairs Riyadh hos-
pital, also known as King Abdulaziz Medical
City. The Radiology department provides screening and diag-
nostic medical imaging services for inpatients and outpatients.
There are several sections under the department serving dif-
ferent specialties including emergency room x-ray, computed
tomography scan, magnetic resonance imaging, ultrasound
imaging, angiography imaging, mammography imaging, and
nuclear medicine imaging.
Sample
The sample consisted of both paper and electronic medical
imaging referral forms. We estimated the sample size to be a
total of 456 paper and electronic-based referral forms using a
simple random sampling method.
Furthermore, we added an inclusion and exclusion criteria:
& Inclusion: any paper or electronic radiological request
available at the radiology department.
& Exclusion: Radiological examination forms, tests or re-
cords that comes from outside the hospital, interns having
less than one-year training as well as visiting or locum
staff.
Study design
We used a cross-sectional retrospective study design to compare
paper and electronic-based medical imaging referral forms. The
study was conducted in November 2015, however, the sample
forms were collected between March 1 and May 15, 2015.
Ethical considerations
Ethical approval was provided by the King Abdullah
International Medical Research Centre in November 2015.
Data collection
Each paper and electronic form was analyzed in order to re-
view their completeness and legibility of the required fields
in both electronic and paper-based forms. Some of the infor-
mation audited in the study included: the date of the request,
patient information, requested procedure, requesting physi-
cian information and user signature.
J Med Syst (2017) 41: 199
The goal of the survey was to collect relevant information
about radiological request paper and electronic form com-
pleteness, including legibility, and errors. We defined com-
pleteness by the number of filled or unfilled mandatory fileds
in the forms. We defined legibility as how understandable,
readable, and clear the information is on the form. For errors,
we defined them as the number of faults that were made in
filling the form, as it would relate to demographic or clinical
information. We collected information on the listed fields in
the paper and electronic forms including patient information,
requested radiological exam, information accuracy and
requesting physician details. All questions listed in the data
collection form were reviewed by 3 researchers in addition to
a radiology technician. Any disagreements between the re-
viewers were resolved through consensus.
Data analysis
Both paper and electronic forms were evaluated for informa-
tion completion which included legibiligy, number of errors
(missing and unclear information) and type of record (paper or
electronic) in each form. Moreover, we used the chi-square
test and descriptive statistics using the Statistical Package for
Social Science (SPSS) software version 21 in the analysis.
Results
We a u d i t e d 4 5 6 p a p e r ( n = 2 2 8 ) a n d e l e c t r o n i c
(n=228) forms checking for completion, legibiligy, and num-
ber of errors. Each form was categorized according to their
departments (X-ray, CT scan, MRI…etc.). Unlike the comple-
tion percentages in the electronic forms, there was variation in
the completion percentages of paper forms as shown in the
cross-tabulation in Table 1. Table 1 shows that 83 incomplete
paper forms were classified as unclear (36.4%) and there were
no incomplete paper forms that were completely understand-
able or readable. However, 43.8% of the complete paper
forms were found to be unclear and only 0.9% were found
to be understandable. Generally, 36.4% of the paper forms
received by radiology departments were found to be
Table 1 Cross-Tab of
Completion vs Legibility in Paper Variables
Form
Completion (Paper) No
Yes
Total
Page 3 of 7 199
incomplete and have missing information in certain fields of
the form.
The result of the analysis shows a significant association
between field legibility and completion in paper forms, χ2 (2)
= 32.093, p < 0.001, Fisher’s Exact Test = 41.756, p < 0.001.
Additionally, investigating the association between com-
pletion and number of errors in paper forms shows that both
complete and incomplete paper forms have a number of errors
in the information filled in the forms. For example, most of the
paper forms have shown at least one error. Both complete and
incomplete paper forms show similar number of 72 forms
with at least one error in completed forms and 74 with at least
one error in incomplete forms, as shown in Table 2.
The result of this analysis shows a significant association
between completion and number of errors in paper forms χ2
(4) = 48.090, p < 0.001, Fisher’s Exact Test = 58.730, p <
0.001 . (Table 2).
Table 3 shows that the percentage of incomplete forms in
the electronic system is 0%. This is because all required fields
are mandatory and must be filled before submitting the re-
quest. In fact, the system does not allow the request to be sent
unless all required fields are completed. However, running the
chi-square test presented a significant association between
completion and type of record χ2 (1) = 101.469, p < 0.001.
Table 4 shows the percentage of unclear forms in the elec-
tronic system is 0%. This is because all required fields are
mandatory and must be filled before submitting the request.
In fact, the system did not allow the request to be sent unless
all required fields are completed.
The result of this analysis shows a significant association
between field legibility and type of record χ2 (2) = 317.952,
p < 0.001. Investigating the relation between completion and
field legibility has shown that all incomplete forms are con-
sidered neither understandable nor readable (Table 5).
However, completed forms show a noteworthy percentage
under the Bunclear^ category which highlights the clarity of
the written/entered information on each form. The result of
this investigation shows a significant association between
completion and legibility χ2 (1) = 151.372, p < 0.001.
On the other hand, examining the association of form com-
pletion and error category as shown in Table 6, shows that
Legibility (Paper) Total
Understandable Readable Unclear
Count 0 0 83 83
% of Total 0.0% 0.0% 36.4% 36.4%
Count 2 43 100 145
% of Total 0.9% 18.9% 43.9% 63.6%
Count 2 43 183 228
% of Total 0.9% 18.9% 80.3% 100.0%
199 Page 4 of 7 J Med Syst (2017) 41: 199

Table 2 Cross-Tab of
Completion vs Number of Errors Number of Errors Number Paper Total
in Paper Form
No Errors 1 Error 2 Errors 3 Errors 4 Errors

Completion No Count 0 74 8 0 1 83
Paper % of 0.0% 32.5% 3.5% 0.0% 0.4% 36.4%
Total
Yes Count 45 72 9 7 12 145
% of 19.7% 31.6% 3.9% 3.1% 5.3% 63.6%
Total
Total Count 45 146 17 7 13 228
% of 19.7% 64.0% 7.5% 3.1% 5.7% 100.0%
Total

the CT department reached 35 out of 83 incomplete forms
where the MRI department reported only 15 cases.
CT forms were found to execute fewer number of cases com-
pared to the MRI department. The result of this investigation
shows a significant association between completion and error
category χ2 (5) = 18.576, p < 0.002, Fisher’s Exact Test =
18.606, p < 0.002.
Likewise, Figure 1 provides a graphical view of the com-
pletion levels among all radiological departments included
during the study.
Lastly, the result of studying the associations among the
variables in different combinations using Chi-Square analysis
shows significant associations among variables in dissimilar
combinations as all p values are less than 0.002. This suggest
that there are significant associations among completion, leg-
ibility, number of errors and category of errors in forms gen-
erated by radiology departments.
Discussion
The results of the study show a significant association of com-
pletion with the type of record and legibility variables. Paper-
based forms appear to have limited capabilities in terms of
completion of the required fields and legibility of written in-
formation that tend to have a greater error rate compared to
electronic-based forms. Moreover, all collected electronic
forms have shown no errors and legibility ranging between
readable and understandable information, where paper forms
have shown a high rate of unclear information.
Based on our results, switching to electronic-based forms
will improve the completeness and legibility of referral forms
with reduced error rates. According to a study conducted by
Hoyt et al., shifting to an electronic system will improve the
quality of care provision and enhance patient safety. Also the
study suggested that an electronic-based system will improve
the legibility level of information entered into the clinical his-
tory field, improve accessibility to information and decrease
duplication of data. [8]. This finding is similar to what we
discovered in this study where electronic paper-based forms
were more complete and legible, which can have positive
impacts on clinical workflows and potentially improve the
accessibility to valid data and reporting. This work evaluated
both paper-based and electronic-based forms by focusing on
completeness, legibility of required fields (whether
Understandable, Readable or Unclear), issuing department ad-
herence to complete all required fields and alteration of re-
corded information. The results of this study show that
electronic-based forms have higher percentages of form com-
pletion compared to paper-based forms. Moreover, using
electronic-based forms enhanced field legibility and
Table 4 Cross-tabulation of Legibility vs Type of Record

Variables Type of Record Total

Table 3 Cross-Tab of Completion vs Type of record
Paper Electronic
Variables Type of Record Total
Legibility Understandable Count 2 94 96
Paper Electronic
% of Total 0.4% 20.6% 21.1%
Completion No Count 83 0 83 Readable Count 43 134 177
% of Total 18.2% 0.0% 18.2% % of Total 9.4% 29.4% 38.8%
Yes Count 145 228 373 Unclear Count 183 0 183
% of Total 31.8% 50.0% 81.8% % of Total 40.1% 0.0% 40.1%
Total Count 228 228 456 Total Count 228 228 456
% of Total 50.0% 50.0% 100.0% % of Total 50.0% 50.0% 100.0%
J Med Syst (2017) 41: 199 Page 5 of 7 199

Table 5 Cross-tabulation of
Legibility vs Completion Variable Legibility Total

Understandable Readable Unclear

Completion No Count 0 0 83 83
% of Total 0.0% 0.0% 18.2% 18.2%
Yes Count 96 177 100 373
% of Total 21.1% 38.8% 21.9% 81.8%
Total Count 96 177 183 456
% of Total 21.1% 38.8% 40.1% 100.0%

eliminated the percentage of radiological requests with un-
clear information. The results also demonstrate a significant
level of radiology staff adherence to complete the referral form.
With regards to the alteration of written information on paper
forms, some forms appeared in the selected sample, during the
data analysis process, to have two different handwriting styles
or having two different physicians filling the same form. This
shows that paper forms have little control to prevent form al-
teration during issuance, transferring or receiving the request.
Unlike the electronic forms, referral requests are only issued
through the system and sent electronically to the radiology
department and received by radiology staff which minimizes
the possibility of unauthorized alteration in the form.Moreover,
electronic-based systems were less prone to error when com-
pared to paper based forms, which may result in less medical
errors and improved patient care.
recommend carrying out further studies evaluating the
impact of keystroke errors on patient care.
Limitations
There are a number of limitations that need mentioning. First,
we only used a limited sample of radiological forms in the study
during a very short time-frame. Expanding the time frame and
taking samples from other periods may lead to new insights.
Second, we conducted the study with less than one year post-
implementation of the new electronic medical record system.
This may have biased the results of the study as the new
workflows introduced by the system may not have been adopted
fully by users. Third, as non-experts in the field of radiology, the
researchers may have overlooked some key information points,
which may have also biased the results of this work.

Future work
During this study, completion and legibility of medical
imaging referrals were addressed and were found that
electronic-based referral systems were more legible and
complete than paper-based systems. Electronic based
forms force users to complete all the required elements
in the form. However, some fields are typed in through
the keyboard which can lead to typing errors. We
Contributions to knowledge
Most studies in the literature concentrate on the completion of
paper forms versus electronic forms or highlight the accuracy
of information filled in both types of forms (electronic vs
paper). This study focused on the legibility and error rates of
information filled in the electronic and paper forms which has
not been done before in Saudi Arabia. The information from

Table 6 Cross-tabulation of Error Category vs Completion

Variables Error Category Total

X-Ray CT Scan MRI Ultrasound Mammo Angio Graph Nuclear Medicine

Completion No Count 9 35 15 19 3 2 83
% Total 2.0% 7.7% 3.3% 4.2% 0.7% 0.4% 18.2%
Yes Count 78 85 119 75 11 5 373
% Total 17.1% 18.6% 26.1% 16.4% 2.4% 1.1% 81.8%
Total Count 87 120 134 94 14 7 456
% Total 19.1% 26.3% 29.4% 20.6% 3.1% 1.5% 100.0%
199 Page 6 of 7
Fig. 1 Form Completion among
radiological departments
this study will help future researchers audit the impact of
moving from paper to electronic-based referral forms, espe-
cially, referral forms related to medical imaging.
Conclusion
In conclusion, electronic-based systems force users to
complete forms thus improving the legibility and reducing
form entry errors. The implementation of an electronic-
based system will help improve the completeness and leg-
ibility of forms while reducing keystroke errors. Overall,
our research shows that although electronic-based referral
forms are more complete and more legible, more work is
needed to understand the impact of the design of
electronic-based referral forms and the impact they may
have on completeness, legibility, and keystroke errors.
Future work should investigate the role of usability design
on completenss and legibility. Although our work shows
interesting insights, it would be difficult to generalize the
findings of our work to other hospitals within Saudi Arabia
or outside, as previous studies have suggested [12–17]. We
say that out of the realization that each system is built
differently, errors and completness cannot be controlled,
while legibility of the writing can be if a keyboard is
used. As we move from a paper-based to a more electronic
based enviornment in Saudi Arabia, more work is needed
to study the impacts of system implementation on com-
pleteness of forms and errors.
J Med Syst (2017) 41: 199
Acknowledgements We would like to thank King Abdullah
International Medical Research Centre (KAIMRC) for providing the re-
search funds to complete this work.
References
1. Tan, G.J., Lim, E.L., and How, C.H., A guide to requesting outpa-
tient and emergency radiographs. Singapore medical journal.
53(7):423–426, 2012.
2. Afolabi, O.A., Fadare, J.O., and Essien, E.M., Audit of Completion
of Radiology Request Form in a Nigerian Specialist Hospital.
Annals of Ibadan Postgraduate Medicine. 10(2):48–52, 2013.
3. Oswal, D., Sapherson, D., and Rehman, A.A., study of adequacy of
completion of radiology request forms. Radiography. 15(3):209–
213, 2009.
4. Troude, P., Dozol, A., Soyer, P., Girard, D., Martinez, F., Montagne,
B., and Segouin, C., Improvement of radiology requisition.
Diagnostic and Interventional Imaging. 95(1):69–75, 2014.
5. Triantopoulou, C., Tsalafoutas, I., Maniatis, P., Papavdis, D., Raios,
G., Siafas, I., et al., Analysis of radiological examination request
forms in conjunction with justification of X-ray exposures.
European Journal of Radiology. 53(2):306–311, 2005.
6. Shah, C.C., Linam, L., and Greenberg, S.B., Inappropriate and
cloned clinical histories on radiology request forms for sick chil-
dren. Pediatric radiology. 43(10):1267–1272, 2013.
7. Jans, L.B., Bosmans, J.M., Verstraete, K.L., and Achten, R.,
Optimizing communication between the radiologist and the general
practitioner. JBR-BTR: Organe de la Societe royale belge de
radiologie (SRBR)= orgaan van de Koninklijke Belgische
Vereniging voor Radiologie (KBVR). 96(6):388–390, 2012.
8. Hoyt, R. E., and Yoshihashi, A. K. Health Informatics: Practical
guide for healthcare and information technology professionals.
Lulu.com, 2014.
9 Arab News. MOH steps up its ehealth services. http://www.
arabnews.com/food-health/news/863416, 2016.
J Med Syst (2017) 41: 199
10 Ministry of National Guard Health Affairs. http://ngha.med.sa/
English/MedicalCities/Pages/default.aspx. Accessed 1 May 2017.
11 Ministry of National Guard Health Affairs. Hospital Information
System BBESTCare^. http://ngha.med.sa/English/AboutNGHA/
bestcare/Pages/Default.aspx. Accessed 1 May 2017.
12 Dogether, M.A., Muallem, Y.A., Househ, M., Saddik, B., Khalifa,
M., The impact of automating laboratory request forms on the qual-
ity of healthcare services. J. Infect. Public Health 9(6):749–756,
2016.
13 Househ, M.S., Borycki, E.M., Rohrer, W.M., Kushniruk, A.W.,
Developing a framework for meaningful use of personal health re-
cords (PHRs). Health Policy and Technology 3(4):272–280, 2014.
14 Mominah, M.A., Yunus, F., and Househ, M.S., A case study on the
impacts of computerized provider order entry (CPOE) system on
hospital clinical workflow. In ICIMTH 190:207-209, 2013.
Page 7 of 7 199
15 Alsulame, K., Khalifa, M. and Househ, M.S., eHealth in Saudi
Arabia: Current Trends, Challenges and Recommendations. In
ICIMTH. 213:233-236, 2015.
16 Al Dogether, M., Al Muallem, Y., Al Assaf, R., Al Ateeq, A., Al
Moammary, E., Al Ghamdi, H. and Househ, M.S., The
Implementation Experiences of an Endoscopy Information System
(EIS) on the Improvement of Workflow Processes in a Saudi
Endoscopy Department. In ICIMTH. 213 29-32, 2015.
17 Alasmary, M., El Metwally, A., Househ, M., The Association be-
tween Computer Literacy and Training on Clinical Productivity and
User Satisfaction in Using the Electronic Medical Record in Saudi
Arabia. J. Med. Syst. 38(8), https://doi.org/10.1007/s10916-014-
0069-2, 2014.
Reproduced with permission of copyright owner. Further reproduction
prohibited without permission.