Otolaryngol Clin N Am

40 (2007) 1237–1259

Assessment of Surgical Competency

Terance T. Tsue, MD, FACSa,*,
James W. Dugan, PhDb, Brian Burkey, MDc
a
Department of Otolaryngology–Head and Neck Surgery, University of Kansas School
of Medicine, 3901 Rainbow Boulevard, Mail Stop #3010, Kansas City, KS 66160, USA
b
Counseling and Educational Support Services, University of Kansas School of Medicine,
3901 Rainbow Boulevard, Kansas City, KS 66160, USA
c
Department of Otolaryngology–Head and Neck Surgery, Vanderbilt University Medical
Center, 7209 Medical Center East-South Tower, 1215 21st Avenue South, Nashville,
TN 37232-8605, USA

Assessment of surgical competency in training and practice is an impor-

tant issue confronting modern medicine. Even beyond the desire to educate
competent otolaryngology–head and neck surgery (OTOHNS) residents,
a focus on outcomes assessment has spread to other aspects of the health
care system, influencing how and where medicine is practiced and how phy-
sicians are compensated. The Joint Commission of Accreditation of Health
Care Organizations has incorporated requirements relating to competency
in the general competencies [1]. Assessment of medical staff and resident
staff competency is now an important focus of the hospital accreditation
process. Such measures are becoming increasingly visible in licensing and
credentialing procedures as well. Practitioner ‘‘performance’’ measures are
being developed through efforts by the Centers for Medicare and Medicaid
Services Pay-for-Performance initiative. Providers will receive payment dif-
ferential incentives to assess patient care quality and use that information to
improve overall patient care. Commercial insurance companies are investi-
gating similar approaches.
This increasing focus on competency has also been embraced in the lay
community. Patients are more medically educated owing to the use of the
Internet. Public confidence in technical performance in surgery has come un-
der increased scrutiny, exacerbated by various highly publicized cases that
suggested poor outcomes were the result of inadequate technical proficiency
[2]. The increasing importance on a complication-free learning environment

* Corresponding author.
E-mail address: ttsue@kumc.edu (T.T. Tsue).

0030-6665/07/$ - see front matter Ó 2007 Elsevier Inc. All rights reserved.
doi:10.1016/j.otc.2007.07.005 oto.theclinics.com
1238 TSUE et al

was emphasized by the British Royal Colleges of Surgery in the reply to the
General Medical Council’s determination on the Bristol case, wherein they
state: ‘‘there should be no learning curve as far as patient safety is con-
cerned’’ [3]. The malpractice crisis has also spread to include some suits al-
leging residency program ‘‘educational malpractice’’ and responsibility of
program directors for purported resident graduate negligence [4]. Reprisal
litigation from residents dissatisfied with or terminated from their training
programs also beckons the need for validated objective assessments during
training.
This focus on outcomes has also spread into the way residents are taught,
evaluated, and certified. In 2001, the Accreditation Council for Graduate
Medical Education (ACGME) initiated its Outcomes Project [5]. This
long-term initiative focuses on the educational outcomes of residency train-
ing programs rather than the previous emphasis on the ‘‘potential’’ for a pro-
gram to educate its residents through an organized curriculum and
compliance with specified program requirements. The ACGME accredita-
tion process has shifted from verifying program components to verifying
the program’s educational product. At a minimum, programs are mandated
to use assessments of their educational outcomes to continuously improve
their educational product: a resident graduate competent in all six of the
ACGME general competencies. This programmatic feedback process in-
volves many levels of assessment beyond measuring just resident knowledge,
skills, and attitudes; it also may require evaluating graduate, faculty, pa-
tient, departmental, and institutional outcomes. Residency programs are ex-
pected not only to consider aggregate learner performance data (eg,
percentile ranking on in-training exams, first-attempt certification exam
pass rate), but also external program performance measures. These ‘‘exter-
nal indicators’’ are not yet defined for OTOHNS programs, but can include
metrics like clinical quality measures, patient survey results, and complica-
tion rates. Although it is expected that such changes to residency program
evaluation will be a dynamic, evolving process, documentation of the feed-
back loop will be necessary for both program and institutional accredita-
tion. Finally, similar information will likely be required in the future as
a component of the maintenance of certification process developed by the
American Board of Otolaryngology (ABOto). The thrust toward board
maintenance of certification requirements is congruent with the sentiments
for continued measurement of physician competency. Although the
ACGME has placed the focus on educational outcomes and not clinical out-
comes, there is obvious significant overlap.
All of these interrelated forces, both public and within the medical profes-
sion itself, have highlighted the need for valid assessments of trainees’ compe-
tency as surgical specialists. Although the thorough evaluation of competency
in all areas of a physician’s practice by a feasible, reliable, and valid assess-
ment process is important, at the core of an OTOHNS practice is surgical
competency. Surgical competency obviously involves more than just doing
 ASSESSMENT OF SURGICAL COMPETENCY 1239

the operation. Darzi and Mackay [6] describe the four essential components
or framework of surgical care in which a surgeon must be competent: diagnos-
tic ability, treatment plan formulation, technical skill performance, and
postoperative care. All of these components involve competency in cognitive
and personality skills such as decision making/judgment, knowledge, commu-
nication, teamwork, and leadership [7]. Thus, surgical competency requires
competency in all of the six ACGME general competencies and not just pa-
tient care. Technical skill performance, of all areas of surgical care, has
been the most challenging in terms of objective assessment. Within OTOHNS
itselfdlet alone any other surgical fielddthese skills remain variable in terms
of their nature and complexity. The current and potential future solutions to
the challenge of evaluating this component of surgical competency remain the
focus of this article.

High-stakes assessments in other fields

The field of surgery depends on a foundation of technical skill that is en-
hanced by technology, knowledge, and judgment. Other fields also depend
on this foundation as a basis and have potentially high-stakes outcomes
that can affect life, limb, or property. Such fields are well ahead of medicine
with regard to expecting certification of core skill competency. High-risk in-
dustries include aviation, nuclear power, chemical manufacturing, and off-
shore oil drilling [8]. These industries employ a number of methods to
assess competency that include objective observational assessment by a su-
pervisor/trainer (the most common), written and oral examinations, elec-
tronic simulations, and behavior marker systems that rate trainee
behavior during a simulation. All of these methods of competency assess-
ment used in the high-risk industries have been adapted to some degree in
the assessment of surgical competency. However, what is distinctive in
high-risk industries, especially aviation and nuclear power, is that technical
competency is regularly assessed in simulations of both routine and emer-
gency situations.
In assessing technical competency in these high-stakes industries, declara-
tive knowledge or knowing the technical skill is the first part of this assess-
ment. Subsequently, knowing how to put that technical skill into safe
practice in a simulation must be demonstrated to achieve competency. Con-
sequently, these industries place significant focus on the evaluation of the im-
pact of ‘‘soft,’’ or nontechnical, skills on the performance of technical skills.
Examples of these nontechnical skills are leadership, teamwork, assertiveness,
communication, and situational awareness. These nontechnical skills have
been described as the underpinnings of technical proficiency, and lapses in
these nontechnical skills have often been identified as the root cause of tech-
nical error [9]. A classic example of this is when, in 1988, the crew of the USS
Vincennes, a guided missile cruiser equipped with the latest technology, mis-
takenly shot down an Iranian commercial airline with 290 passengers on
1240 TSUE et al

board. The root-cause analysis of this deadly error identified the crew’s in-
creased stress and fatigue that contributed to poor decision making, commu-
nication, and teamwork [10]. To measure competency in the integration of
nontechnical skills during a technical procedure, behavioral marker systems
have been developed and are used widely in these industries. These assess-
ments allow a qualified trainer to identify and rate the behavior of the trainee
during a simulation. For almost 25 years, the aviation industry has used a be-
havioral marker system called crew resource management, which has been
shown to significantly improve aviation safety.
In the last few years, two behavioral marker systems have been developed
for training residents in anesthesiology and surgery: Anesthetists’ Non-
Technical Skills and Surgeons Non-Technical Skills [11,12]. These behav-
ioral marker systems identify elements of behavior such as communication,
teamwork, situational awareness, and decision making. Currently, the use of
simulations and behavior marker systems, though certainly demonstrating
their value in the aviation and nuclear power industries, presents consider-
able cost and time challenges for inclusion in residency training.

Assessment: ideal and reality

Assessment is defined as the ‘‘process of collecting, synthesizing, and in-
terpreting information to aid decision-making’’ [13]. In medical training, as-
sessments are used to measure progress of residents toward attainment of
the program goals and objectives (‘‘formative’’ evaluation), which ultimately
lead to a level of competency and beyond for program-defined outcomes
(‘‘summative’’ evaluation) [14]. The assessment process includes not only
the ‘‘test’’ or collection of data, but also applicable analysis and interpreta-
tion of the data, followed by communication and feedback to both evalua-
tor and learner. This ‘‘feedback loop’’ should improve the educational
efficiency for the learner, including directing him or her to priority areas
of self-study and curricular focus. For the teacher, feedback should prompt
pinpoint refinement of the curriculum. This process should result in a con-
stantly improving education product or outcome.
The ideal assessment method should be feasible, requiring minimal cost,
time, and effort by both learner and evaluator to complete and analyze. The
ideal tool should be simple, be useful for all levels of training and expertise,
and assess all areas of the field. Objectivity and anonymity would be pro-
vided as well as assessment reliability (consistency or reproducibility of mea-
surement) and validity (instrument truly measures what it is supposed to
measure) in a standardized and reproducible testing environment. The as-
sessment metric would be quantifiable, with competency lying in the middle
of the scaled score. Finally, the ideal assessment would provide immediate
informative feedback to direct both specific individual and programmatic
changes.
 ASSESSMENT OF SURGICAL COMPETENCY 1241

In reality, there is no ‘‘ideal’’ assessment that fulfills all of the above re-
quirements. There is no assessment that evaluates all of the objectives or
outcomes that need to be measured. Thus, difficult choices must be made
about what can realistically be assessed. Progression through the OTOHNS
residency has classically been based on the apprenticeship model, relying on
the traditional graded-responsibility experienced-based model. The main
feature of this model is a teacher-centered approach based on loosely struc-
tured, one-on-one supervised situations where principles are taught and the
learner is assessed on the basis of the teacher’s interpretation of current
standards of practice [15]. This traditional approach has helped to exacer-
bate the current ‘‘reality’’ of the limitations of today’s surgical competency
assessment techniques. Progress and eventual graduation rely on subjective
evaluations by faculty. This requires accurate evaluator recall of past inter-
mittent and widely varied events and generally stems from an overall ‘‘ge-
stalt’’ rather than any objective measures. Anonymity remains difficult for
OTOHNS programs because of their smaller size, making concerns or
threat of retaliation a reality. The number of faculty evaluators on a given
rotation is even smaller, and each has a potentially different definition of
competency. Additionally, the influence of resident duty-hour limitations,
decreased clinical reimbursements, a continuing trend toward superspecial-
ization, and a focus on increasing health care resource efficiency has also
hampered progress toward an ‘‘ideal’’ assessment system. These influences
decrease the amount of educational resources availablednamely moneyd
and faculty and student time. The comparably rapid expansion of knowl-
edge, technology, and techniques within OTOHNS not only tends to use
these already limited resources at a faster rate, but also provides a moving
target in terms of what needs to be evaluated.

Making it feasible
Incorporating a feasible assessment system, even given the above-de-
scribed constraints and challenges, is a realistic and necessary goal. Box 1
summarizes some general steps that can help with the implementation of
an efficient evaluation process. Incorporating these steps within an otolaryn-
gology residency program is discussed below.
First, delineate the minimum requirements needed. The ACGME Com-
mon Program requirements delineate the necessary minimum assessment
methods for residency programs, but these minimums may be influenced
by the JCAHO and local credentialing requirements as well. Current recom-
mendations include the use of an end-of-rotation global assessment tool and
at least one other method. As most of the technical skill component of sur-
gical competency falls under the patient care competency, recommendations
suggest a focused assessment method such as direct observation and concur-
rent evaluation. Use of multiple assessment methods to measure technical
1242 TSUE et al

Box 1. Incorporating feasible assessments

1. Know the minimum assessment requirements
2. Identify available resources and limitations
3. Adopt or adapt currently used or available assessment
methods
4. Involve multiple evaluators and assess at performance
milestones
5. Educate both the learner and the evaluator
6. Use the latest electronic technology available

skill reduces the subjectivity of the process as well as overcoming the differ-
ent limitations inherent in each particular method.
Second, it is important to identify what resources are available for the as-
sessment system. Limitations on learner and evaluator time as well as avail-
able personnel, equipment, facilities, and funds that can be dedicated to the
activity need to be determined. Coordination across surgical subspecialties is
an excellent way to facilitate availability of more resource-intensive assess-
ment methods. Even mobility and sharing of techniques between OTOHNS
programs are possible and certainly would add to a particular method’s
attractiveness.
Third, use and/or adapt assessment methods currently in use. This not
only reduces the sense of ‘‘change’’ by evaluator and learner, but also saves
significant implementation time and effort. Additionally, there may be
proven external performance measures that are currently used by the univer-
sity or department that can be easily adapted for learner assessment (eg,
quality control measures, Press Ganey patient surveys). Additionally,
proven assessment methodologies from other fields that also require compe-
tency evaluation of high-stakes skills are potential resources for adoption
(see below).
Fourth, use multiple evaluators and perform the assessment at multiple
performance milestones throughout the training program. Engaging differ-
ent evaluators spreads out the responsibility and should not influence the
outcome of a reliable assessment method. Focusing on specific program
milestones, and spreading the assessments out over the 5-year training pe-
riod, should improve the usefulness of the evaluation outcomes by matching
a specific skill with its assessment. If assessments can be combined with the
learning activity, the efficiency of the process should be even higher. This
can even extend to involving learners in the development and application
of the assessment process. Learners who have achieved a level of compe-
tency can assess those still progressing toward that goal. This interaction
should educationally benefit both parties. In contrast, routine assessment
that is temporally based, such as after each rotation or academic year,
and not related to a specific milestone level, can dilute this feedback efficacy.
 ASSESSMENT OF SURGICAL COMPETENCY 1243

Fifth, educate both the evaluator and the learner about the assessment tools
and processes. Providing specific objective definitions of assessment levels,
such as what ‘‘competent’’ or ‘‘satisfactory’’ means, should improve the use-
fulness and applicability of the tool across learners. Learners are then mea-
sured against a known scale rather than against each other. This can also
allow more self-assessment by the resident, as the objectives are well known
and defined, potentially guiding more independent study and practice.
Sixth, use the latest technology available to administer the assessment as
well as collect and analyze evaluation results. Electronically administered
tools are easier, especially for the technologically advanced, and can be ac-
cessed from nearly anywhere when an evaluator has time to complete the
process. Less completion time required should increase compliance while si-
multaneously allowing faster analysis and shorter time to feedback.

Closing the feedback loop

The quality of an assessment is only as good as its ability to effect change.
Closing the feedback loop in a constructive manner that prompts both
learner and teacher to improve is difficult. The nature of the output of the
planned assessment method needs to be an important factor in its selection
as an evaluation tool. The output data need to be easily analyzed. Ten thou-
sand data points are not readily understandable in raw form, and it can be
time consuming to analyze the data down statistically. Assessment results
that are able to point to specific skill components (ie, ‘‘competent at recur-
rent laryngeal nerve identification’’) are more useful than ones that give gen-
eralized results (ie, ‘‘competent at thyroidectomy’’). A quantitative measure
eases comparisons and allows linear comparison in a longitudinal fashion
throughout the resident’s tenure. This measure also allows peer compari-
sons. A graph can be helpful to visually demonstrate performance trends
over time. As stated previously, aiming for a competency level in the middle
of the quantitative measure is most useful.
Performance feedback (formative evaluation) is an important component
of the evaluation process. Confidential and organized meetings between the
program director and resident are most effective, especially those done in
a timely fashion after the assessment. Even if the feedback is delayed until
the requisite semi-annual program director meetings, a simple, focused,
and understandable presentation to the resident can still yield change. Those
residents behind their peers may need more frequent meetings, and feedback
should be adjusted to their needs. Prioritizing specific results (ie, ‘‘top three’’
and ‘‘bottom three’’ lists) can help a learner remember and focus on where
effort needs to be applied. Recognizing positive improvement is beneficial
for many obvious reasons. Residents should also be given the opportunity
to provide feedback on the assessment methods themselves.
An essential component of feedback for the program director is provid-
ing guidance to areas of need. Priority areas, should be accompanied by
1244 TSUE et al

suggested solutions and approaches. Several approaches should be dis-

cussed and documented, as learning styles can differ between residents.
An associated timeline for improvement and expectations of future assess-
ment results should also be negotiated. Key to this success is mentorship
and accountability between assessments. Mutually selecting a faculty
member other than the program director to provide nonthreatening over-
sight, daily guidance, and objective progress reports is paramount for
those with significant need for improvement. Some evaluation results
that are a widespread problem may not only require individual feedback,
but most likely will also require tangible programmatic and curricular
change.

Available assessment methods

Many skills assessments have been or are potentially adaptable to the as-
sessment of the OTOHNS resident. However, data on assessment tool reli-
ability and validity in the OTOHNS or in other surgical fields are sparse
and/or premature. Some methods are more established in other fields, and
some are only research tools at this time. Written examinations, such as
the Board Certification and In-Training Examination, are good assessors
of surgical knowledge but are not intended to measure technical proficiency.
These evaluation methods do not necessarily correlate with technical skill
and operative performance [16]. Thus, other methods must be employed
to measure all aspects of surgical competency.
Surgical technical proficiency can be broken down into dexterity, spatial
orientation, and operative flow [17]. Dexterity further involves the psycho-
motor aspects of tasks, tissue handling/respect for tissue, and economy/flu-
ency of movements [6]. Objective assessments should include these
components to fully measure competency. The following methods are cur-
rently available assessments and are foci of contemporary practice and
study:
1. Direct observation
2. Resident case log reports/key indicator operative procedures (KIP)
3. End-of-rotation global assessments
4. Objective structured assessment of technical skill (OSATS)
5. Final product analysis
6. Hand motion analysis
7. Simulation
8. Possible new techniques

Direct observation
This method involves a senior colleague, usually a faculty member, ob-
serving a learner during a surgical task. The observer then documents an
 ASSESSMENT OF SURGICAL COMPETENCY 1245

opinion on competency based on these observations. Feedback during the

observation is possible and can be immediate and effective. This process oc-
curs daily in every residency program. Unfortunately, unlike most of the
methods described below, the judgments derived from simple direct observa-
tion rely on unsystematic and unstructured observations. Without judgment
criteria, the assessment can be vulnerable to many biases, including the fac-
ulty evaluator’s own memory. The assessment is usually relayed to the pro-
gram director in a retrospective and generalized manner or through a signed
resident procedure competency checklist. As different faculty have varied
definitions of competency, this tends to yield unreliable and imprecise as-
sessments [18]. However, physicians continue to rely on this process as a gen-
eral method, as they make their judgments from faculty input and personal
interactions, and then sign Board certification exam application materials,
residency graduation certificates, and hospital and insurance company cre-
dentialing forms based on this information. Hopefully, incorporating
some of the more reliable and valid methods described below will help phy-
sicians to make more objective and accurate assessments.

Resident case log reports/key indicator operative procedures

Another universally employed measure of progress toward surgical com-
petency is the operative case log system. The cornerstone of surgical training
has been one of apprenticeship, with graduation and competency believed to
be resulting from a well-documented exposure and participatory experience
in a graded, supervised environment. Not only is exposure important in
building competency, but this system also requires progressive levels of re-
sponsibility. As defined by the ACGME, this progression starts as assistant
surgeon, through resident surgeon, to (finally) resident supervisor surgeon.
The implicit understanding is that progress toward and through competency
with a given procedure follows these stages. Competency is requisite before
moving to the supervisory level. Although all operative experiences are re-
corded by each resident, the ACGME has focused on select representative
procedures, referred to as Key Indicator Procedures (KIP).
Classically, these numbers have been used by the Residency Review Com-
mittee (RRC) to determine a program’s ability to provide their residents ad-
equate experience in the various OTOHNS procedures. Acceptable levels
have been determined by national percentile ranking for program graduates.
Unfortunately, the use of percentiles always results in some programs falling
below the desired threshold for experience. The electronically derived KIP
report, available to both residents and program directors, helps the learner
to focus on specific procedural and subspecialty areas of needed concentra-
tion. It also gives useful information as to the appropriate and balanced
graded progression of operative experience, from assistant surgeon through
resident supervisor. Although only national graduate means are available,
keeping annual Post-Graduate Year KIP experience averages allows
1246 TSUE et al

comparisons to be made to a resident’s predecessors in addition to a resi-

dent’s current peers. Unfortunately, changes in present-day medicine
threaten the usefulness of this competency measure. These changes include
duty-hour limitations, bottom-line–driven emphasis on operative time effi-
ciency, and a desire to mitigate against medical errors. This has noticeably
limited the level of resident autonomy in the operating room and probably
hindered the efficiency of his or her technical development.
The assumption essential to the use of operative log numbers as an assess-
ment tool is that ‘‘adequate’’ experience results in surgical competency.
Carr, through a survey of OTOHNS program directors, identified 16 proce-
dures in which competency is achievable in the PGY-1 through PGY-5
levels [19]. In most of these procedures, the graduating residents’ mean op-
erative experience was higher than the number thought necessary by the sur-
veyed program directors to achieve competency. This assumption of
jumping from experience to competency is hindered by many factors. Expe-
rience numbers do indicate an educational environment adequate or inade-
quate for the resident to ‘‘potentially’’ obtain competency. However, this
assessment method lacks validity as it does not record quality of the opera-
tive experience [20]. Technically completing a procedure as resident surgeon
or even as a resident supervisor in a supervised environment does not ensure
the ability to perform that procedure independently. The levels of responsi-
bility are subjectively recorded by the residents themselves, not the supervis-
ing faculty teacher. Despite rigid ACGME definitions, interpretations still
vary. Self-assessment by surgeons is notoriously variable and generally over-
states ability [21]. Also, adequate experience that results in competency will
differ for each resident, because learning rates differ between residents and
for different procedure types (eg, open versus microscopic). This variability
is also affected by supervision and teacher nurturing bias.
Time taken for a procedure has also been used as a measure of surgical
performance. However, time also does not necessarily reflect the quality
of the procedure and is generally unreliable due to the influence of many un-
controllable factors [22]. Morbidity and mortality data are often implied as
surrogate markers of surgical performance outcome, but are heavily influ-
enced by patient factors and probably do not reflect an accurate measure
of surgical competency.

End-of-rotation global assessments

These assessments are also common and are frequently divided into sec-
tions corresponding to each of the six ACGME core competencies. Assess-
ment of resident surgical technical skill applicably falls into the patient care
core competency. A faculty rater scores a specific resident on very broad as-
pects of surgical skill, ranging from surgical efficiency and progression to
handling of tissues. These scores generally are on a Likert-like scale using
a scaling method to measure either a degree of positive or negative response
 ASSESSMENT OF SURGICAL COMPETENCY 1247

to a statement. Some global assessments give qualitative performance state-

ment examples to correspond with the numerical scale, whereas others, like
the sample available on the ACGME Web site, provide statements for just
the extreme scores [23]. Satisfactory or competency is the middle score;writ-
ten comments are also allowed. These ratings are generally completed at the
end of a specific rotation, which could be even a few months in duration.
Evaluator scores are retrospectively derived from impressions and other
sources, including memory of specific interactions or clinical outcomes.
These global assessments can provide a quantitative summary of overall
surgical skill that can be assessed longitudinally for a given resident and
also used for comparison between peer residents. The results of many raters
can be averaged as well. The forms are easy to construct and readily made
available electronically to shorten completion and analysis time. Unfortu-
nately, the large number of numerical data points over a resident’s tenure
can rapidly become untenable, and is not as useful in a formative fashion to
help guide a given resident toward a focal area of improvement. Additionally,
these scales are subject to distortion from several causes, including central ten-
dency bias (avoiding extreme response categories), acquiescence bias (agree-
ing with the presented statement), and/or social desirability bias (portrayal
of faculty rater’s specific rotation in a favorable light). These distortions are
even more significant and the tool less reliable with untrained evaluators.
The ACGME rates this method as only a ‘‘potentially applicable method.’’
‘‘Three Hundred and sixty degree’’ (360 ) global assessments, with com-
petency evaluations and comments from staff who work side by side with the
learner, can provide a more real-life assessment but are subject to the same
limitations described above. As these assessors are usually not faculty (eg,
scrub nurse), and thus not fully competent performers of the skill them-
selves, their expertise in assessing the learner and his or her biases usually
focuses only on a particular aspect of the assessed skill. This assessment
can still be useful and provide the basis or confirmation of informative feed-
back. The ACGME does rate this method a ‘‘next best method.’’

Objective structured assessment of technical skill

The OSATS technique was developed by Reznick and colleagues [24] for
general surgery use and is based on the original objective structured clinical
examination (OSCE) method. The OSCE is increasingly used in medical
schools, National Board of Medical Examiners licensing exams, and also
many international certification and licensure boards [25]. The OSATS are
being considered as a standard part of many board certification examina-
tions to demonstrate technical competency [26]. During an OSATS, each
learner rotates through a series of self-contained stations within a limited
predetermined time. Each station is composed of a standardized surgical
task (eg, laceration repair), and participants are assessed by the same trained
observer in a standardized fashion using objective criteria. Thus, examiners
1248 TSUE et al

are observers rather than interpreters of behavior, thereby minimizing the

subjectivity of the evaluation process. The use of simulated models allows
standardization and avoidance of the problem of finding adequate real pa-
tients. Cadaver sections can also be used, and Dailey and colleagues [27] de-
scribed the use of laryngeal specimen stations to practice and assess both
open and endoscopic laryngeal procedures. Reznick assessed the learners
by using both a valid and reliable checklist and a global scoring sheet.
The checklist is a series of 10–30 longitudinal ‘‘yes’’ or ‘‘no’’ items based
on the specific task being assessed. This list includes the essential compo-
nents for an ideally performed operation, and aims to reduce subjectivity
of an evaluator’s specific experience. According to the ACGME Table of
Suggested Best Methods for Evaluation, this checklist assessment is one
of the most desirable methods of evaluating procedural skill. The disadvan-
tages of the checklist method include the inability of the examiner to indi-
cate that a particular task on the checklist was performed well but at an
inappropriate stage. The global scoring sheet includes 5–8 overall perfor-
mance measures, such as ‘‘flow of operation,’’ ‘‘instrument handling,’’ and
‘‘technique familiarity,’’ that are scored from 1 (poor) to 5 (excellent). As
this global rating is not task specific, it has broader applicability, and has
generally been shown to be a more effective discriminator than the checklist
[24]. A separate performance score is derived for each station, and scores are
generally combined across tasks to determine a pass/fail assessment. Several
stations are recommended to provide a reliable performance measurement.
The OSATS are useful in assessing technical skills in terms of knowledge
and dexterity, but they cannot assess surgical judgment as easily. Also,
checklists are task specific and therefore must be developed and validated
for each task. Global rating forms, though more flexible, also tend to
have a poorer faculty completion rate [28]. The OSATS are difficult to de-
velop and administer due to their resource intensiveness (equipment, time,
and manpower), and tend to be more useful for assessing simpler tasks
and thus for assessing more junior trainees [29]. Limited resources in
OTOHNS programs, which tend to be on the smaller size, can limit OSATS
availability. However, this form of evaluation can be more cost effective if
resources and expertise are shared between programs within an academic in-
stitution or a given geographic area.
The OSATS use of inanimate procedural simulation (bench top) assess-
ments has been shown to translate to actual surgical performance in the op-
erating room (OR) [30,31]. So deconstructing an operation into its
component parts can provide a simpler and less costly bench model for sur-
gical competency. This should not replace eventual assessment within the
OR, though. Roberson and colleagues [32] developed and validated an
OSAT-based instrument to measure tonsillectomy performance. Their in-
strument was shown to be both reliable and valid and confirmed that the
global rating evaluation will probably be a more meaningful and readily ap-
plicable tool for OTOHNS. Assessing a videotaped procedure with
 ASSESSMENT OF SURGICAL COMPETENCY 1249

structured criteria can also be used, possibly providing a more favorable en-
vironment for trainee feedback. Such a system allows multiple step-by-step
reviews with many learners and focused identification of specific errors. This
method does have a higher cost in terms of materials and editing time, and
does not necessarily improve on reliability or validity [20,33,34]. In contrast,
by condensing the edited video, evaluator time should be decreased, and vid-
eotaping procedures allows for better learner anonymity, eliminating gen-
der, racial, or seniority biases [35].

Final product analysis

More efficient objective assessments have been proposed that seem to cor-
relate well with the OSATS. Datta proposed the surgical efficiency score
(SES) and snapshot assessment (SS) techniques [36]. The SES combines
evaluation of final skill product quality and hand-motion analysis (see be-
low); the SS uses OSATS scoring of a 2-minute edited video of the task per-
formance. In surgical model task assessment, both the SES and the SS
showed correlation with traditional OSATS evaluations. Szalay also as-
sessed final product quality after the performance of six bench model tasks
[37]. These results also demonstrated construct validity and correlation with
OSATS results. Leak rates and cross-sectional lumen area outcomes after
bench-model vascular anastamoses were significantly correlated with
hand-motion analysis [38]. Using a different approach to final product anal-
ysis, Bann and colleagues [39] studied the ability of trainees to detect simple
surgical errors in models containing purposely made mistakes. This was
a valid predictor of qualitative performance on the same bench tasks. These
results suggest that these less labor- and time-intensive assessments may be
as useful as the OSATS, making surgical skill assessments more feasible.
More research into the quality of these metrics needs to be performed.

Hand-motion analysis
Efficiency and accuracy of hand movements are a trademark of an expe-
rienced surgeon’s dexterity. Hand-motion analysis during a standardized
surgical task is possible using the commercially available Imperial College
Surgical Assessment Device. Through the use of passive trackers on the dor-
sum of each hand while performing a task through a magnetic field, currents
are induced in the trackers that allow hand position to be determined using
Cartesian coordinates. Number of movements, path length, speed of mo-
tion, and time on task can be measured and compared as a valid assessment
of skill during a standardized procedure. Streaming video allows segmental
focus into specific key steps of the observed procedure. These objective mea-
surements have been shown to be an effective index of technical skill in both
endoscopic and open procedures [40–43]. They have also been shown to
have a good concordance with OSATS results [44].
1250 TSUE et al

Hand-motion analysis is generally employed for simpler standardized

tasks that are components of more complex tasks. The method is limited
in assessing more complex tasks in their entirety. An enormous amount of
raw data is generated that can be summarized numerically and visually.
These data need to be analyzed and compared, and although motion anal-
ysis can be used to compare learners and monitor individual progress, hand-
motion analysis is more of a summative than formative assessment tool.
Specifically telling the learner the results provides little information on
how to improve his or her individual scores. The challenge for the program
director is to provide what practice or teaching is specifically needed to im-
prove on a particular skill, and interpreting needed areas of focus from the
provided scores also is a challenge [45]. Additionally, there is a significant
cost and technical expertise need associated with this methodology that
has limited its availability at present, though continued research into the
overall general validity and reliability of this method as an assessment of
surgical competency is increasing.

Simulation
Simulation methods attempt to imitate or resembledbut not duplicated
real-life clinical situations. Like real cases, simulation can provide a number
of options to the learner but in a safe, standardized, and reproducible testing
environment that removes the worry of compromising patient safety or out-
come. Without the inhibiting fear of an irreversible change from an error,
feedback can be immediate, focused, and efficient. A controlled environment
can allow a ‘‘cleaner’’ and more subtle assessment of performance that may
not be possible in real-life situations. Simulation can simultaneously provide
improved learning and assessment, and it affords the learner the opportunity
of repeated practice of a noncompetent area, measuring that progress with an
objective metric. Simulator metrics can provide motivation for the trainee,
and eventually set standards for certification, allowing objective comparison
of trainees both to each other and to a normative value. Simulation must al-
ways be considered an adjunct to competency judgments determined by ex-
pert assessment of observed performance in the OR and by measured
outcome variables from real procedures. Many studies need to be done to
fully validate each simulator, especially in the realm of predictive validity.
Simulation involves a wide range of growing techniques as technology
progresses. Most current simulators are able to distinguish between novice
and competent trainees, but are not yet sophisticated enough to distinguish
between the competent and the expert. Thus, simulators may be more appli-
cable to assessing the early phases of technical learning and skills [46]. Low-
fidelity simulators tend to be mechanical representations of a procedure’s
smallest fundamental components. These are generally organized into timed
stations and require faculty evaluators to observe the learner at each station.
This method forms the core of the above-described OSATS method. Such
 ASSESSMENT OF SURGICAL COMPETENCY 1251

inanimate devices (eg, sewing a Penrose drain laceration) are relatively inex-
pensive and made from readily available products, but still require a signif-
icant time commitment by evaluating faculty. Body part models, which can
further improve the semblance to real life, are expensive. As stated above,
the OSATS method using bench-top models has been shown to correlate
with OR performance, but direct translation to a broader range of surgical
procedures still needs to be proved [30].
Live animal models or human cadavers can further improve the simula-
tion. Live animal models can simulate the ‘‘feel’’ of real surgery, as they are
living tissue, but generally do not reflect the exact anatomic correlate as hu-
man cadaver models can. Cadaver models do lose the feel of real tissue han-
dling, and the temporal bone laboratory is an example of this. The OSATS
using bench-top models shows good correlation with both animal and ca-
daver models, but at a significantly higher overall cost [24,47]. Higher-fidel-
ity simulators include mannequins that incorporate electronics to simulate
normal and pathologic conditions, and have the ability to respond realisti-
cally to interventions by the trainee. Human models with high-performance
simulator technology that go well beyond ‘‘resuscitation Annie’’ are now
available. These are frequently used by anesthesiologists for critical-incident
and team training, but can have obvious direct applications to airway situ-
ations in the OTOHNS as well [25].
Computer-based simulators are becoming increasingly available. Such
‘‘virtual reality’’ simulators also have varying degrees of fidelity. They range
from using abstract graphics that measure partial task skills to full-OR sim-
ulators. Users are able to interact in real time with a three-dimensional com-
puter database through the use of their own senses and skills. The main
challenges of creating more advanced simulators include simulating realistic
surgical interfaces (coupling of instrument to tissue); geometric modeling of
objects and their interactions; and an accurate operative field with advanced
signal processing to simulate such phenomena as texture, light, smoke, and
body fluids [48]. The first virtual reality system used in surgical skills assess-
ment was the Minimally Invasive Surgical Trainer-Virtual Reality, which
was a lower-fidelity system that focused on simulating basic laparoscopic
skills rather than the appearance of the surgical field [49]. It was developed
as a collaboration between surgeons and psychologists who performed
a skills analysis of the laparoscopic cholecystectomy. The Advanced Dundee
Endoscopic Psychomotor Tester is another example that is essentially a com-
puterized system connected to standardized endoscopic equipment [50].
Computers are now better able to replicate not only realistic organ sur-
face image and topography, but also the instrument ‘‘feel’’ a surgeon would
expect from a real patient (realistic haptic fidelity). Rapid advances in tech-
nology, and successful use in certification in many other high-stakes fields
(see above), have made the availability of simulators in measuring surgical
competency a reality. The major thrust of development has been in mini-
mally invasive procedures, especially laparoscopic, because of the more
1252 TSUE et al

straightforward surgical interfaces compared with open surgery (ie, endo-

scopic instruments are levers on fulcrums with fewer degrees of freedom).
Fortunately, simulators that assess basic open procedures such as vascular
anastamosis suturing are now being increasingly studied. Simulators are be-
ing developed and tested in many areas, including the following:
 Ophthalmic surgery
 Colonoscopy
 Arthroscopic surgery
 Limb trauma (musculographics)
 Pericardiocentesis
 Diagnostic peritoneal lavage
 Interventional/endovascular procedures (CathSim Simulator)
 Ultrasound
 Cleft-lip surgery
 Bronchoscopy [51–53]
Additionally, full-OR simulators are being studied to increase the assess-
ment to include other aspects of overall surgical performance, including
hemorrhage control, aseptic technique, and elements of team communica-
tion [54,55].
Virtual reality simulators probably provide the most objective measure-
ment of a technical skill in the most standardized and reproducible environ-
ment currently available. Precision, accuracy, and error metrics are easily
obtained without being labor intensive for the evaluator. Studies have shown
the effectiveness of simulation primarily for lower-level learners, but further
larger-scale validation studies are needed [56]. The main drawback of simula-
tors remains cost, and further study is needed to determine whether this extra
investment is worthwhile. In most cases, low-fidelity simulators may be as ed-
ucationally beneficial in training and assessment, and this may help ultimately
to keep simulators affordable and more generally available. Although less fac-
ulty time is needed, there are increased initial and maintenance costs compared
with other assessment methods. This cost is increased when the simulation is
broadened to include multiple procedure types and increased complexity.
Hopefully, as computing technology improves, costs will fall, making simula-
tors more affordable and readily studied.
Many low-fidelity model simulators have been designed and used to train
and assess procedures in the tympanic membrane, such a tympanocentesis
and myringotomy with pressure equalization tube insertion [57–60]. These
simulators were well in use before any computer models were available. A
virtual reality temporal bone (VR TB) simulator has been developed at
the University of Hamburg (VOXEL-MAN TempoSurg Simulator). High-
resolution images of the temporal bone are used to create computer-gener-
ated images that are modified in real time as the trainee drills and receives
haptic feedback, such as pressure changes depending on the material being
drilled. Glasses provide a three-dimensional image that can color code
 ASSESSMENT OF SURGICAL COMPETENCY 1253

different aspects of temporal bone anatomy. Zirkle and colleagues [40] stud-
ied the use of the VR TB as an assessment tool for OTOHNS trainees. Ca-
daveric temporal bone and VR TB drilling were assessed by both expert
observers and hand-motion analysis. Experts reviewed videotaped sessions
and were able to distinguish novice and experienced surgeons (construct val-
idity) on the cadaver models but only a trend toward doing so on the VR
TB. Experienced trainees outperformed novices in all hand-motion analysis
metrics on the VR TB and only on the time-on-task metric for the cadaveric
models. This limited study of 19 trainees concluded that the VR TB is an
appropriate assessment of trainees for transition from laboratory-based to
operative-based learning. More research needs to be performed to confirm
temporal bone simulator validity and reliability as a competency assessment
tool [61].
In otolaryngology, just as in general surgery, simulation technology fo-
cuses on endoscopic approachesdmost notably endoscopic sinus surgery.
For example, a low-fidelity simulator using a force-torque sensor during
gauze packing in a human nasal model was able to differentiate experienced
and intermediate endoscopic sinus surgeons [62]. More experience has been
gained in the OTOHNS with an endoscopic sinus surgery simulator (ES3)
developed by Lockheed Martin (Akron, Ohio). The ES3 comprises four
principal hardware components: a simulation host platform (high-powered
Silicon Graphics workstation); a haptic controller that provides coordina-
tion between the universal instrument handler and the virtual surgical in-
struments; a voice-recognition instructor that operates the simulator; and
an electromechanical platform that holds the endoscope replica, universal
surgical instrument handle, and rubber human head model. Simulated sur-
gical tasks range from vasoconstrictor injection to total ethmoidectomy and
agar nasi dissection. The ES3 has a novice mode, thought to be a good tool
to assess skill competency, whereas the intermediate mode seems best suited
for surgical training. The advanced mode has potential as a practice and re-
hearsal tool for trained learners. Fried and colleagues [63] have performed
extensive construct validation studies of the ES3 to demonstrate its discrim-
inative capabilities. It appears to be a viable assessment tool for various en-
doscopic skills, especially if used in the novice mode, and correlates strongly
with other validated measures of perceptual, visuospatial, and psychomotor
performance [64,65]. Their extensive experience observing expert perfor-
mance in the ES3 has allowed benchmark criteria to be developed that
will be useful in the future to establish objective levels of proficiency. Its use-
fulness in predicting endoscopic sinus surgery skills in the OR (predictive
validity) remains to be shown.

Possible new techniques

Several other experimental adaptations of technology to assessment
methods have been proposed or are currently being investigated. Eye
1254 TSUE et al

tracking during procedures is possible from vestibular testing technology

and may complement hand-motion analysis. Functional brain mapping is
an area of early current investigation clinically, and its usefulness in surgical
assessment is not far-fetched. Downloading specific patient data into simu-
lators could strengthen the correlation between performance in the simula-
tor and in the OR. With intraoperative CT and MRI scans entering the
market, real-time updating of stereotactical guidance systems and simula-
tors should be eventually possible. Also, operative performance metrics
could be developed from the real-time results of these intraoperative scans.
Intraoperative videotaping technology is also constantly improving, and
cameras are available in headlights and overhead lights, making routine
use of this technique more available.

Final determinants of competency

Whatever the methodology used, and regardless of whether it is in the
laboratory or the OR, measurements of patient outcomes by individual sur-
geons must still be used in the final determinant of competency. The chal-
lenge lies in simplifying the metric for such diverse outcomes as operative
blood loss to malignancy recurrence rate. Additional research needs to be
done linking these measurements with today’s chosen skill assessment tech-
niques. Despite some progress induced by ‘‘pay-for-performance’’ regula-
tions, which have different end goals than those of trainees, the surgical
academic community has placed inadequate focus on this important metric.
A more objective and comparable method needs to be developed to allow
measurement of trainee progress, but also comparison between trainees
and community norms. Unfortunately, unless physicians themselves develop
these assessments, they may be imposed on the profession by regulatory
agencies.

The Otolaryngology–Head and Neck Surgery Resident Review Committee

pilot study
The ACGME OTOHNS RRC, in the spirit of the ACGME Outcome
Project, has begun to direct its efforts toward outcomes-based evaluations
rather than process-based evaluations. This includes the assessment of
both technical competency and overall surgical competency. Rather than
waiting for the ideal assessment method, or continuing to rely on just the
resident operative case logs, the RRC, in coordination with the ABOto, is
piloting a project on the use of surgical checklists to aid in assessing surgical
competency.
The RRC and the ABOto have defined a list of approximately 40 core
procedures in which all otolaryngology residents should be proficient at
the conclusion of their training. At specified intervals during their training,
 ASSESSMENT OF SURGICAL COMPETENCY 1255

residents will be assessed on their competency with these procedures, and

this assessment maintained as part of the resident’s permanent record.
This assessment approach is based on work done in the urologic and tho-
racic surgery communities, and includes not only technical proficiency,
but also the overall understanding of the procedure, its operative flow,
and requisite perioperative patient care. The resident is graded on his or
her performance on a scale from novice to competency to independence.
Assessments can be completed by the supervising faculty in real time, dur-
ing, or at the conclusion of the appropriate procedure, or performed more
globally by faculty committees on a semi-annual basis. A limited number of
OTOHNS programs are currently piloting this project. Data should reveal
which procedures the majority of residents are competent in during their
training, as well as indicating at which training level competency is
achieved.
The focus of this national pilot study includes not just the technical as-
pects of the surgical procedure, but also the understanding of the procedure
flow and interactions with other health care personnel during the perioper-
ative period. This makes more sense than relying on numbers of procedures
performed, as the number of cases required to achieve competency for any
one procedure will vary with the procedure, the resident involved, and the
teaching effectiveness of the faculty. It is anticipated that the completed
checklists can then be provided by the program to support the competency
of the individual at the time of board certification and the effectiveness of
the program at the time of ACGME accreditation.

The future
All assessment efforts should be focused on the goal of producing the
most outstanding graduating residents in the OTOHNS possible. No single
assessment will be the panacea to the struggle to prove surgical competency
in the trainees; instead, a mixture of assessment tools will be required. The
resident must pass each assessment in a specified longitudinal fashion, rather
than having a passing average for a group of assessments. Advancement of
the residents through their training should depend on these well-defined
milestones of competency rather than one mostly dependent on time and ex-
perience. This may make some training periods longer for some and shorter
for others. For example, technical surgical progress through the early years
of residency could be assessed every 6 months on bench models of core fun-
damental surgical techniques. These techniques would be made up of core
components of both basic and advanced OTOHNS procedures. As compe-
tency is progressively obtained and documented, the trainee is allowed to
progress to a more senior status, and regular assessments with higher-fidelity
bench models and, ultimately, virtual reality simulators could be integrated.
Annually, each resident could participate in an annual competency fair,
1256 TSUE et al

testing more in-depth skills using different methods with the entire resident
complement (junior and senior trainees). This could all take place in parallel
with objective structured observations during live or videotaped level-appro-
priate procedures throughout the year. Objective testing of every procedure
may not be possible, but competency in defined seminal procedures that
form the basis of an OTOHNS practice must be demonstrated at each level
of competency-based advancement. The trainees would be required to main-
tain a portfolio of this stepwise structured progress in surgical technical
competency, and advancement would depend on successful completion of
each objective assessment. If this were standardized nationally, it could be
adopted as part of the ABOto certification process. Objective documenta-
tion of the progress toward surgical competency, especially technical skill
competency, can be monitored during training rather than from an ‘‘after
graduation’’ certification examination, when the usefulness of feedback is
less timely. This approach would make the certification of the residents’
progress to technical competency more formative rather than summative,
and thus, help to further their progress toward surgical competency.

Summary
Classic surgical training and assessment have been based on the appren-
ticeship model. The vast majority of residents are trained well, so radical
changes in the methodology must be approached with caution. Technical
skill remains only one component of overall surgical competency, but has
been one of the most difficult to measure. Assessment methods are currently
subjective and unreliable and include techniques such as operative logs, end-
of-rotation global assessments, and direct observation without criteria.
Newer objective methods for assessing technical skill are being developed
and undergoing rigorous validation andinclude direct observation with cri-
teria, final product analysis, and hand-motion analysis. Following the exam-
ple set in fields in which high-stakes assessment is paramount, such as in
aviation, virtual reality simulators have been introduced to surgical compe-
tency assessment and training. Significant work remains to integrate these
assessments into both training programs and practice and to demonstrate
a resultant improvement in surgical outcome. Continuous assessment and
subsequent real-time feedback provided by these methods are important
in the structured learning of surgical skills and will prove to be increasingly
important in the documentation of the trainees’ surgical competency.

References
[1] The Joint Commission. Available at: www.jointcommission.org. Accessed July 9, 2007.
[2] The Bristol Royal Infirmary Inquiry. The inquiry into the management of care of children
receiving complex heart surgery at the Bristol Royal Infirmary. Available at: www.
Bristol-inquiry.org.uk. Accessed July 9, 2007.
 ASSESSMENT OF SURGICAL COMPETENCY 1257

[3] Giddings T, Gray G, Maran A, et al. Response to the general medical council determination
on the Bristol case. London: The Senate of Surgery of Great Britain and Ireland; 1998.
[4] Dibb CB. Medical residency: when are program administrators liable? Journal of Legal Ed-
ucation 2007;281:1–8.
[5] Outcome Project: Enhancing residency education through outcomes assessment. Available
at: www.acgme.org/Outcome. Accessed July 9, 2007.
[6] Darzi A, Mackay S. Assessment of surgical competence. Qual Health Care 2001;10(Suppl
II):ii64–9.
[7] Yule S, Flin R, Paterson-Brown S, et al. Non-technical skills for surgeons in the operating
room: a review of the literature. Surgery 2006;139(2):140–9.
[8] Wright M, Turner D, Harburg C. Competence assessment for the hazardous industries. Sud-
bury (Great Britain): Greenstreet Berman Ltd. For Health and Safety; 2003.
[9] Hamman WR. The complexity of team training: what we have learned from aviation and its
application to medicine. Qual Saf Health Care 2004;13:72–9.
[10] Collyer SC, Malecki GS. Tactical decision making under stress: history and overview. In:
Cannon-Bowers JA, Salas E, editors. Making decisions under stress: implications for indi-
vidual and team training. Washington, DC: American Psychological Association; 1999.
p. 3–15.
[11] Fletcher G, Flin R, McGreorge P, et al. Anaesthetists’ Non-technical Skills (ANTS): evalu-
ation of a behavioral marker system. Br J Anaesth 2003;90:580–8.
[12] Flin R, Yule S. The non-technical skills for surgeons (NOTSS) systems handbook v1.2. 2006.
Available at: http://www.abdn.acuk/iprc/notss. Accessed July 9, 2007.
[13] Airasian PW. Classroom assessment. 3rd edition. New York: McGraw-Hill; 1997.
[14] Kern DE, Thomas PA, Howard DM, et al. Curriculum development for medical education:
a six-step approach. Baltimore (MD): Johns Hopkins University Press; 1998.
[15] Cosman PH, Cregan PC, Martin CJ, et al. Virtual reality simulators: current status in acqui-
sition and assessment of surgical skills. ANZ J Surg 2002;72:30–4.
[16] Scott DJ, Valentine RJ, Bergen PC, et al. Evaluating surgical competency with the American
Board of Surgery In-Training Examination, skill testing, and intraoperative assessment. Sur-
gery 2000;128(4):613–22.
[17] Adrales GL, Donnelly MB, Chu UB, et al. Determinants of competency judgments by expe-
rienced laparoscopic surgeons. Surg Endosc 2004;18(2):323–7.
[18] Reznick RK. Teaching and testing technical skills. Am J Surg 1993;165:358–61.
[19] Carr MM. Program directors’ opinions about surgical competency in otolaryngology resi-
dents. Laryngoscope 2005;115:1208–11.
[20] Grantcharov TP, Bardram L, Funch-Jensen P, et al. Assessment of technical surgical skills.
Eur J Surg 2002;168:139–44.
[21] Moorthy K, Munz Y, Adams S, et al. Self-assessment of performance among surgical
trainees during simulated procedures in a simulated operating theater. Am J Surg 2006;
192(1):114–8.
[22] Moorthy K, Munz Y, Sarker SK, et al. Objective assessment of technical skills in surgery.
BMJ 2003;327:1032–7.
[23] Available at: http://www.acgme.org/acWebsite/RRC_280/280_resEval.asp. Accessed July
9, 2007.
[24] Martin JA, Regehr G, Reznick R, et al. Objective structured assessment of technical skill
(OSATS) for surgical residents. Br J Surg 1997;84(2):273–8.
[25] Siker ES. Assessment of clinical competence. Curr Opin Anaesthesiol 1999;12(6):677–84.
[26] Pandey VA, Wolfe JH, Liapis CD, et al. The examination assessment of technical compe-
tence in vascular surgery. Br J Surg 2006;93(9):1132–8.
[27] Dailey SH, Kobler JB, Zeitels SM. A laryngeal dissection station: educational paradigms in
phonosurgery. Laryngoscope 2004;114(5):878–82.
[28] Gosman GG, Simhan HN, Guido RS, et al. Focused assessment of surgical performance:
difficulty with faculty compliance. Am J Obstet Gynecol 2005;193(5):1811–6.
1258 TSUE et al

[29] Darzi A, Datta V, Mackay S. The challenge of objective assessment of surgical skill. Am
J Surg 2001;181:484–6.
[30] Datta V, Bann S, Beard J, et al. Comparison of bench test evaluations of surgical skill with
live operating performance assessments. J Am Coll Surg 2004;199(4):603–6.
[31] Beard JD, Jolly BC, Newble DI, et al. Assessing the technical skills of surgical trainees. Br
J Surg 2005;92:778–82.
[32] Roberson DW, Kentala E, Forbes P. Development and validation of an objective instrument
to measure surgical performance at tonsillectomy. Laryngoscope 2005;115(12):2127–37.
[33] Naik VN, Perlas A, Chandra DB, et al. An assessment tool for brachial plexus regional an-
esthesia performance: establishing construct validity and reliability. Reg Anesth Pain Med
2007;32(1):41–5.
[34] Hance J, Aggarwal R, Stanbridge R, et al. Objective assessment of technical skills in cardiac
surgery. Eur J Cardiothorac Surg 2005;28(1):157–62.
[35] Saleh GM, Gauba V, Mitra A, et al. Objective structured assessment of cataract surgical
skill. Arch Ophthalmol 2007;125:363–6.
[36] Datta V, Bann S, Mandalia M, et al. The surgical efficiency score: a feasible, reliable, and
valid method of skills assessment. Am J Surg 2006;192(3):372–8.
[37] Szalay D, MacRae H, Regehr G, et al. Using operative outcome to assess technical skill. Am
J Surg 2000;180:234–7.
[38] Datta V, Mandalia M, Mackay S, et al. Relationship between skill and outcome in the lab-
oratory-based model. Surgery 2002;131:318–23.
[39] Bann S, Khan M, Datta V, et al. Surgical skill is predicted by the ability to detect errors. Am
J Surg 2005;189(4):412–5.
[40] Zirkle M, Roberson DW, Leuwer R, et al. Using a virtual reality temporal bone stimulator to
assess otolaryngology trainees. Laryngoscope 2007;117(2):258–63.
[41] Taffinder N, Smith SG, Huber J, et al. The effect of a second-generation 3D endoscope
on the laparoscopic precision of novices and experienced surgeons. Surg Endosc 1999;
13:1087–92.
[42] Datta V, Mandalia M, Mackay S, et al. Relationship between skill and outcome in the lab-
oratory based model. Surg 2002;131(3):318–23.
[43] Datta V, Mackay S, Mandalia M, et al. The use of electromagnetic motion tracking analysis
to objectively measure open surgical skill in the laboratory-based model. J Am Coll Surg
2001;193:479–85.
[44] Datta V, Chang A, Mackay S, et al. The relationship between motion analysis and surgical
technical assessments. Am J Surg 2002;184(1):70–3.
[45] Porte MC, Xeroulis G, Reznick RK, et al. Verbal feedback from an expert is more effective
than self-accessed feedback about motion efficiency in learning new surgical skills. Am J Surg
2007;193(1):105–10.
[46] Reznick RK. Surgical simulation: a vital part of our future. Ann Surg 2005;242(5):640–1.
[47] Anastakis D, Regehr G, Reznick RK, et al. Assessment of technical skills transfer from
the bench training model to the human model. Am J Surg 1999;177(2):167–70.
[48] Seymour NE, Rotnes JS. Chellenges to the development of complex virtual reality simula-
tions. Surg Endosc 2006;20:1774–7.
[49] Wilson MS, Middlebrook A, Sutton C. MIST-VR: a virtual reality trainer for laparoscopic
surgery assesses performance. Ann R Coll Surg Engl 1997;79:403–4.
[50] Hanna GB, Drew T, Clinch P, et al. Computer-controlled endoscopic performance assess-
ment system. Surg Endosc 1998;12:1997–2000.
[51] Moorthy K, Smith S, Brown T, et al. Evaluation of virtual reality bronchoscopy as a learning
and assessment tool. Respiration 2003;70(2):195–9.
[52] Henderson BA, Ali R. Teaching and assessing competence in cataract surgery. Curr Opin
Ophthalmol 2007;18(1):27–31.
[53] Schendel S, Montgomery K, Sorokin A, et al. A surgical simulator for planning and perform-
ing repair of cleft lips. J Craniomaxillofac Surg 2005;33(4):223–8.
 ASSESSMENT OF SURGICAL COMPETENCY 1259

[54] Moorthy K, Munz Y, Adams S, et al. A human factor analysis of technical and team skills
among surgical trainees during procedural simulations in a simulated operating theatre. Ann
Surg 2005;242(5):631–9.
[55] Moorthy K, Munz Y, Forrest D, et al. Surgical crisis management training and assessment:
a simulation-based approach to enhancing operating room performance. Ann Surg 2006;
244(1):139–47.
[56] Reznick RK, MacRae H. Medical education: teaching surgical skills-changes in the wind.
N Engl J Med 2006;355(25):2664–70.
[57] Baer S, Williams H, McCombe A. A model for instruction in myringotomy and grommet
insertion. Clin Otolaryngol 1990;15:383–4.
[58] Holt GR, Parel SM, Shuler SL. A model training ear for teaching paracentesis, myringot-
omy, and insertion of tympanostomy tubes. Otolaryngol Head Neck Surg 1983;91:333–5.
[59] Hantman I. An ear manikin. Teaching and training device. Arch Otolaryngol 1968;88:
407–12.
[60] Neal SL, Harris JP, Davidson TM. Artificial eardrum for instruction in myringotomy and
PET tube insertion. Laryngoscope 1985;95:1008–9.
[61] Sewell C, Morris D, Blevins NH, et al. Validating metrics of a mastoidectomy simulator.
Stud Health Technol Inform 2007;125:421–6.
[62] Kumagai T, Yamashita J, Morikawa O, et al. A new force-based objective assessment of
technical skills in endoscopic sinus surgery. Stud Health Technol Inform 2007;125:235–7.
[63] Fried MP, Sadoughi B, Weghorst SJ, et al. Construct validity of the endoscopic sinus surgery
simulator: II. Assessment of discriminant validity and expert benchmarking. Arch Otolar-
yngol Head Neck Surg 2007;13:350–7.
[64] Arora H, Uribe J, Ralph W, et al. Assessment of construct validity of the endoscopic sinus
surgery stimulator. Arch Otolaryngol Head Neck Surg 2005;131(3):217–21.
[65] Available at: http://www.acgme.org/acWebsite/resEvalSystem/reval_otolaryngology.asp.
Accessed July 9, 2007.