Improving the continuity of patient care

through teaching and researching novel

patient handover processes in Europe









WP3-Deliverable: Development of Learning
Outcomes [Public Part]





2
Project information
Project acronym: PATIENT
Project title: Improving the continuity of patient care through teaching
and researching novel patient handover processes in
Europe
Project number: 527620-LLP-1-2012-1-NL-ERASMUS-FEXI
Sub-programme or KA: Erasmus Multilateral Projects
Project website: http://www.patient-project.eu
Report Type Public
Reporting period: From 01/04/2013
To 30/09/2013
Report version: 1 Public
Date of preparation: 04/10/2013
Contributors Mariona Secanell Carola Orrego, Lina Stieger, Susanne
Druener, Sasa Sopka

Authors (Partner) Helen Hynes, Patrick Henn, Bridget Maher, Hendrik
Drachsler, Slavi Stoyanov

Contact Person Patrick Henn (p.henn@ucc.ie)
Work Package Work Package 3 - Development of Learning Outcomes
Project coordinator: Hendrik Drachsler
Project coordinator organisation: Open Universiteit Nederland, CELSTEC
Project coordinator telephone number: +31 45 576 2218
Project coordinator email address: hendrik.drachsler@ou.nl

This project has been funded with support from the European Commission.
This publication [communication] reflects the views only of the author, and the
Commission cannot be held responsible for any use, which may be made of the
information contained therein.


2008 Copyright Education, Audiovisual & Culture Executive Agency.
The document may be freely copied and distributed provided that no modifications are
made, that the source is acknowledged and that this copyright notice is included.



3
Executive !"##$%&
("%)*+,
1he alm of Work ackage 3 of Lhe A1lLn1 ro[ecL ls Lo develop, by consulLaLlon, agreed
Learnlng CuLcomes for Leachlng of Pandover ln a healLhcare conLexL, conLrlbuLlng Lo
sLandardlzaLlon of handover procedures ln Lurope. 1hese learnlng ouLcomes are Lo be
generaLed uslng consensus from a panel of experLs.
-,./*0
1he meLhod for developlng learnlng ouLcomes lnvolved Lhe parLlclpanLs ln acLlvlLles LhaL mosL
professlonals are used Lo: ldea generaLlon, sorLlng of ldeas lnLo groups and raLlng Lhe ldeas on
some values (e.g. lmporLance and dlfflculLy Lo achleve). 1he analysls applles mulLl-dlmenslonal
scallng and clusLer analysls Lo vlsually deplcL Lhe experLs' shared represenLaLlons on Lhe
learnlng ouLcomes as LhemaLlc groups. Cne of Lhe dlsLlngulshlng characLerlsLlcs of CCM ls Lhe
vlsuallsaLlon of Lhe resulLs from Lhe analysls. vlsuallsaLlon allows for grasplng aL once Lhe
emerglng daLa sLrucLures and Lhelr lnLerrelaLlonshlps Lo supporL declslon-maklng. Croup
ConcepL Mapplng produces Lhree maln Lypes of vlsuallsaLlons: concepLual maps, paLLern
maLches and go-zones.
1*234"+5*2+
1hls sLudy provlded noL only an emplrlcal basls for ldenLlfylng Lhe maln learnlng ouLcomes
areas, buL also suggesLed how Lo operaLlonally deflne Lhem (Lhrough Lhe sLaLemenLs ln each
clusLer).
1hese learnlng ouLcomes are exLenslve and may be more sulLable for lncorporaLlon lnLo Lhe
medlcal currlculum as a whole raLher Lhan slmply a speclflc Lralnlng module on handover.
Powever Lhose ouLcomes consldered mosL lmporLanL were also among Lhose consldered mosL
dlfflculL Lo dellver.




4
!"#$% '( )'*+%*+,
1. Introduction ............................................................................................................................. 5
2. Method ..................................................................................................................................... 6
2.1 LxperL SelecLlon ...................................................................................................................................... 6
2.2 Croup ConcepL Mapplng ........................................................................................................................ 6
3. Results ..................................................................................................................................... 8
3.1 uemographlcs ......................................................................................................................................... 8
3.2 SLaLemenLs ............................................................................................................................................. 8
3.3 ClusLerlng 8esulLs ................................................................................................................................... 8
4. Discussion ............................................................................................................................... 9
5. References ............................................................................................................................. 10




5
1. Introduction
1he alm of Work ackage 3 of Lhe A1lLn1 ro[ecL ls Lo develop, by consulLaLlon, agreed
Learnlng CuLcomes for Leachlng of Pandover ln a healLhcare conLexL, conLrlbuLlng Lo
sLandardlzaLlon of handover procedures ln Lurope. 1hese learnlng ouLcomes are Lo be
generaLed uslng consensus from a panel of experLs.
1here are Lwo ma[or lssues wlLh onllne experL consulLaLlons: (a) generaLlng a comprehenslve
seL of learnlng ouLcomes, and (b) reachlng agreemenL on Lhem. uurlng Lhe preparaLlon of a
llsL of learnlng ouLcomes, parLlclpanLs mlghL be focused on Lhe currenL pracLlce of a course
already runnlng. 1hls would narrow Lhe scope of Lhe learnlng ouLcomes Lo be deflned. An
agreemenL on learnlng ouLcomes mlghL even be more dlfflculL Lo achleve: Lhe parLners
represenL dlfferenL healLhcare sysLems and lndlvldually Lhey mlghL have raLher dlfferenL
Lhlnklng sLyles. 1he parLlclpanLs mlghL also noL agree on how much emphasls should be puL on
each learnlng ouLcome. AddlLlonally, durlng llve meeLlngs Lhere ls always Lhe phenomenon of
'groupLhlnk', or 'peer-pressure', (Lhe negaLlve effecL of Lhe group on Lhe oplnlons of Lhe
lndlvldual members).
MeLhods for experL consulLaLlon, such as locus groups, AfflnlLy dlagram and Lhe uelphl
meLhod are some of Lhe mosL used sLrucLured approaches almed aL achlevlng consensus.
Powever, Lhe analysls of focus group daLa lmposes pre-deLermlned classlflcaLlon schemas,
whlch can be elLher non-exhausLlve or lmpose blases. ln afflnlLy dlagram sesslons, parLlclpanLs
Lyplcally would suggesL dlfferenL clusLerlng soluLlons, boLh ln Lerms of number of clusLers and
Lhe conLenL of Lhe clusLers, whlch makes lL dlfflculL for researchers Lo come up wlLh a unlfled
vlslon on how besL Lo sLrucLure Lhe lnformaLlon. 1he uelphl meLhod requlres several lLeraLlve
rounds before clalmlng consensus ln Lhe group. 1he consensus ls more or less forced and Lhe
sub[ecLlve approach ls always Lhere. Cur soluLlon Lo Lhe lssues descrlbed ls Croup ConcepL
Mapplng (CCM) (kane & 1rochlm, 2006, 1rochlm, 1989). 1hls research meLhodology, whlle
bulldlng on Lhe sLrengLhs of locus groups, AfflnlLy dlagrams and Lhe uelphl-meLhod mlLlgaLes
some of Lhelr weaknesses. lL has been successfully applled ln Lhe medlcal domaln ln Lhe pasL
by (SLoyanov eL al., 2012)
1he prlmary research quesLlon for Lhe A1lLn1 learnlng ouLcome sLudy Lo answer ls:
!"# %&' #( )*++",- +&,-'(,) -" &,,./( &- &' &0,((1 )(- "2 -3( 4567896
!&'1"/(, :"1*;( ;(&,'.'0 "*-%":()<
1he reporL ls sLrucLured as follows: ln Lhe meLhod secLlon, we lnLroduce Lhe deslgn of Lhe
sLudy uslng Lhe Croup ConcepL Mapplng meLhodology. ln Lhe resulLs secLlon we presenL
parLlclpanL demographlcs and Lhe resulLs from Lhe Croup ConcepL Mapplng on clusLerlng and
raLlng of aggregaLed learnlng ouLcome sLaLemenLs. 1he dlscusslon secLlon crlLlcally reflecLs on
Lhe ouLcomes. llnally, Lhe concluslons and suggesLlons secLlon presenLs recommendaLlons for
furLher developmenL of Lhe A1lLn1 Pandover educaLlonal module.




6
2. Method
2.1 Expert Selection
1he consorLlum members agreed on a selecLlon framework for ldenLlfylng experLs Lo
conLrlbuLe Lo Lhe group concepL mapplng (CCM) process for Lhe ldenLlflcaLlon of learnlng
ouLcomes for Lhe handover sLudy module for undergraduaLe medlcal sLudenLs.
2.2 Group Concept Mapping
Croup ConcepL Mapplng (CCM) ls a sLrucLured, mlxed approach applylng boLh quanLlLaLlve and
quallLaLlve measures Lo ob[ecLlvely ldenLlfy an experL group's common undersLandlng abouL a
parLlcular lssue, ln our case Lhe A1lLn1 Pandover module learnlng ouLcomes. 1he meLhod
lnvolved Lhe parLlclpanLs ln a few acLlvlLles LhaL mosL of professlonals are used Lo: ldea
generaLlon, sorLlng of ldeas lnLo groups and raLlng Lhe ldeas on some values (e.g. lmporLance
and dlfflculLy Lo achleve). 1he parLlclpanLs work lndlvldually buL lL ls Lhe advanced sLaLlsLlcal
Lechnlques of mulLldlmenslonal scallng and hlerarchlcal clusLer analysls LhaL quanLlLaLlvely
aggregaLes lndlvldual lnpuLs of Lhe parLlclpanLs Lo reveal ob[ecLlve paLLerns ln Lhe daLa. Cne of
Lhe dlsLlngulshlng characLerlsLlcs of CCM ls vlsuallsaLlon, whlch ls a subsLanLlal parL of Lhe
analysls. vlsuallsaLlon allows for grasplng aL once Lhe emerglng daLa sLrucLures, Lhelr
lnLerrelaLlonshlps, and Lhelr lnLerpreLaLlon Lo supporL declslon-maklng. Croup ConcepL
Mapplng produces Lhree maln Lypes of vlsuallsaLlons: concepLual maps, paLLern maLches and
go-zones.
ln conLrasL Lo Lhe uelphl meLhod, ln CCM, Lhere ls only one round of daLa sLrucLurlng as Lhe
parLlclpanLs work lndependenLly and anonymously of each oLher Lo llmlL Lhe posslblllLy of
'groupLhlnk' or 'peer-pressure'. unllke lnLervlews and focus groups, CCM does noL rely on pre-
deLermlned classlflcaLlon schemas. 1he meLhod does noL need lnLer-coder dlscusslon Lo come
up wlLh an agreemenL. When sorLlng Lhe sLaLemenLs lnLo groups, Lhe parLlclpanLs, ln facL,
'code' Lhe LexL Lhemselves. 1hen mulLlvarlaLe sLaLlsLlcal analysls aggregaLes Lhe lndlvldual
codlng schemas across Lhe parLlclpanLs. Consensus ls noL forced, buL emerges from Lhe daLa.
Croup ConcepL Mapplng supporLs Lhe researcher ln deallng wlLh dlverse lnformaLlon,
sLrucLured ln varlous ways, whlch ls a problem ln AfflnlLy dlagram sesslons.
1he CCM procedure conslsLed of flve phases: (1) ldea generaLlon (bralnsLorm) and ldea
prunlng, (2) sorLlng of ldeas lnLo groups, (3) raLlng on Lwo values (lmporLance and dlfflculLy Lo
achleve), (4) analysls of Lhe daLa and (3) lnLerpreLaLlon of Lhe resulLs. All pro[ecL members
were lnvlLed Lo parLlclpaLe ln Lhe learnlng ouLcomes sLudy Lhrough Lhe pro[ecL's onllne
managemenL sysLem. See Appendlx A for leLLer of lnvlLaLlon. An lnvlLaLlon Lo some exLernal
experLs was senL by emall. All parLlclpanLs were fully lnformed abouL Lhe purpose, Lhe
procedure, and Lhe Llme needed for compleLlng Lhe acLlvlLles. 1he parLlclpanLs were provlded
wlLh a llnk Lo Lhe bralnsLormlng page of a web-based Lool for daLa collecLlon and analysls
(ConcepL SysLem Clobal, 2013). 1hey could vlslL Lhe web slLe, as many Llmes Lhey needed
uslng Lhelr own username and password. 1he parLlclpanLs were asked Lo generaLe ldeas
compleLlng Lhe followlng Lrlgger sLaLemenL: Cne speclflc learnlng ouLcome of Lhe Pandover
module ls." 1he ldeas should be a shorL phrases or sLaLemenLs expresslng one LhoughL.



7
We purposely dld noL ask Lhe parLlclpanLs Lo follow sLandard formaLs for deflnlng learnlng
ouLcomes. lnLroduclng such a formaL would be counLerproducLlve, as lL would resLrlcL Lhe free
flow of ldeas. arLlclpanLs were lnsLrucLed LhaL Lhe ldeas generaLed should Lake Lhe form of
shorL phrases or sLaLemenLs, each expresslng a slngle LhoughL. 1he parLlclpanLs were glven Lwo
weeks Lo compleLe Lhe ldea generaLlon Lask.
AfLer compleLlng Lhe ldea generaLlon phase, a small group of professlonals from Lhe
consorLlum was lnsLrucLed Lo check, edlL and lf needed reduce Lhe ldeas Lo a manageable seL
(abouL 100) for Lhe nexL sLages of sorLlng and raLlng. 1he guldellnes were as follows: look for
sLaLemenLs LhaL conLaln more Lhan one ldea and lf needed spllL Lhem, remove ldenLlcal ldeas,
check wheLher Lhe ldeas address Lhe focus prompL, make sure LhaL each unlque ldea ls
lncluded ln Lhe flnal llsL, and make sure LhaL Lhe ldea ls clear, conclse, undersLandable (a
crlLerlon ls how easy or dlfflculL an ldea can be raLed on Lhe Lwo values: lmporLance and
dlfflculLy). 1he flnal llsL, randomlsed, was made Lhen avallable Lo Lhe parLlclpanLs, flrsLly for
Lhe sorLlng of ldeas lnLo groups (based on slmllarlLy ln meanlng), glvlng names Lo Lhe groups,
and secondly for Lhe raLlng of Lhe ldeas on Lwo values - lmporLance and dlfflculLy. lor Lhe
sorLlng, we advlsed Lhe parLlclpanL Lo noL puL all sLaLemenLs ln one group and LhaL for such
sorL of sLudles beLween 3 and 20 groups works well, also noL Lo make a 'resldual' group,
oLher" or , lL ls beLLer Lo puL a sLaLemenL ln lLs own group lf dlfflculL Lo flnd place for lL. lor Lhe
raLlng we lnsLrucLed Lhe parLlclpanLs Lo raLe each sLaLemenL on a 3 polnL LlkerL scale, and
expllclLly asked Lhem Lo use Lhe full range of raLlng. 1he parLlclpanLs were glven Lhree weeks Lo
compleLe boLh sorLlng and raLlng. A remlnder senL afLer Lwo weeks. As ln Lhe bralnsLormlng
phase, Lhe parLlclpanLs could save Lhelr work and reLurn laLer Lo conLlnue.
1he analysls lncludes mulLldlmenslonal scallng (MuS) and hlerarchlcal clusLer analysls (PCA) for
sorLlng Lhe daLa, and means, sLandard devlaLlons, correlaLlons and slgnlflcance LesLs for Lhe
raLlng daLa. MuS analysls Lakes flrsL Lhe LoLal slmllarlLy maLrlx, whlch aggregaLes all lndlvldual
maLrlces of Lhe parLlclpanLs. 1he sorLlng of every parLlclpanL ls sLored as an lndlvldual maLrlx
An lndlvldual maLrlx conslsLs of 1s and 0s. 1 ls puL aL a cross-secLlonal cell lf Lwo sLaLemenLs
have been grouped LogeLher, oLherwlse Lhe value glven ls 0. A cell ln Lhe LoLal slmllarlLy maLrlx
shows how many people have grouped any Lwo sLaLemenLs. lL can range from 0 (no one has
clusLered Lhem) Lo Lhe maxlmal number of parLlclpanLs (all sorLed Lhe Lwo sLaLemenLs). 1he
LoLal slmllarlLy maLrlx can be represenLed as a coordlnaLe maLrlx. lrom Lhe coordlnaLes Lhe
dlsLances beLween all palrs of polnLs can be compuLed and furLher presenLed as a maLrlx of
dlsLances beLween polnLs (1rochlm, 1989). Croup ConcepL Mapplng uses Lhe Ward
agglomeraLlve hlerarchlcal clusLer analysls. lL sLarLs wlLh Lhe assumpLlon LhaL all ldeas are
lndlvldual clusLers, SLlll, human experLs need Lo look aL Lhe soluLlon proposed and declde upon
Lhe number of clusLers LhaL represenLs Lhe daLa ln Lhe besL posslble way and reflecLs Lhe
conLexL of Lhe sLudy.






8
3. Results
3.1 Demographics
61 parLlclpanLs reglsLered lnlLlally Lo Lhe sysLem for onllne daLa collecLlon, supporLlng Lhe CCM
approach, (ConcepL SysLem Clobal, 2013) creaLlng a username and password. 1hey gave Lhelr
lnformed consenL Lo parLlclpaLe. Cf Lhem 43 conLrlbuLed effecLlvely Lo Lhe bralnsLormlng
sesslon and 22 compleLed Lhe sorLlng and raLlng phases.
3.2 Statements
1he experLs produced sLaLemenLs durlng Lhe ldea generaLlon phase. AfLer compleLlng Lhe ldea
generaLlon phase, a small group of professlonals from Lhe consorLlum was ldenLlfled for Lhe
prunlng of ldeas. 1hey were lnsLrucLed Lo check, edlL and lf needed reduce Lhe ldeas Lo a
manageable seL of 100 or so ldeas for Lhe nexL sLages of sorLlng and raLlng.
3.3 Clustering Results
ClusLerlng analysls ldenLlfled clusLers and raLed Lhem on lmporLance and dlfflculL Lo achleve,
see flgure 1 as an example of a 16 clusLer soluLlon and flgure 2 as an example of merglng lL Lo a
3 clusLer soluLlon

llgure 1. 16-clusLer soluLlon.




9

llgure 2. Merglng 16-clusLer soluLlon Lo 3-clusLer soluLlon
1he nexL sLep ln maklng sense of Lhe daLa was Lo aLLach meanlngful labels Lo Lhe clusLers.
1here are Lhree meLhods avallable for labellng. 1he flrsL meLhod ls Lo check whaL Lhe sysLem
suggesLs. 1he sysLem suggesLs a label for a group of sLaLemenLs, based on Lhe label glven by a
parLlclpanL, whose cenLrold ls Lhe closesL Lo Lhe cenLrold of Lhe clusLer formed by Lhe
aggregaLlon of Lhe daLa from all Lhe parLlclpanLs. 1he second meLhod ls Lo look aL Lhe brldglng
values of Lhe sLaLemenLs composlng Lhe clusLer. 1he sLaLemenLs wlLh lower brldglng values
beLLer represenL a clusLer. 1he Lhlrd meLhod ls Lo read Lhrough all Lhe sLaLemenLs ln a clusLer
and Lo deflne ln a label whaL ls Lhe sLory behlnd Lhe sLaLemenLs (whaL ls lL LhaL Lhe clusLer
wanLs Lo Lell us). 1o deflne Lhe clusLer labels (e.g., collecLlve Lheme of Lhe sLaLemenLs, or
caLegory) we comblned all Lhree meLhods.
4. Discussion
43 parLlclpanLs conLrlbuLed effecLlvely Lo Lhe bralnsLormlng sesslon and 22 parLlclpanLs from 4
Luropean counLrles compleLed Lhe sorLlng and raLlng phases. 1he prlnclple LhaL dlsLances
beLween lndlvldual ldeas ln Croup ConcepL Mapplng maLLer, applles Lo dlsLances beLween Lhe
clusLers as well. 1he closer Lhe clusLers are Lo each oLher, Lhe closer Lhey are concepLually. 1he
clusLer analysls ldenLlfled 10 clusLers for conslderaLlon as learnlng ouLcomes for a handover
Lralnlng module for medlcal sLudenLs.
lL seems LhaL Lhe A1lLn1 CCM sLudy ldenLlfled learnlng ouLcomes noL only for a slngle
module on handover buL also for a whole currlculum on handover LhaL gradually can be
developed ln Lhe fuLure. urachsler eL al. (urachsler eL al., 2012) already provlde rlch
educaLlonal maLerlals on Lhe Pandover Loolbox LhaL can be Laken lnLo accounL for Lhe deslgn
of Lhe selecLed learnlng ouLcomes. Maher eL al. provldes a sulLable moblle app Lo Lraln



10
accuraLe and compleLe handovers wlLh medlcal docLors (Maher eL al., 2013, Maher, urachsler,
kalz, & SpechL, 2012).
1he groups of learnlng ouLcomes Lhrough Lhe sLaLemenLs generaLed by Lhe experL parLlclpanLs
provlde an emplrlcal basls for deflnlng learnlng ouLcomes of Lhe A1lLn1 Pandover module
applylng Lhe sLandard of deflnlng learnlng ouLcomes. 1he learnlng ouLcomes of Lhe A1lLn1
pro[ecL have ldenLlfled learnlng ouLcomes from whlch a handover Lralnlng module for medlcal
sLudenLs mlghL be deslgned.
5. References

ConcepL SysLems Clobal [CompuLer SofLware]. ConcepL SysLems, lnc. lLhaca, n.?, 2012.
Drachsler, H., Kicken, W., van der Klink, M., Stoyanov, S., Boshuizen, H. P. A., & Barach, P.
(2012). The Handover Toolbox: a knowledge exchange and training platform for improving
patient care. BMJ quality & safety, 21, 11141120. doi:10.1136/bmjqs-2012-001176
Kane, M., & Trochim, W. M. K. (2006). Concept mapping for planning and evaluation. Concept
mapping for planning and evaluation. (p. 216). Sage Publications.
Maher, B., Drachsler, H., Kalz, M., & Specht, M. (2012). The CLAS App - A mobile training tool
to improve handover procedures between hospital interface and family doctors. In S.
Marcus, S. Mike, & M. Jari (Eds.), Proceedings of the 11th International Conference on
Mobile and Contextual Learning (pp. 3845). Retrieved from http://ceur-ws.org/Vol-
955/papers/paper_29.pdf
Maher, B., Kalz, M., Specht, M., Lezcano, L., Drachsler, H., & Henn, P. (2013). Use of Mobile
Applications for Hospital Discharge Letters - Improving Handover at Point of Practice.
International Journal of Mobile and Blended Learning, 5(3), 129.
Stoyanov, S., Boshuizen, H., Groene, O., van der Klink, M., Kicken, W., Drachsler, H., &
Barach, P. (2012). Mapping and assessing clinical handover training interventions. BMJ
Quality & Safety, 21, 150157. doi:10.1136/bmjqs-2012-001169
Trochim, W. M. K. (1989). An introduction to concept mapping for planning and evaluation. (W.
M. Trochim, Ed.)Evaluation and Program Planning, 12(1), 116. doi:10.1016/0149-
7189(89)90016-5
Ward, !. P., !r. (1963), "Plerarchlcal Crouplng Lo CpLlmlze an Cb[ecLlve luncLlon", !ournal of
Lhe Amerlcan SLaLlsLlcal AssoclaLlon, 38, 236-244