c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 4 ( 2 0 0 9 ) 2638

journal homepage: www.intl.elsevierhealth.com/journals/cmpb

Development of a web database portfolio system with PACS connectivity for undergraduate health education and continuing professional development
Curtise K.C. Ng a, , Peter White a,1 , Janice C. McKay b,2
a

Department of Health Technology & Informatics, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong b Discipline of Medical Imaging, Department of Imaging and Applied Physics, Curtin University of Technology, PO Box U1987, Perth, WA 6845, Australia

a r t i c l e
Article history:

i n f o

a b s t r a c t
Increasingly, the use of web database portfolio systems is noted in medical and health education, and for continuing professional development (CPD). However, the functions of existing systems are not always aligned with the corresponding pedagogy and hence reection is often lost. This paper presents the development of a tailored web database portfolio system with Picture Archiving and Communication System (PACS) connectivity, which is based on the portfolio pedagogy. Following a pre-determined portfolio framework, a system model with the components of web, database and mail servers, server side scripts, and a Query/Retrieve (Q/R) broker for conversion between Hypertext Transfer Protocol (HTTP) requests and Q/R service class of Digital Imaging and Communication in Medicine (DICOM) standard, is proposed. The system was piloted with seventy-seven volunteers. A tailored web database portfolio system (http://radep.hti.polyu.edu.hk) was developed. Technological arrangements for reinforcing portfolio pedagogy include popup windows (reminders) with guidelines and probing questions of collect, select and reect on evidence of development/experience, limitation in the number of les (evidence) to be uploaded, the Evidence Insertion functionality to link the individual uploaded artifacts with reective writing, capability to accommodate diversity of contents and convenient interfaces for reviewing portfolios and communication. Evidence to date suggests the system supports users to build their portfolios with sound hypertext reection under a facilitators guidance, and with reviewers to monitor students progress providing feedback and comments online in a programme-wide situation. 2008 Elsevier Ireland Ltd. All rights reserved.

Received 11 April 2007 Received in revised form 31 October 2007 Accepted 6 October 2008 Keywords: Electronic portfolio Picture archiving and communication system Education Continuing professional development Database Reection

Corresponding author. Tel.: +852 3400 8587; fax: +852 2362 4365. E-mail addresses: curtise ng@hotmail.com, curtise ng@yahoo.com.hk (C.K.C. Ng), htpwhite@polyu.edu.hk (P. White), j.mckay@curtin.edu.au (J.C. McKay). 1 Tel.: +852 3400 8568; fax: +852 2362 4365. 2 Tel.: +61 8 9266 7151; fax: +61 8 9266 4344. 0169-2607/$ see front matter 2008 Elsevier Ireland Ltd. All rights reserved. doi:10.1016/j.cmpb.2008.10.004

c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 4 ( 2 0 0 9 ) 2638

27

1.

Introduction

Increasingly, the use of a web database portfolio system is noted in areas of medical and health education, and for continuing professional development (CPD) [13]. Within health care, for either taught programmes or CPD purposes, both academic and professional development should be noted, and students or practitioners should be encouraged to make the connection between the two, to develop the ability to adapt and adjust learning informed by experience and practice. A portfolio is an instrument which can support both the outcomes-based educational approach and requirements for meeting competence in health care education and professional accreditation processes. The purpose of both is usually to foster reection on previous learning and practice experiences, leading to applied learning and continuous development in a cyclic manner [47]. Electronic portfolios create increased opportunities, when compared with the traditional paper-based formats, to support and facilitate professional development from pre-registration education to later CPD activities. Based on the enormous capability of storage and management, they provide a link for the illustration of professional development to different parties such as academic institutions, registering bodies and employers [811].

What is important is that the system can facilitate authors to meet the content requirements of a portfolio rather than the technical process of portfolio building. This paper presents the development of a tailored web database portfolio system, built for use by radiographers, which is strongly based on the portfolio pedagogy, so as to capture their reective learning and professional development. Flexible tools are granted to users as authors of the portfolio rather than as content providers or uploaders of artifacts, with some standardization to ensure efciency. Picture Archiving and Communication System (PACS) connectivity is also injected into the system to provide a means of facilitating CPD and evidence of life long learning for radiographers by capturing, for example, their workplace outcomes, such as medical images produced. Since CPD is developing among all health professions, it could also be extended to other professional areas so as to provide evidence of practitioners competences throughout their professional lives.

3.

Design considerations

2.
2.1.

Background
Analysis of related work

Web database portfolio systems create the possibility to promote a large scale portfolio practice and to provide learning and assessment opportunities, and tracking capability of development for pre-registration education and CPD. The existing commercial web database portfolio systems, however, do not t with these purposes probably due to the consequences of the electronic portfolio boom [9,12]. The great nancial potential has attracted corporate and academic institutions such as McGraw-Hill and Wadsworth, and prestigious universities such as Stanford University and the University of California, Los Angeles (UCLA) to share in this huge market. However, nancial protability depends on efciency leading to distortion of the portfolio pedagogy. The functions of these systems are not aligned with the pedagogy, i.e. the ability to collect, select and reect on evidence of development/experience. Instead, there is a branching effect to provide functions other than those normally required of a portfolio, including online resumes, course management tools, advising portfolio tools, institutional portals and content management. From a pedagogical perspective, the widening foci create a threat to the effectiveness of the portfolio pedagogy by users using the related functions provided rather than building their own portfolios. Together with the standardization of the portfolio building process to increase efciency, the core component of the portfolio, i.e. reection, is lost [9].

2.2.

Prior work leading to current design decisions

An emphasis on reective learning and professional development can be captured using electronic portfolio systems.

To develop a web database portfolio system based on the portfolio pedagogy, a portfolio framework was determined in advance. The following discussion describes the setting up of a portfolio system used on a voluntary basis by students on an undergraduate radiography programme. Students expect to receive strong support in their professional development, including progress towards and ability to meet required graduating professional competences. A reection workshop to teach the students how to collect and select relevant artifacts was arranged after appropriate standards of progress were delivered at corresponding stages. Students prepared evidence to demonstrate meeting these stages from, for example, assessment instruments. This was placed into the portfolio along with their own reective writing to link up the individual pieces of evidence and to illustrate the level of competence attainment relevant to the standard. This incorporates the mechanism of the portfolio in tying up evidence from multi-dimensional assessment strategies [13]. Throughout the academic year, a facilitator was available to provide additional support and guidance to students who had difculties with the portfolio pedagogy, such as assisting them to recognize their own development. Evaluation of the portfolio by faculty was conducted by reading the reective writing and following the links to the corresponding individual artifacts. They provided ratings of attainment of each competence statement noted. Faculty then provided feedback to each participant to inform them of their progress of competence development along with suggested remedies where necessary. Through this process, faculty obtained a more in-depth view of each students learning and development progress for programme evaluation. This process was repeated over two years. As a formal component of a programme, this practice can be repeated throughout the students study until graduation as a means of monitoring their progress towards, and ability to meet, graduating professional competences. Although the portfolio was developed based on an undergraduate radiography programme, the framework is suitable for other health care programmes or CPD purposes by substituting corresponding competences for different professions [14].

28

c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 4 ( 2 0 0 9 ) 2638

Fig. 1 A model of a web database portfolio system with PACS connectivity.

4.

System description

4.1. Model of web database portfolio system with PACS connectivity

A model of a web database portfolio system is proposed (Fig. 1) taking into consideration the situation of the portfolio framework and the possibility of providing a means of facilitating CPD and providing evidence of life long learning by capturing artifacts such as medical images produced. Web clients such as undergraduate students, clinical practitioners, facilitators and reviewers can use browsers to communicate with a web server through internet, to access the system. The web server runs appropriate Active Server Pages (ASP) scripts (ASP-VBScript) to provide requested services such as user authentication, portfolio building and viewing, feedback and grading, internal message communication, memorandum and changes of user preferences. The ASP scripts provide functions of le upload and connectivity with mail, database and PACS servers apart from general web page rendering. The web server in this study is a Dell PowerEdge 2850 using Internet Information Server (IIS) 6.0 from the Microsoft Server 2003. The mail server uses the default Simple Mail Transfer Protocol (SMTP) virtual server inside the IIS 6.0 to send out automated Electronic Mail (e-mail), so as to alert users. The database server is another Dell PowerEdge 2850 machine running Microsoft SQL Server 2005 in a Microsoft Server 2003 platform. This is responsible for archiving and retrieval of data for system contents rendering and users input through Object Linking and Embedding Database (OLEDB) connection. Capturing of medical images produced by clinical radiographers is achieved by the Query/Retrieve (Q/R) service class of Digital Imaging and Communication in Medicine (DICOM) standard. The Q/R broker converts Hypertext Transfer Protocol (HTTP) requests from clients to Q/R service class including C-Find, C-Move and CStore for a PACS server, and retrieved DICOM images/data to HTTP responses for browser display and archiving in a web database [15].

graduate radiography programme. There were thirty-ve students (authors) from the rst year group, twenty from the second year and nineteen from the nal year groups. An evaluation questionnaire of the system called the ePortfolio questionnaire was delivered to them after they had experienced two cycles of electronic portfolio practice. This aimed to gather users views on the electronic portfolio practice from their experience of using it, including conceptual and technical aspects. Feedback then informed the revision/modication/improvement of the concept and system. For the questionnaire, closed ended questions using a ve point scale (strongly disagree, disagree, neutral, agree and strongly agree) were used to collect specic feedback on three main aspects, including features, benets, and technical performance of the electronic portfolio system. The items in the rst two groupings were developed using secondary data from the literature on (electronic) portfolios [46,9,12,13,1618]. Those in the last category were adapted from the user-perceived web quality instrument developed by Aladwani and Palvia [19]. Open questions were developed to gather general comments on the electronic portfolio practice and the think-aloud technique was conducted with ve volunteers from the user group as a pilot test. They were required to verbalize their thoughts and perceptions when completing the questionnaire, so as to identify any unclear or ambiguous questions. This further ensured its validity [20]. The questionnaires collected from students and staff were analyzed separately. Although the same questionnaire was used for both students and staff, they were part of two cohorts. The mean was used to determine the central tendency and variability was measured using standard deviation (SD) of each closed-ended question. Content analysis was used for the open questions of the student set with quasi-statistics as an accounting system. The proportions of the frequency were then put into tables for data presentation. The responses from open questions by staff have been used as quotations in the discussion as necessary [20,21].

5.
5.1.

Status report
System overview

4.2.

Implementation and evaluation

The system was piloted with seventy-seven volunteers (seventy-four authors and three reviewers) from an under-

Using the proposed model in Fig. 1, a tailored web database portfolio system (http://radep.hti.polyu.edu.hk) was developed. There is a common login page to direct users to author (Fig. 2), reviewer or administrator mode based on their credentials, and to request passwords if forgotten. The functionalities of the author and reviewer modes are similar. However, a Build Portfolio area is provided in the author mode and it becomes Grade Portfolio in the reviewer mode. One additional function, Memo, is also included in the reviewer view. It is an electronic space for reviewers to put down their own experiences and feelings from their teaching, for example, and to accumulate such to provide an evaluation of their teaching performance over a period of time. The administrator interface is for granting special permission to users and maintaining contents of the system such as competence statements archived in the database.

c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 4 ( 2 0 0 9 ) 2638

29

Fig. 2 A snapshot of the Build Portfolio function of the author view.

5.2.

Build Portfolio

Build Portfolio is one of the most important functions of this web database portfolio system. This allows users to conveniently develop their portfolio. In Fig. 2, the rst rectangle highlights the main menu. Users can switch to other functions by clicking the appropriate links. The second rectangle shows the sub-menu (menu of the Build Portfolio function) that allows users to select which stage and competence category they would like to work on. The bottom section is the action menu generated by contents from the database and the ASP script provides a framework for its rendering. There is one heading and four columns. The Competence column shows the corresponding competences required for that stage under the selected category. Criteria further denes the expected levels of development. The Portfolio Covering Letter and Reect column provide links to reection input forms and the Upload Evidence directs users to the le upload page. When a user wants to build a portfolio, the appropriate stage and competence categories are selected in the main menu. Evidence of development against the competence category/statements should be provided rst through accessing the links under the Upload Evidence column. A popup window with probing questions for collection and selection of evidence is shown together with the le upload page. This reinforces users to review their experience for the selected

area, to collect evidence that can represent it, and choose the best evidence out of the collection. Normally, a maximum of two les can be uploaded for each area, so as to support the evidence selection process. If the previous uploaded les occupy the quota, accommodation of new les can be achieved through replacing the previous ones. Also, permission for increasing the quota to a maximum of three can be granted through the administrator interface if the user has a strong reason. The le upload page also has a link to the Q/R interface for DICOM images from PACS. This interface uses a patient root Q/R information model. Clients can input patient level keys such as patient name, patient ID, date of birth and study date to query images [22]. These HTTP inputs are then converted to the DICOM standard through a third party Component Object Model (COM) object, i.e. the Q/R broker called by the ASP page. The Q/R broker becomes the Service Class User (SCU) of the C-Find service and the Service Class Provider (SCP), i.e. the PACS server sends back the query result. Again, it is translated back to HTTP responses. The subsequent retrieval procedure is translated to C-Move and C-Store services by the Q/R broker. Eventually, the required DICOM images are sent to the web server. Another function of the Q/R broker is to convert the retrieved DICOM les to Joint Photo-graphics Experts Group (JPEG) format for browser display and archiving in the web database [15].

30

c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 4 ( 2 0 0 9 ) 2638

Fig. 3 A snapshot of the retrieved images from PACS in JPEG format for display and further archiving.

Fig. 3 shows a snapshot of the Q/R interface displaying the retrieved Magnetic Resonance (MR) images of a patient called PACS Interface Demo taken on 10/07/2004 in JPEG format. The rst rectangle shows the competence area that the images are for. Following is a hyperlink to the le upload page. The third rectangle highlights the information about the images shown underneath it. The images will be archived as evidence for the competence area once the thumbnail is clicked. The last rectangle lists the image le archived in this way. The last step in the portfolio building is reective writing, using the reection input form. Again, a browser window with guiding questions for reection pops up together with the form when the reect hyperlink is clicked. It explains to authors that reection is the written account/justication of the evidence provided and it should include responses to the following questions:

In Fig. 4, a snapshot of a reection input form is shown. The heading is the area for reection. On the left hand side is a text area for users reection input. Opposite is a column headed Evidence Insertion listing out the evidence archived. These are buttons to insert hyperlinks of corresponding evidence les into the text area using JavaScript. At the bottom, there are buttons for saving and submitting the reection. When a user nishes building a portfolio, it is necessary to click the submit button in the left lower corner of the Build Portfolio page to declare it is ready for review. The electronic portfolio is then locked. The hyperlinks of the le upload page and reection input forms are hidden, and access to them is denied to block any subsequent modication. Also, automated e-mails are generated for notifying the responsible reviewer and informing the user of the submission status.

5.3.

Grade Portfolio

1. How do you think the experience selected shows your competence development for this area? 2. Do you see any area requiring further development based on the experience? 3. How can you improve your competences next time? 4. From this evidence what do you see as your particular strengths?

The Grade Portfolio is the most important function of the reviewer interface. As shown in Fig. 5, there are lter and search utilities for locating authors portfolios. The query result is a list of hyperlinks to corresponding authors portfolios. The bottom browser is an authors portfolio called Grade Portfolio Form as it has utilities for providing levels of attainment and feedback. The layout is similar to the authors Build Portfolio page. The major difference is an absence of the

c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 4 ( 2 0 0 9 ) 2638

31

Fig. 4 A snapshot of a reection input form.

criteria column which is substituted by two new columns, i.e. grade and comments. Again, the portfolio is divided into different categories. There is a checkbox called Benchmark Guide to aid the reviewer in providing levels of development. After checking this box, the expected levels of development, as shown in the criteria column of the author mode, are lled in the grade column. This provides a reference for the reviewer to determine the attainment level and saves reviewing time as it is only necessary to make the necessary adjustment. The symbols under the reection and evidence columns signify that there is reection and evidence for the area as hyperlinks. The utilities under the grade and comments columns enable the reviewer to provide the levels of attainment and feedback after reviewing the materials. Eventually, the checkbox Finished Review? is checked to send out automated e-mails notifying authors of the availability of feedback.

folios, i.e. View Portfolio Form. The general layout is also similar to the Grade Portfolio Form. However, grade and comments cannot be modied in this interface. If the relevant contents such as reection, evidence, grade and comments are present in the portfolio, corresponding information or hyperlinks are shown. For example, there is an overall comment for the portfolio shown in Fig. 6 but not elsewhere. There is no hyperlink under the comments column. Also, the font colour of the grades change according to their values. Red is used to highlight an attainment below standard while blue indicates outstanding achievement. Grey represents appropriate development progress. In author mode, the View Portfolio Form is named View Portfolio and the interface is more or less the same.

5.5.

Other utilities

5.4.

View Portfolio

The View Portfolio function aims to increase the readability of portfolios for authors and reviewers. In reviewer mode, there are lter and search utilities to locate relevant portfolios (Fig. 6). Filter parameters include year of study, completion, overall grade and problems with portfolios. This is similar to the Grade Portfolio interface. The search results, i.e. a list of authors names, serve as hyperlinks to corresponding port-

Other utilities of the system include an introduction, user guide, internal message, settings changing and logout. In the introduction, the portfolio pedagogy and its implementation settings are presented. The user guide serves as a quick technical reference. The internal message utility provides a channel for text communication between authors and reviewers. Its advantage over e-mail is to centralize the communication, so as to make it more organized and holistic over time. It seems that e-mail is not an appropriate choice because messages for

32

c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 4 ( 2 0 0 9 ) 2638

Fig. 5 A snapshot of the Grade Portfolio function.

this purpose may scatter over the whole mailbox and mix with other e-mails or even be deleted after a period of time. Automated e-mail notication will re once there is any message for a user. Changes of password and an option of receiving automated e-mail can be made inside the settings changing utility and the logout function is to foster the system security.

facilitation of collaborative learning. The other two questions (Q.18 and Q.24) relate to the Technical Performance of the ePortfolio including ease of use and attractive layout. Using the same rounding up approach, there were four questions (Q.4, Q.13, Q.16 and Q.24) which have a neutral response from the reviewers perspective while a positive response is noted for the remaining issues.

5.6.

Findings of the ePortfolio questionnaire

6.
6.1.

Lessons learned
Alignment with the portfolio pedagogy

Sixty-two questionnaires were collected from the student participants, yielding a response rate of 83.8% (62 out of 74). The three reviewers returned their responses to their questionnaires, yielding a 100% response rate. Table 1 shows only two questions (Q.13 and Q.24) that have mean values below three (neutral) while the remaining questions have positive responses as noted in the students column. However, when a more stringent approach is used by rounding up the mean to integers, Q.11, Q.12, Q.13, Q.16, Q.18 and Q.24 should be considered as neutral responses. The students have provided a positive response to most of the issues raised in the questions. The rst four questions relate to the Benets of the ePortfolio (Q.11, Q.12, Q.13 and Q.16) including development of personal and professional attributes and higher order skills, assurance of education quality, strengthening of teacherstudent relationships and

A good web database portfolio should align with the central underlining values of portfolio pedagogy, which are collect, select and reect. They are the three most important features of portfolio pedagogy, as set out below [9]. 1. A collection of materials from an individuals experiences. 2. A selection of materials from the collection with annotations to justify the selection action which is the outcome of reective thinking. 3. Reection to narrate an individuals development process and attainment through the contents of portfolio for evaluation. 4. Presumption of development.

c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 4 ( 2 0 0 9 ) 2638

33

Fig. 6 A snapshot of the View Portfolio function.

5. Diversity of contents along and across portfolios. 6. A means of communication to readers of an individuals progress and attainment. 7. An evaluation tool ([16], p. 130). In this system, a popup window with guidelines and probing questions of collect, select and reect are shown when authors access le upload pages and reection input forms. Although covered in the reection training workshop, this technical arrangement reminds and reinforces these important aspects. The limitation in the number of les to be uploaded is another measure to foster authors to exercise their reective thinking when making their selection. The Evidence Insertion functionality in the reection input form links the individual uploaded artifacts with the reective writing, so as to increase the coherence of the portfolio as a whole. This arrangement removes the major criticism of web database portfolio systems, i.e. the standardization of the portfolio building process shifts the authors role to that of content provider or uploader of artifacts, leading to a loss of reection [9]. The authors are required to state how they think the experience selected demonstrates their competence development for this area followed by the JavaScript generated hyperlinks to corresponding evidence, and achieves the real spirit of the selection process as noted above [9,10,16]. Together with reective comment to suggest attainment and further improvement, this forms the presumption of development. This realizes the hypertext reection in a web database

portfolio environment which helps to contextualize and support issues to be justied, which is difcult to accomplish in a paper-based portfolio [23]. Accommodation of diversity of contents along and across portfolios is achieved through the system tolerance to different uploaded le formats from traditional JPEG, Portable Document File (PDF) to DICOM. Various reection input forms such as portfolio covering letter, reection form for competence category and for individual statements, also provide exibility for different users. The portfolio covering letter serves to present an organized, overall picture of development, which is an efcient channel for both authors and reviewers. However, detailed reection is best noted using individual reection forms. Combined use of the three channels accommodates a great variety of contents. The Grade Portfolio function provides a convenient interface for portfolio evaluation by use of lter and search utilities to locate authors portfolios, automated generation of hyperlinks to highlight submitted contents, and Benchmark Guide to facilitate the grading process. Communication of attainment to readers is the major use of the View Portfolio page, enhancing the readability of portfolios through functions such as lter and search utilities, automated hyperlinks generation, and colour highlighting of development. Other communication functionalities include internal messages and automated e-mail generation after portfolio submission, evaluation and receipt of messages. However, functionalities other than for portfolio practice are absent in the system, to eliminate any branching out effect.

34

c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 4 ( 2 0 0 9 ) 2638

Table 1 A summary of ndings of the closed-ended questions. Questions Student (n = 62) Meana
Features of the ePortfolio 1. It serves as a collection of materials (referring to the collect procedure) from an individuals experiences 2. It involves a selection of materials from the collection (referring to the select procedure) 3. It contains annotations (written accounts) to justify the selection action, based on the outcome of the individuals reective thinking 4. It contains reections indicating an individuals development process and attainment, following evaluation of these contents 5. There is a presumption (self-evaluation) of development 6. It is capable of accommodating diversity of contents, such as different kinds of evidence over time and covering different development issues 7. It provides a means of communication with readers on an individuals progress and attainment 8. It serves as an evaluation tool Benets of the ePortfolio 9. It promotes reective learning 10. It facilitates recognition of strengths for mapping of career pathways 11. It facilitates development of personal and professional attributes and various higher order skills, such as self-motivation and critical thinking 12. It assures education quality 13. It strengthens teacherstudent relationships 14. It illustrates competence development 15. It supports resume writing and continuing professional development (CPD) 16. It facilitates collaborative learning (involving sharing of portfolios with peers, and providing comments and feedback on others works) Technical Performance of the ePortfolio 17. It is a secure platform 18. It is easy to use 19. It is always reliable and available 20. Its speed of page loading is fast 21. It is interactive and generates dynamic contents 22. It is exible for accommodation of your work 23. Its organization of contents is clear 24. Its layout is attractive
a

Reviewer (n = 3) Meana
4.67 (5) 3.67 (4) 4.33 (4) 3.33 (3) 4.00 (4) 4.67 (5) 4.00 (4) 4.33 (4)

SD
0.61 0.59 0.53 0.70 0.61 0.61 0.67 0.72

SD
0.58 1.53 0.58 1.15 0.00 0.58 1.00 0.58

3.87 (4) 3.82 (4) 3.69 (4) 3.81 (4) 3.98 (4) 3.73 (4) 3.52 (4) 3.85 (4)

3.63 (4) 3.50 (4) 3.48 (3) 3.39 (3) 2.92 (3) 3.51 (4) 3.65 (4) 3.42 (3)

0.79 0.72 0.83 0.86 0.77 0.65 0.66 0.74

4.00 (4) 4.33 (4) 4.00 (4) 4.67 (5) 3.33 (3) 4.00 (4) 4.00 (4) 3.33 (3)

1.00 1.15 1.00 0.58 0.58 1.00 1.00 0.58

3.87 (4) 3.40 (3) 3.50 (4) 3.82 (4) 3.50 (4) 3.52 (4) 3.56 (4) 2.84 (3)

0.71 0.98 0.94 0.78 0.72 0.76 0.80 0.71

4.00 (4) 4.33 (4) 4.33 (4) 4.00 (4) 3.67 (4) 3.67 (4) 3.67 (4) 2.67 (3)

0.00 0.58 0.58 1.00 0.58 0.58 0.58 1.53

Legend: 1 = Strongly disagree; 2 = Disagree; 3 = Neutral; 4 = Agree; 5 = Strongly agree (mean rounding up to integers).

6.2. Discussion of the current electronic portfolio practice

Positive mean scores were noted with respect to Features of the ePortfolio in the ePortfolio Questionnaire (Q.s 18, except Q.4 from the reviewer perspective, Table 1). This not only indicated good alignment of the system functionalities with portfolio pedagogy, as suggested in Section 6.1, but also ensured that quality portfolio processes were in place in the current electronic portfolio practice. This is because a successful portfolio process should posses the features of portfolio pedagogy, such as collect, select and reect of evidence of development [9,11,24]. The only neutral response (Q.4) from the staff ndings could be explained by one reviewers suggestions for improvement: There should be on-going (embedded in their learning, in the form of reective diary) documentation of their learning experiences rather than an overall retrospective summary after the semester. In this way, it may provide quality data. (Reviewer of First Year Portfolios)

From that reviewers perspective, the reections contained in the electronic portfolios cannot indicate an individuals development process since, it is suggested, this is only an overall retrospective summary after the completion of the academic year. The reviewers comments are understandable. The most important issue is that the accumulated reections throughout the three years of study should be able to demonstrate an individuals development process. It is useful to request students to note down their important experiences throughout the year as is common practice with keeping a reective journal, as they may forget some issues over a longer period of time [8]. However, the point is whether it is necessary to expose all experiences without a selection process. If this is the case, the reviewers may be overloaded with reviewing quantities of students reections. It is also not good for the students development of portfolio building skills since the ability to select is one of the essential elements of portfolio building, not only evidence but also reection. The reection which the reviewer works on is actually the nal masterpiece of the students selection effort [4,9]. However, it may be useful to provide an electronic reective journal facility

c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 4 ( 2 0 0 9 ) 2638

35

in the system where students are encouraged to put down thoughts during the year, and from which they select their nal masterpiece. Although only four benets have been perceived by student participants, as noted in the Benets of the ePortfolio (Table 1), the most important ones promotion of reective learning and illustration of competence development were identied in the responses. The capability of electronic portfolio practice for competence diagnosis is the core component of the study aim, while the reection element is essential for this practice to succeed [9,13,24,25]. Identication of the two elements as benets further conrms the success of the current practice. The importance of reection is also noted in a reviewers response to the open question, the Most Successful Aspects of this Exercise/System: . . . ability to encourage students to be reective learners. (Reviewer of Third Year Portfolios) Students and reviewers responded positively to Technical Performance of the ePortfolio (Table 1), and positive ndings were noted in the items: security of platform, reliability and availability, fast page loading, interactivity and generation of dynamic contents, exibility for accommodation of work, clear organization of contents. One more positive response was shown in the reviewers ndings for this category, that it was easy to use (Q.18, Table 1). Further positive responses to the open questions from the reviewer of the third year portfolios are noted below: Benets Gained from the ePortfolio Exercise: . . . it is easy to use. (Reviewer of Third Year Portfolios) the Most Successful Aspects of this Exercise/System: . . . ease of use. (Reviewer of Third Year Portfolios) Suggestions for Improvement: Use of the portfolio does require a good introduction, although once this has been done, it is easy to use. (Reviewer of Third Year Portfolios) The reviewer of the third year portfolios suggested that ease of use depended on a good introduction. The introduction for reviewers took place on a face-to-face, individual basis in contrast to a group brieng session for student participants. This may account for the students neutral response to the issue of ease of use (Q.18). Reinforcement of technical issues may be necessary for students in the training workshop.

cipline, should also be considered as an indicator of their professional development. The use of this is a derivative of the competence assessment, medical record review. The medical records, as the readily available evidence of clinical skills, can be used to infer individuals holistic competence [26,27]. It is noted that the mechanism of the portfolio is to tie up evidence from multi-dimensional assessment strategies [13]. The current system provides a one-stop solution for radiographers to retrieve their evidence of professional development in the workplace, i.e. medical images produced. These images can be saved into the system as evidence of development for portfolio building. This is the collection and selection of evidence of development. Reective writing can then be provided to justify their professional development supported by the retrieved images, thus linking the evidence together. Although this can be achieved by obtaining images from the web based PACS and uploading them to the ePortfolio system, the importance of this integration is to provide automation or ease of use, so as to promote the use of workplace outcomes as evidence for portfolio building in the CPD situation. New features have now been added to the DICOM standard, such as integration with inputs other than traditional radiological images, for example, visible light images and waveforms. The use of the PACS interface is not limited to radiography/radiology but can also serve other related health care disciplines for which these objects can provide evidence of workplace outcomes. Using PACS connectivity may create possible breaches of patient privacy, so user-authentication of the system and conversion of DICOM images received to JPEG are the measures in place to safeguard patient condentiality and prevent abuse by users. User-authentication can validate the identity of users and limit accessibility to certain images archived in the PACS based on their proles. The entire conversion procedure is carried out in the system server and this prevents users from obtaining the original DICOM les. In effect, users can only receive those JPEG images from which all patient identiers have been removed. The conversion of DICOM to JPEG les also facilitates image distribution through the internet in terms of transmission speed and compatibility [14,15].

6.4.

Study advancement

6.3.

PACS connectivity

The electronic portfolio system established in this study also provides PACS connectivity for retrieval of medical images from any PACS. It seems its importance is not signicant for pre-registration education as students assessment outcomes can serve as readily available evidence to demonstrate their professional attainment. However, when extending the use of the electronic portfolio system from pre-registration education to later CPD, it seems the workplace outcomes, for example, medical images produced in the radiography dis-

This tailored web database portfolio system addresses the major criticism of existing products, which is overstandardization leading to a shifting role of those building the portfolio from authors to content providers or uploaders and hence losing reection capabilities [9]. A balance between standardization and exibility is achieved in several ways. The standardized components include the xed interfaces for portfolio building, viewing, grading and internal communication. These serve as the framework to render contents such as competence standards and users data into a meaningful sense. It is crucial to standardize these elements to ensure efciency with little interference to authors reection activities. Its effect on efciency over the existing web portfolio practice, i.e. portfolios housed in a personal homepage style, includes time saving to learn homepage creation and determination of its layout [9,11,28]. Maintenance and administration of portfolio practice is also another benet of standardization, so as to retain its original aim, i.e. to serve a large population

36

c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 4 ( 2 0 0 9 ) 2638

Fig. 7 A snapshot of the electronic portfolio built through the use of CPD Now.

[9,12]. As the standardized components are those provided by the stakeholders, such as education institutions and registering authorities, this approach should not hinder reection practices. However, the most important function of the standardized components is to trigger and facilitate self-directed reective thinking. The remaining functionalities should be exible enough to facilitate this. It is suggested that the portfolio is an effective and efcient way to measure and record individuals personal achievements and professional accomplishments in terms of occupational standards and competences. Through critical reection on learning experiences, individual, separate evidence of professional and competence development are integrated and organized into a meaningful sense, i.e. a simple, clear and manageable display of attainment which is useful for stakeholders and students/practitioners [13,17]. This forms the foundation for designing exible tools to facilitate reection. The Evidence Insertion utility of the reection input forms is developed based on this agenda. It enables authors to use their reective writing to link up with individual artifacts into a holistic masterpiece through cyberspace. This technical arrangement inside a web database portfolio system is an innovative way to address the criticisms of web database portfolios including role shifting, hypertext reection of web portfolios, and the contribution of portfolio pedagogy in the competence movement [9,13,17,23]. Other

exible tools include capability to accommodate a variety of uploaded le formats and choices of various reection input forms for different purposes. Together with the innovative development of PACS connectivity in a web database portfolio system, an advanced portfolio practice for undergraduate medical and health education, and CPD, should be achieved. For reinforcement of the claim of innovation, the CPD Now system developed by the inuential body of the radiography profession, the Society and College of Radiographers, UK, can be used to illustrate this issue. To use the UK system for portfolio building, the user needs to go to My CPD (portfolio building) area and create his/her own CPD Framework. Under this framework, the user submits the learning plans and evidence by putting down relevant information for the different elds inside the standard templates. The major technical difference between the templates for learning plan and evidence, is the le upload functionality provided in the evidence template. After completion of this part, the portfolio building process is completed and the user can go to the My Portfolio (portfolio viewing) area to view his/her portfolio, as noted in Fig. 7. The electronic portfolio contains three main parts, My Personal Details, My Progress and Download My Careers. Under the My Progress section, there are three items including My CPD Evidence, My Learning Plans and My Certicate. The My CPD Evidence and My Learning Plans are the core contents of the portfolio which are lists of inputs from the templates

c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 4 ( 2 0 0 9 ) 2638

37

of learning plans and evidence. In Fig. 7, My CPD Evidence is shown. It lists the previous evidence input in a sequential order and without any access to the supporting evidence les (attachments) [29,30]. This is a typical example of the existing web database portfolio systems. The portfolio building process is greatly restricted by the template, i.e. over-standardization leading to role shifting [9,12]. Also, hypertext reection cannot be achieved [23]. Furthermore, PACS connectivity is not available inside this system [29,30].

7.

Conclusion

Evidence to date suggests the system supports users to build their portfolios with sound hypertext reection under a facilitators guidance, and with reviewers to monitor students (authors) progress and provide feedback and comments online in a programme-wide situation. Positive mean scores were noted with respect to Features of the ePortfolio in the ePortfolio questionnaire. These responses indicated good alignment of system functionalities with portfolio pedagogy and quality portfolio processes. The web database portfolio systems format and use seem to have been well accepted by students and reviewers. On face value it may appear that the use of the electronic portfolio system is not closely related to retrieval of clinical data from PACS, but the clinical data is an integral part of any practitioners workplace outcomes and development. Ways must be provided to enable such evidence to form part of their portfolios, for example, by using the le upload facility. The provision of PACS connectivity in the current electronic portfolio system has illustrated the possibility of providing a more convenient way to address this issue. Subsequently, this becomes an integral part of the development of an advanced electronic portfolio system for practitioners, and goes further than merely addressing the major criticisms of existing products.

8.

Future plans

The outcomes of the electronic portfolio system with PACS connectivity once fully established could be extended to monitor CPD of clinical radiographers and other professional areas, so as to provide evidence of practitioners competences throughout their professional lives. Although this system is established for the radiography discipline, the standardized framework provides ease of substituting relevant competence standards of other disciplines into the database through the administrator interface. In this way, a large population of medical and health care students and workers can be served, so as to achieve the ultimate goal of web database portfolio technology without sacricing the most important value of portfolios, i.e. reection. Finally, it is the functionalities of reection facilitation rather than powerful or attractive technology that are the most crucial factors.

references

[1] P. Bridge, D. Eddy, The virtual portfolio: from conception to reality, J. Radiother. Pract. 5 (2006) 17.

[2] M. Lawson, D. Nestel, B. Jolly, An e-portfolio in health professional education, Med. Educ. 38 (2004) 569 570. [3] G. Duque, A. Finkelstein, L. Winer, S. Gold, Immediacy and efcacy of giving feedback and evaluating medical students during their rotations in geriatric medicine: the McGill electronic portfolio, J. Am. Geriatr. Soc. 51 (2003) S197. [4] J. Gordon, Assessing students personal and professional development using portfolios and interviews, Med. Educ. 37 (2003) 335340. [5] A. Tiwari, C. Tang, From process to outcome: the effect of portfolio assessment on student learning, Nurs. Educ. Today 23 (2003) 269277. [6] J.G. Alexander, S.W. Craft, M.S. Baldwin, G.W. Beers, G.S. McDaniel, The nursing portfolio: a reection of a professional, J. Contin. Educ. Nurs. 33 (2002) 5559. [7] Open University School of Health & Social Welfare, Building portfolios, Nurs. Manag. 9 (2002) 3033. [8] V. Dagley, B. Berrington, Learning from an evaluation of an electronic portfolio to support general practitioners personal development planning, appraisal and revalidation, Educ. Primary Care 16 (2005) 567574. [9] M. Kimball, Database e-portfolio systems: a critical appraisal, Comput. Composit. 22 (2005) 434458. [10] S.L. Ramey, M.L. Hay, Using electronic portfolios to measure student achievement and assess curricular integrity, Nurs. Educ. 28 (2003) 3136. [11] G. Pullman, Electronic portfolios revisited: the efolios project, Comput. Composit. 19 (2002) 151169. [12] T. Batson, The electronic portfolio boom: whats it all about? Campus Technol. (2002) (http://www.campus-technology. com/article.asp?id=6984 Accessed 29 November 2005). [13] M. McMullan, R. Endacott, M.A. Gray, M. Jasper, C.M.L. Miller, J. Scholes, C. Webb, Portfolios and assessment of competence: a review of the literature, J. Adv. Nurs. 41 (2003) 283294. [14] C.K.C. Ng, P. White, J.C. McKay, Establishing a method to support academic and professional competence throughout an undergraduate radiography programme, Radiography (2007) (available online at http://www.sciencedirect.com). [15] H.K. Huang, PACS and Imaging Informatics: Basic Principles and Applications, John Wiley & Sons, Inc., New Jersey, 2004. [16] K.B. Yancey, Portfolio, electronic, and the links between, Comput. Composit. 13 (1996) 129133. [17] M. Ryan, K.H. Carlton, Portfolio applications in a school of nursing, Nurs. Educ. 22 (1997) 3539. [18] D.O. Weddle, S.P. Himburg, N. Collins, R. Lewis, The professional development portfolio process: setting goals for credentialing, J. Am. Diet. Assoc. 102 (2002) 14391444. [19] A.M. Aladwani, P.C. Palvia, Developing and validating an instrument for measuring user-perceived web quality, Inform. Manag. 39 (2002) 467476. [20] B. Johnson, L. Christensen, Educational Research: Quantitative and Qualitative Approaches, Allyn and Bacon, Boston, 2000. [21] D.F. Polit, C.T. Beck, Nursing Research: Principles and Methods, seventh ed., Lippincott Williams & Wilkins, Philadelphia, 2004. [22] M.V. Herk, L. Zijp, Conquest DICOM Server 1.4.11 Documentation/MicroPACS Conformance Statement, The Netherlands Cancer Institute, Amsterdam, 2005. [23] S. Watkins, World wide web authoring in the portfolio-assessed, (inter)networked composition course, Comput. Composit. 13 (1996) 219230. [24] R. Mason, C. Pegler, M. Weller, E-portfolios: an assessment tool for online courses, Br. J. Educ. Technol. 35 (2004) 717 727.

38

c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 4 ( 2 0 0 9 ) 2638

[25] K.A. Karlowicz, The value of student portfolios to evaluate undergraduate nursing programs, Nurs. Educ. 25 (2000) 8287. [26] J.A. Sliwa, K.J. Kowalske, Assessing resident clinical competence, Am. J. Phys. Med. Rehabil. 79 (2000) 468473. [27] E.S. Holmboe, R.E. Hawkins, Methods for evaluating the clinical competence of residents in internal medicine: a review, Ann. Intern. Med. 129 (1998) 4248.

[28] P.E. McShane, Electronic portfolio: a paperless portfolio for interns to track accomplishments and market themselves, J. Am. Diet. Assoc. 105 (2005) 37. [29] Axia Interactive Media Limited, CPD Now: SOR Professional Development, Axia Interactive Media Limited, West Yorkshire, 2007 (http://sor.cpdnow.net/Logon.aspx Accessed 9 May 2007). [30] S. Kelly, CPD Now, Synergy (March 2005) 2123.