Group Rubric Assignment:

Delivery Platform Evaluation Rubric

Mel Burgess, Colleen Ruddy, Diane Underhill, Emma Sarbit, Bruno Chu

University of British Columbia ETEC 565A

Submitted to: John P. Egan Friday January 31st 2014

Introduction to the Scenario Our group was tasked with developing a rubric to assess an LMS system that could support the UBC Undergraduate Medical Progra ms (UMP) third year students as they embark on Rural Family Practice Clerkships. The Universities of Victoria and Northern British Columbia need to work together with UBC on reliably assessing medical tasks completed by these students while on distance practicums. The students will be geographically separated from some of the faculty members who will be assessing them and providing feedback on their performances. To this

end, the students will be videotaped performing specific clinical skills. The universities will require an LMS system capable of supporting potentially large video files. In addition, there needs to be the added functionality of annotations within the video in order to provide the students with task specific feedback. The students will self asses, as well as have three peer reviews and two faculty assessments. This means that the same clip will be uploaded several times making reliability of the video playback technology vital to the success of the program. Furthermore, third year students will be expected to complete a variety of educational tasks that the LMS will need to host and support. Multimedia projects, discussion forums, and medical case studies in the form of collaborative clinical reasoning, are just a few of the tasks the students will be asked to complete within the LMS. Using the Bates and Pooles (2003) SECTIONS model, we have developed a rubric to assist in evaluating potential LMS systems that will best meet the learning objectives of the program as well as the technological needs to support them from a distance. Ease of use, as well as reliability, for both faculty and students will be paramount to the success of this online learning environment, as the tech support in these rural areas could pose a challenge. A high degree of interactivity will also be required in order for distant faculty to provide the rural practicum students with timely and constructive feedback within the performance task videos. We have developed a three point scale by which the universities can asses the technology under consideration and weight their decision. We have also developed a set of case specific criteria customized to the Bates and Poole SECTIONS framework for selecting and using technology. This is a comprehensive rubric tailored to UMPs educational ob jectives of their third year Rural Family Practice Clerkships program.

Assessment Rubric:
S.E.C.T.I.O.N.S. Framework Students Is it appropriate for your student population? Criteria
Access: Easy to access from rural areas Performance stability

Not Recommended for Use (1)

Difficult to access from off-campus locations. Functionality (speed and stability) severely affected in low bandwidth areas.

Acceptable for Use (2)

Easy to access regardless of location Performance (including speed and stability) is not ideal in areas with low bandwidth

Ideal for Use (3)

Easy to access regardless of location. Performance(including speed and stability) does not differ based on available bandwidth

Privacy Student and instructor privacy considerations LMS security information

privacy policies inconsistent with university policy or not readily available access unrestricted: no password protection

privacy policies in place, mostly consistent with university policy password protection included but not encrypted limited access (students, faculty, help desk staff) Interface is user friendly and relatively intuitive to learn Customization options available Available on at least 2 major mobile platforms Limited support available, may be during limited periods of time Limited self-help capabilities.

privacy policies consistent with universities password protected and encrypted restricted access: users must be granted access User friendly and intuitive to learn quickly (within first 20 min of use) Easily customizable Available on all major mobile platforms A variety of options for both instructor and student support is offered Full self-help functionality built-in to the system Cost of LMS design and implementation timeframe is within UBC budget Managing video files is effortless. User impressed by fast upload times. All the major video file formats accepted Smooth video playback is unaffected by number of users. Video annotations easy to add and edit Interactive annotations available (access to further information & extension of knowledge.

Ease of Use Is the interface and design reliable and intuitive to the learner?

User Friendly Easy to use and customize Supports mobile use Available learning supports

Interface is confusing, hard to navigate, and uncomfortable for the novice user Limited customization options Not available for use on major mobile platforms Minimal support available. No self-help capabilities and the user manual is nonexistent or is only available on hardcopy. Cost of LMS design and implementation timeframe does not meet restrictions of UBC budget Difficult to manage video content (uploading and/or deleting video files) Uploading video content is onerous and/or frustrating Only 1 video file format accepted Delays in playback experienced. Video quality adversely affected by number of users Annotation of video content not supported and/or difficult to view and/or edit.

Cost Is the production cost reasonable? Teaching and Learning Will it enhance learning?

Budget UBC budget considerations Functionality Ability to upload and manage large video files of various file formats Ability to have several learners accessing video content simultaneously Ability to annotate videos for assessment

Cost of LMS design and implementation timeframe falls slightly outside of UBC budget Relatively easy to upload video content (within bandwidth constraints) Decent upload speeds Restricted to 2-3 video file formats. Playback of video is generally smooth but playback may be negatively impacted during peak viewing periods (ex. slight delays). Annotations easy to add and edit, thus supporting efficient feedback process

Interactivity Does it move the learner beyond just reading, viewing and listening?

Synchronicity Synchronous and asynchronous interaction Integration of multimedia available Facilitates community building

Interaction is limited scheduled times to view and/or add content. Multimedia objects not supported Educational artifacts are accessible only via links which take user outside of LMS. Only one way feedback is supported (instructor to student)

Asynchronous interaction only. Scheduling flexible. Several multimedia objects and artifacts can be embedded Some artifacts accessible via links which take user outside of LMS. Two-way feedback and dialogue encouraged between students and instructors.

Synchronous and asynchronous interactions afforded. Scheduling is flexible. Variety of multimedia objects and artifacts can be embedded Multi-way dialogue and feedback supported between students/students and students/instructors

Organization Is an effective organizational system in place? Novelty Does it appeal to students Speed Is it quick to implement and update?

Stage Length of time software has been in existence Other institutional use Student engagement and motivation

Very few active institutions currently using or LMS remains under beta testing. Most of the main features still under development or non-existent. Content still heavily reliant on text. Student opinion research not overly positive Laborious set up process No automated update feature. Updates rarely available (average of once per year)

At least 2 versions of software being actively used (educational settings) Most of the main features are complete and functional. Content promotes some active participation. Student opinion research generally positive Easy to set up, but can be time consuming Some automated updates but requires manual intervention. Updates created on a quarterly basis.

LMS is an industry leader and has a large number of active users in the educational field. Platform and includes advanced features Content promotes active participation. Student opinion research positive Set up process can be completed easily in minimal time Fully automated update system Updates are created on a weekly basis.

Updates System set up and maintenance

Rationale for Rubric: In a dynamic world where technology is rapidly altering learning environments, there is a greater need for seamless communication across sites. For this reason it is important that UBCs Undergraduate Medical Program develop an appropriate strategy for selecting an LMS. Not only will the LMS facilitate video-based clinical assessment across a number of sites, but it will also be expected to support multimedia objects and discussion forums in addition to other educational artifacts. A rubric is an effective tool for collecting information deemed relevant to the decision making process. As Bates and Poole (2003, p 80) suggest: ....teachers or educational administrators making decisions about educational technology should have some theoretical model or framework that guides the choice of media and technology. If not, they will be constantly driven by the latest technology development, whether or not they are appropriate. Our rubric was designed based on the prioritized feature-set as identified in the case study. SECTIONS was chosen as a framework for the assessment rubric due to comprehensive nature. Rubric questions were developed to correspond with the factors under consideration, and may be useful to stakeholders such as (1) administration involved in LMS selection, (2) instructors who will determine specific interactivities required, and (3) students who will be engaged in the knowledge building interactivities to be contained within the LMS. The Year 3 Rural Family Practice

Clerkships give 3rd year medical students an opportunity to spend 4 weeks apprenticing under the direction of a family physician in rural and underserved community practices. While the universities involved are in Vancouver, Prince George and Victoria, the students may be placed in Vernon, Chilliwack or Trail, BC. Communication across sites needs to be efficient and seamless. Other specific considerations in our scenario include: Medical students and their faculty will have a heavy workload, and as such reliable playback of video with minimal download waits is imperative. As students and faculty will be reviewing videotaped examples of clinical skills containing images of patients, privacy is critical. Medical students, like the general population, may have variable technology skills. As such, the interface needs to be intuitive and easy to navigate and not be a barrier to enhancing clinical practice/reflection skills. Capabilities for mobile use are critical as these students will be taking the video on the worksite, away from a computer. It seems likely that the video will be taken via cell phones or tablets and uploaded to the LMS site. In sum, we feel that we have created an effective rubric to assist administration, faculty and students in evaluating LMS features to facilitate the learning interactivities of Year 3 Rural Family Practice Clerkships.

References: 2010 Summer - MD Undergrad Education, UBC Faculty of Medicine (2012). Retrieved January 31, 2014, from http://mdprogram.med.ubc.ca/files/2012/02/2010_Summer17229.pdf. Bates, A.W. & Poole, G. (2003). Chapter 4: a Framework for Selecting and Using Technology. In Effective Teaching with Technology in Higher Education: Foundations for Success. (pp. 77-105). San Francisco: Jossey Bass Publishers. Trail | MD Undergrad Education, UBC Faculty of Medicine. (2013). Retrieved January 31, 2014, from http://mdprogram.med.ubc.ca/programinformation/integrated-community-clerkships/trail/. Year 3 | MD Undergrad Education, UBC Faculty of Medicine. (2013). Retrieved January 31, 2014, from http://mdprogram.med.ubc.ca/year-3/.