Evaluation of Moodle Course Management System for the World Languages Program at Drake University

Presented by: Clayton Mitchell

Summary The World Languages & Cultures (WLC) program at Drake University primary mission is to help students gain a functional competence in a language other than English. The WLC offers instruction in seven languages, Arabic, Chinese, French, German, Japanese, Russian and Spanish. The average student enrollment for all languages is around 200 per semester with slight variations depending on semester and year. The program has three fulltime faculty in Spanish and Japanese combined, and a fulltime program director who teaches an intercultural communication course. The remaining language faculty are adjunct instructors. For this evaluation the WLC faculty were surveyed about their opinions regarding the use of technology in the classroom particularly looking at the Moodle learning management system. The scope of the evaluation was to see if Moodle is still meeting the needs of the WLC faculty and the program in general. The evaluation was not intended to suggest replacement for the Moodle LMS, but only to evaluate if its continued use was warranted. A short meeting discussing evaluating alternative LMS solutions with the faculty was conducted. The faculty showed little interest in switching to alternative systems including the Blackboard, which is supported by the campus at large. An interview was conducted with the director of the WLC program in which he outlined objectives by which to evaluate a learning management system. The evaluation found that the WLC faculty are generally pleased with Moodle and do not see any need to make a change at this point in time. An additional finding was that the objectives that were set out by the director are not in line with what the faculty believe are the core components of an LMS. Program Description Because the WLC program places an emphasis on speaking, the program conducts classes in a manor that is not typical. Students studying a language meet with their professors twice a week in addition to working with a native speaker that acts as a speaking tutor one hour per week (the practice session). Classes are comprised of no more than 5 students per practice session, with many being no more than 3 students in the higher levels. During these practice sessions, students are not learning about the language, but practicing the speaking skills they are trying to master. English is not spoken during these practice sessions. There is also not supposed to be any teaching of grammar, and very little explanation about why a particular language is spoken the way it is. The native speakers are comprised primarily of foreign national students that are attending Drake University seeking degrees other than languages. They are not teachers and are not trained in language acquisition. These practice sessions allow the students to participate in an immersive session that is a unique experience to most of the student body. The class sessions that the students meet in twice per week with the language faculty are comprised of up to 20 students from different sections at the same level. The language levels are broken down into first, second, third, and forth semesters. Instruction above the forth semester is topically based depending on the language of study. Each language level follows a common syllabus across sections. All sections of a given language level move through the course at the same pace. Because of this boutique style of instruction, having a communication method that an instructor can use to disseminate and collect assignments, give feedback and keep all sections organized is tantamount.

The WLC has been using the Moodle LMS for the past six years. When the WLC began using Moodle, there really were no social media outlets that students were using and maintaining a stand alone LMS made the most sense. With the rise of stable online alternatives to the traditional LMS and the prevalence of social media platform use by the student population, the WLC felt it was prudent time to evaluate the continued use of Moodle. The reasons given by the WLC director are as follows: Reasons for looking at new platforms 1. Unified communication with students 2. Common element to unify program 3. Opportunity to allow students to maintain existing digital identities 4. Want to move to a more open digital environment. Remove walls placed by traditional LMS. 5. Potential Social network tie in. 6. Wanting to move from a course centric to a student Personal Learning Environment. Evaluation The WLC program is relatively small and because of this all fulltime faculty were surveyed. There were a total of 5 responses, three from the fulltime faculty, one from the director and one from the Chinese faculty member who had a one year appointment. The polling was done early in the summer semester as all of the faculty were either leaving after their contract expired or would not be responsive to survey because of travel and research obligations. The results of the survey can only show the attitudes of the faculty about using one online instructional platform, Moodle. Other platforms were not directly compared to the current system. Based on the results from the respondents some information can be inferred about the desire of the faculty in using alternative systems and the facultys comfort level of switching to a new system. The survey was given in the final few weeks of the spring semester as this was the only time that faculty were available. All of the faculty had completed their coursework for the semester when the survey was taken. The survey consisted of 14 questions with 4 questions using a 5 point likert scale 2 using a 7 point scale one using a 3 point scale with the remainder using short answer and ranking. Along with the survey, one-hour meeting was held with the faculty in which the topic of testing alternatives to Moodle course management system was discussed. As a follow up to this meeting, an interview was conducted with the director of the WLC program in which he offered up a set of criteria by which to evaluate any LMS including the current system. Results Of the 5 possible participants in the survey, all 5 responded. Each response showed that technology and in particular Moodle is important to the faculty and their teaching. The most significant results came from direct questions about technology and its perceived usefulness.

In your courses, how important is the role technology plays in achieving learning outcomes?
# 1 2 3 4 5 6 7 Answer Not at all Important Very Unimportant Somewhat Unimportant Neither Important nor Unimportant Somewhat Important Very Important Extremely Important Total Response 0 0 0 0 1 4 0 5 % 0% 0% 0% 0% 20% 80% 0% 100%

Please describe how you think your students feel about the technology you use in your course.
# 1 2 3 4 5 6 7 Answer Very Useless Useless Somewhat Useless Neutral Somewhat Useful Useful Very Useful Total Response 0 0 0 0 1 4 0 5 % 0% 0% 0% 0% 20% 80% 0% 100%

What are the 5 most critical things your courses must have? Please place in order of importance.
1 Textbooks upload things for students to read 2 Moodle and websites ability to provide meaningful feedback 3 Quizzes 4 Assignments and projects 5 Blog

robust gradebook Integrated tools to participate in lectures, questions, answers, and forums with members outside our campus Blog View and create video

Integrated information: assignments, calendar, instructions

Integrated tools to create videos and interact with other participants

Tools to create adaptive materials such as quizzes and practice exercises

Tools to create adaptive materials: grammar, reading, vocabulary

Moodle

Online quizzes Listen and create audio

online conference tool Threaded discussion boards Online grammar and vocabulary practice

LMS

Note that Moodle or LMS came up in the top 2 answers for 3 of the participants. This was surprising given the open ended nature of the question. The remaining questions all had responses in the affirmative, see appendix for survey questions. In addition to the survey, the meeting was conducted with the faculty in which the topic of the continued use of the Moodle LMS was discussed. The faculty unanimously stated that they did not see any need to look at alternative solutions, at this time. Of note was their inclusion of not moving to the updated Blackboard LMS that the campus has recently adopted. This is of importance because the move to a more widely supported system would have better continuity across campus and expanded support for their classes. The main reasons given were that Moodle is currently meeting their needs and that they were not convinced that the time and efforts that they have placed into the Moodle LMS would not have to be duplicated if a move to a new system were to occur.

Descussion The intended purpose of the survey was to see if Moodle was currently meeting the WLC faculty needs and/or illustrate any limitations of the current platform. The results of the survey show that overall the faculty are satisfied with Moodle and the current LMS offerings. The results also show that as a whole the faculty are not interested in using tools that extend beyond an LMS platform. While questions related to the use of tools external to an LMS or on the use of

social media were not directly asked in the survey, the open ended possibilities of the questions would have allowed the faculty to express this desire if they were so inclined. Additionally these topics were not broached during the meeting with the faculty indicating little or no interest in the subject. None of the faculty mentioned the lack of communication or need for a social media tie in within the LMS. The responses indicate that the faculty would actually like the system to offer greater control of how the student moves through the required course. The evaluation does have several shortcomings that should be addressed in order to get a more complete picture of the use of technology for all participants within the WLC. Firstly the evaluation only speaks from one perspective, that of the tenure seeking faculty of the program. The adjunct faculty were not surveyed and their opinions may differ significantly from those stated by the respondents. Secondly, no students were surveyed. This is leaves out the largest stakeholder of the WLC program. If student attitudes about the current use technology were known, this may be reason for the faculty to consider an alternative. Lastly, the survey was given and the meeting was held at the very beginning of the process and no additional formative evaluation could take place as the faculty were not available beyond their initial input. Follow up interviews could have been conducted with the faculty to clarify their positions to the answers given. Additional questions could have been asked that directly related to the use of alternative tools in light of the criteria devised by the director of the WLC. The criteria devised while working with the director of the WLC to help evaluate the needs of the program LMS are as follows. LMS Criteria 1. Adhere to new content and learning theories. 2. Mobile compliant 3. Be able to integrate current and future web 2.0 tools into platform a. Minimum look integrated b. More integrated the better 4. Single communication for multiple interfaces a. SMS b. Email c. Text d. Twitter e. Facebook 5. Moderate quiz abilities 6. The ability to distribute and receive audio and video material 7. No more than $2-3000 per year to operate 8. (Not a must have) Lecture capture/distance attendance 9. Analytics of student activity in the courses When looking at the above criteria it is clear that Moodle currently meets most of those stated. There are some criteria where Moodle falls short like in the analytics of student activities. Note that the criteria for the evaluation of an LMS and the responses given by the faculty are not in direct alignment. None of the faculty mentioned the need to for mobile devices, or communication outside of the forums and email. This speaks to the need for the WLC program to have an internal dialog about what the program actually needs and what the priorities of the program are. One of the possible reasons for the disconnect between the criteria and the faculty

responses could be the perspective between the director and the faculty. The director of the WLC was looking at the future (3 to 5 years) of the program when developing these criteria. The faculty responses indicate that they were thinking of this evaluation process in terms of the classes they have taught or will teach in the near future. These criteria may be used as a sort of roadmap for continued evaluation of an LMS solution. The inclusion of the students in this process can not be overstated.

Project Cost Invoice

Date: 7/24/2012 INVOICE # [100] TO: World languages & cultures Drake University Clayton MItchell ESC 2820 University Ave Des Moines IA, 50311 515 271-2867

Salesperson Job Clayton Mitchell Number of days 15 Evaluation of Moodle as a viable LMS platform

Payment Terms Due on receipt Daily rate $500

Due Date

Description Number of days worked on the project

Line Total $7500

Total

$7500

Appendix Survey questions Please rank the choices (By clicking and dragging) in terms of items you use most often to least often in your courses. Think of your answers in terms of what the students are required to use in your course. ______ Creating Audio ______ Quizzes ______ Written assignments ______ Gradebook ______ Creating Video ______ Attendence ______ Rubrics ______ View Videos ______ Listen to Audio ______ Wiki ______ Blog ______ Synchronous online communication (Skype, Connect) ______ Handouts (syllabus, readings, PDF documents)

What are the 5 most critical things your courses must have? Please place in order of importance. 1 2 3 4 5

What are the things that you are using that you could do without? 1 2 3 4 5

If you could do without the above items, what would you do instead to achieve the same outcomes?

In your courses, how important is the role technology plays in achieving learning outcomes? Not at all Important Very Unimportant Somewhat Unimportant Neither Important nor Unimportant Somewhat Important Very Important Extremely Important

Please describe how you think your students feel about the technology you use in your course. Very Useless Useless Somewhat Useless Neutral Somewhat Useful Useful Very Useful

Please pick the option the best describes your classes Most of the work is done in my lecture sections Most of the work is completed online Most of the work is completed in the sessions with the native speakers

Are your students prepared when they come to your lecture classes? Never Rarely Sometimes Most of the Time Always

Are your students prepared when they come to the native speaker sessions? Never Rarely Sometimes Most of the Time Always

Do you feel that if your students were to complete the assignments outside of class they would be more prepared for class? Please explain

Do you feel you are effectively communicating with your students? Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree

Do you feel that students understand what is expected from them, with regards to assignments, throughout the semester? Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree

Are students reaching the level of profeciency you expect after completing your courses? Far short of expectations Short of expectations Equals expectations Exceeds expectations Far exceeds expectations

If you could change one thing about the way you deliver your courses what would it be?