Etec 613 Formative Evaluation Draft 8

Running head: HOW TO WRITE A SELF-REFLECTION
Formative Evaluation How to Write a Self-Reflection Based on a Biblical Message

Eric Hermosura ehermosura@gmail.com Macy Huynh mjhuynh@hawaii.edu Susan St. John sstjohn@hawaii.rr.com
Submitted to Catherine P. Fulford, Ph.D. in partial fulfillment for course requirements for ETEC 613 2011 Spring Semester May 4, 2011 I certify that I have contributed 33.33% of the effort to the conceptualization, information collection and analysis, and writing of this paper. If asked, I will be able to succinctly describe any portion of the paper. I understand that I am responsible for all of the contents and will be evaluated for the total project.
______________________
______________________
______________________
Eric Hermosura
Macy Huynh
Susan St. John
HOW TO WRITE A SELF-REFLECTION Abstract Ninth grade students attending San Marciano Catholic High School are expected to do well in their religious studies courses. Students are already proficient in their reading comprehension and English writing skills. They are also expected to do particularly well in the area of writing self-reflections. Students at the Catholic school must have the proper tools to guide them towards constructing their own spiritual and theistic understanding. This module is intended to guide students towards writing a spiritual self-reflection, as well as, helping them to understand and to connect with Bible stories, such as the story of Job. Ultimately, the designers hope that students are able to write a self-reflection and to be able to connect with big themes, such as the human condition of suffering. Although the module did well to encourage students to select meaningful and personal experiences for their writing exercises, it was difficult for students to connect and relate to the story of Job and his suffering. This was further compounded by having close consecutive test dates and heavy amounts of writing. Overall, students self-confidence and self-reflective writing skills increased after taking the module. The designers concluded that future modules
ii
should contain passages that are more relevant to 9th grade students, such as love and friendship, so that students are more motivated about the writing. In addition to that, future modifications will also include spacing out the test dates and having more varied, creative test questions, such as fill-in the-blanks and analysis exercises of the components that make a good self-reflection.
HOW TO WRITE A SELF-REFLECTION Table of Contents
iii
Table of Contents............................................................................................................................iii Methodology....................................................................................................................................6 Introduction..................................................................................................................................6 Subjects and Procedures..............................................................................................................9 Analysis and Results......................................................................................................................14 Peer and Instructor Reviews......................................................................................................14 One-to-One................................................................................................................................15 Small Group Demographics.......................................................................................................17 Small Group Data..........................................................................................................................17 Test Analysis..............................................................................................................................17 Anecdotal Data...........................................................................................................................26 Attitude Survey Data..................................................................................................................26 Conclusion.....................................................................................................................................28 Recommended Revisions...........................................................................................................28 Discussion..................................................................................................................................34 References......................................................................................................................................37 Student.......................................................................................................................................68 Question 10................................................................................................................................68 Question 11................................................................................................................................68 Question 12................................................................................................................................68
List of Figures Figure 1: Revised hierarchy chart ...9 Figure 2: Instructional analysis comparing pre-test and post-test scores by percentage...19 Figure 3: Individual scores by objective with summary statistics ................22
HOW TO WRITE A SELF-REFLECTION
iv
Figure 4: Line graph representation of test scores by objective ...25 Figure 5: Bar graph representation of test scores pertaining to change in emotions.....25 Figure 6: Likert scale attitude survey responses in percentages........27 Figure 7: Recommended revisions 30
HOW TO WRITE A SELF-REFLECTION Appendices
Appendix A: Formative Evaluation Plan.......38 Appendix B: Small Group Demographic Survey..............40 Appendix C: Pre-Test Questions...............41 Appendix D: Embedded Test Questions................48 Appendix E: Post-Test Questions..59 Appendix F: Attitude Survey............ 65 Appendix G: Attitude Survey Comments..67 Appendix H: Attitude Survey Summary Statistics....69
HOW TO WRITE A SELF-REFLECTION Methodology Introduction The purpose of this formative evaluation is to examine the effectiveness of the module, How to Write a Self-Reflection. The objective is to teach the cognitive skill of self-reflection, which is necessary for spiritual growth. The designers chose to have students demonstrate their learning through writing. The module is evaluated according to three areas: content, process, and format. Major questions that the designers hope to answer in this report are: Does the module work? That is, do students learn what was intended for them to learn? Did it fit the audience? Was it appropriate to their level, needs and interests? Does it measure what the designers wanted to measure? Is the testing parallel to the instruction, and does it match the content and skills the module is teaching? Instructional goal. The 9th graders at San Marciano High School are required to take two semesters of religion. The purpose of this module is to provide students with the tools to construct their own spiritual identity and theistic understanding. One such tool is the ability to write a personal, spiritual self-reflection. An effective reflection is one that lends itself to spiritual growth. It is a genuine and sincere examination of the human condition. A reflection takes the themes and issues which appear in biblical readings and relates them to the students own lives. Through the act of
writing, students progress from understanding the concepts to using this biblical understanding to solving moral dilemmas of their own. The act of writing takes students through the progression of skills described in Blooms taxonomy: from first remembering, to understanding, to applying, analyzing, and finally, to creating their own stories of spiritual growth.
HOW TO WRITE A SELF-REFLECTION As class time is primarily devoted to reading and discussing biblical passages, a selfinstructional, print-based module will be given to each student to complete on their own. A primary goal of instruction is to show students the components of an effective reflection. The module will contain both examples and non-examples of personal, spiritual reflections. In addition, the module will demonstrate how to analyze and address the reflection assignment or prompt. The student will be able to discriminate between true reflection, which analyzes
experience, and description, which simply recounts experience. The student will be able to make connections between their observations and the philosophical themes and issues which appear in the readings. Finally, the student will be able to develop their own ideology synthesized from their readings, analysis, personal experiences, and observations. In summary, once they complete the module, students will demonstrate spiritual reflection by analyzing their own experiences, connecting prior knowledge of biblical themes to their own life stories and observations, and developing their own spiritual doctrine in a free-write journal format. Target population. The freshmen students of San Marciano have obtained entry level skills of being able to read and write English fluently that is appropriate to their age group. The students are able to write in paragraph form, and have done reflection writing in either their English or religion classes. The students must pass English and religion during the school year; otherwise they will be assigned to summer school. Generally, they are motivated to do well in these classes. Most students have had eight years of English and several religion classes. Some of the students, who have gone to public schools, have only had one prior semester of religion. About half of the
HOW TO WRITE A SELF-REFLECTION students come from a Roman Catholic background, another forth from other forms of Christianity, and the last forth are a mix of other religious, or are non-religious. San Marciano students are interested in topics that they deal with as teenagers, such as body changes, romance and sex, feelings of emotional intensity and risk-seeking behaviors
(Dahl, 2004). They may not have made the connection between their own human condition with the human conditions expressed in the Bible; however, they may be interested in a wide array of topics that one can write about in a reflective journal. The school has an average graduation rate of 99%, and 85% of students continue onto post-secondary education. The value of education is also relatively high since students parents are willing to pay for private school education. Knowing how to write a reflective journal will provide students with a format for expressing and sharing thoughts through writing. The general learning experiences of students at San Marciano are lectures and discussions for the majority of their classes. The percentage of time spent in lectures varies among classes. Description of module. The module is self-instructional and print-based. It will be given to each student to complete on their own during the class time. A primary goal of instruction is to show students the components of an effective reflection. The module will contain both examples and non-examples of personal, spiritual reflections. In addition, the module will demonstrate how to analyze and address the reflection assignment or prompt. As depicted in the revised hierarchy chart in Figure 1, the module is clustered into six main themes: emotions, values, actions, Jobs conflicts, Jobs relationship with God, and finally, the personal reflection. In the first three sections, the learner is instructed to differentiate between changes in emotion, values, and actions that are meaningful and others that are less
HOW TO WRITE A SELF-REFLECTION significant. In themes four and five, learners draw analogies between their own life and Jobs experience. Themes one through five each have one multiple choice question and one shortanswer question. Theme six is a one-page, self-reflection essay.
Figure 10. Revised hierarchy chart.
Subjects and Procedures Design team. The design team consisted of Eric Hermosura, Macy Huynh, and Susan St. John. All members are currently enrolled in the Masters in Educational Technology program at the University of Hawaii at Mnoa. Eric was the teams main subject matter expert because he is a teacher of religious studies for 9th graders. Susan was the subject matter expert in the area of writing. Macy assisted the team with the organization of the project, as well as the editing and formatting of the module. All team members contributed to the content, layout, design, and
10
formative evaluation of the module. Each team member conducted their own individual, one-onone session; however, all members attended the peer reviews and the small group session. Peer and instructor. The instructor and teaching assistant of the course reviewed the design teams content analysis. Additionally, two peers were also assigned to each of the designers. The content analysis contains the following items: the instructional goal, an audience analysis, an instructional analysis, performance objectives, learner motivation, Gagnes Nine Events of Instruction, criterion-referenced tests, and references. The instructor and teaching assistant were given one, bounded copy of the content analysis to share, and a stapled copy was given to each of the six peer reviewers. All reviewers were given approximately a week to review the content analysis and to provide constructive feedback. Each peer reviewer was instructed to check the content analysis against the content analysis checklist. In addition, each reviewer was expected to do the following: make editorial marks regarding spelling and grammar, provide positive feedback on what is working, make marginal notes and suggestions on what is not working, summarize notes and suggestions into a one-page document, and complete a score sheet. Comments from the instructor and teaching assistant were: be mindful of design implications of the language components of the module, such as word connotations; think about the affective dimension during the audience analysis, such as varied religious backgrounds of students; clarify the strategies to be used to raise student motivation; include the story of Job in the module; present more information about how to write a self-reflection; provide better feedback for why good examples are good and why non-examples are bad; develop a better rubric for students to evaluate themselves after the short-answer questions and the self-reflection; teach students on how to evaluate what they are reading in the examples; be more definitive on what the categories--such as emotions, values, actions, and thoughts--mean; use selective
HOW TO WRITE A SELF-REFLECTION perception tools, such as bold, italics, or underlining, to provide clues and information; make sure multiple-choice questions and answers are parallel; make better linkages from the Bible stories to the multiple-choice questions; and use actual Bible passages as examples.
11
Some common suggestions received from peer reviewers were: add more information in the embedded tests, as some of the multiple-choice answer options were very similar and a concern was made to make sure the test met the one hour limit. One-on-one. All members of the design team completed their one-on-one evaluations. Two of the one-on-one subjects were of the same age as the target audience. The third one-onone was a college student in his second year. During the one-on-one, some of the same concerns and observations were expressed by all three of the designers. They were: 1) the module still took more than one hour to complete, 2) there was difficulty with applying Jobs message to a personal experience, and 3) there was difficulty with applying the experience to a personal, meaningful experience. One subject was a female, 9th grade student, who attends a private school on Oahu. She began the module at 7:25 p.m. on Sunday, March 27, 2011, and ended at 8:55 p.m. The designer and subject met in the subjects dining room. The subject is the daughter of one of the designers. The subject did not complete the personal reection as it was getting late in the evening. The module was too long; in addition, she was reading very carefully and making detailed notes to help the designers in their revisions. Another one-on-one was conducted at the school where the test subjects attend. It was done on March 18, 2011, after school at 2:30 p.m. and was completed at 3:35 p.m. The test subject was very anxious about completing the module because it was the last day of school before spring break. The student quickly tried to complete the module and to do as much as he could without having to read all of the directions.
HOW TO WRITE A SELF-REFLECTION The last one-on-one subject was a student in his second year of college. He completed the module with one of the designers via distance, using Skype, E-mail and telephoning. The subject was in the middle of his spring break and was not motivated to do extremely well. The
12
problem was further compounded by the fact that it was taken online and asynchronously. The designer relied on the subject to be true and honest. The student managed to complete the multiple-choices and short-answers within the hour, but did not make it to the final selfreflection essay. Small group. There were a total of thirteen freshmen test subjects that are enrolled in an Old Testament religion class. The pre-, embedded, and post-tests were conducted in the same class, but on different days. The pre-test was given with extra instructions from the teacher because there was no previous instruction given about Job. The story of Job, his relationships, and suffering were the primary focuses of the test questions. On the day of the embedded test, all of the subjects completed the test during the same one-hour class time. All three designers were present during the embedded module testing. The design team observed the students taking the test and made anecdotal notes. Once the test was completed, all three of the group members read, reviewed, and graded the tests. The post-test was administered by the teacher during the same one-hour class time and the test subjects were distributed among different classes. The pre-test, embedded, and post-tests were completed between April 6-21, 2011. All three tests were done as a part of the Old Testament Religion class, where their participation was part of their class grade. The tests were completed in the test subjects normal religion classroom, which is a 40 x 40 foot classroom, with 24 chairs arranged in a U-shape pattern. The class size is between 12 to 20 students. Students were allowed to sit anywhere, including right next to another student. The class is air conditioned at a comfortable room temperature. The classes are about an hour long, and students were given the whole class period to complete each
HOW TO WRITE A SELF-REFLECTION of the pre-, embedded, and post-test. The test environment was casual. Students have spent
13
seven months together, resulting in a level of comfort and familiarity with one another. Students were not reprimanded for occasional or limited talking. However, extensive talking was not allowed, such as speaking beyond 3-4 sentences. As pre-test procedures, students were required to read the module introduction first, which stated that the module would not be graded for the percentage correct, but a class participation grade would be assigned for the completion of the test. It also stated that the test module was to test the design of the module, and not the students themselves. Lastly, the students were encouraged to do their best. It was also verbally conveyed that students would not have all the information to answer all of the pre-test questions correctly. Test directions were then given. There were five multiple-choice questions and five short-answer questions. The pretest concluded with a final reflection essay addressing the following questions: How is Jobs message meaningful to you? How is it similar to your own personal life experience? All of the test subjects were assigned a number once they completed the pre-test, which remained the same for each student during the embedded and post-tests. There were a total of 30 test items, or objectives. A point was assigned to each multiple-choice question, four points to each short-answer question, and five points to the essay section, with a total of 30 points possible. The design team was present for the embedded test. The embedded test was completed on April 12, 2011 from 12:30 p.m to 1:30 p.m. Because this was the last class of the day, students were eager to complete the test and leave. The embedded test was conducted in the same manner as the pre-test, where the module introduction was read, the design team was introduced, the directions were given, and the test was distributed.
HOW TO WRITE A SELF-REFLECTION Unlike the pre- and embedded tests, not all of the freshmen students had to do the post-
14
test. The only students that completed the post-test were the thirteen test subjects. The post-test was given in different classes. Students that were not part of the data collection results were given an alternative assignment. The students who were part of the data collection results expressed some resentment about having to perform three, straight days of testing while other students did not have to. After completing the post-test, the test subjects received a goody bag with candies and cookies as a show of appreciation. The goody bags were made by Macy and Susan. The students were appreciative of the gift and expressed their thanks. The teacher also expressed gratitude for their cooperation and help in assessing the instructional design module. The post-test was given over the course of two days. Because test subjects were the only ones taking the test, they tried to finish the test quickly and did not utilize the time allotted. The second time the post-test was given, students were given the same test they had started previously and had an additional 30 minutes to complete it. Analysis and Results Peer and Instructor Reviews One of the main peer review recommendations concerned time: the module took more than the hour allocated. The design team eliminated the entry-level test questions, where they concerned word connotations and tone. The designers felt that the students are experienced enough in their English skills to be able to identify and distinguish such things as a positive tone from a negative one. The decision to eliminate these test items left the module with one multiple-choice question and a short-answer question for each of the five remaining objectives.
HOW TO WRITE A SELF-REFLECTION One-to-One One subject pointed out that although spiritual reection was mentioned in the
15
introduction, the designers never fully explained what a spiritual reection was. She had many excellent points about clarifying the instructions, such as adding an extra phrase or word to make it more clear for the reader. She also suggested that the formatting and titles needed to be more distinct to mark changes in the themes. She also commented on the change in tenses in the examples, such as the examples were in third-person and students were asked to write in the rstperson. She did not understand the purpose or meaning of some of the graphics, and found some of the graphics distracting. The subject could not think of a personal experience that was similar to Jobs conict with God. The conict in the story is veiled and ambiguous. It was difficult for the subject to relate a personal experience to Jobs. The subject suggested that the module explicitly point out interpretations of the story of Job. For example, in the section on Jobs relationship with God, she suggests that the designers make clear that Job speaks to God, and that God listens. These ideas were suggested in the examples, but were never made explicit by the designers. Similarly, the second and third test subjects were not able to complete the module in the allotted time. Much of the time was spent re-reading the passages for clarification. The subjects stated that re-reading did not allow for the final reflection to be written. In the final reflection, the second subject started the essay, but was not able to complete it, while the third subject never even made it to the final essay. The second subject did not have any experiences that were similar to the relationship between Job and God. He took the relationship literally and not symbolically. Leading up to the final reflection, the test subjects were able to answer the questions about emotions, actions and values, but stated that there were no definitions of those concepts.
HOW TO WRITE A SELF-REFLECTION The design teams one-on-ones were done in less than ideal circumstances. They were
16
done late at night, after school on the day before spring break, and through Skype during spring break. The design team thought that this added to the inability of the test subjects to finish the module. While designers made adjustments to streamline the time, they felt that those adjustments were sufficient. The post-test remained the same because it kept the themes of change in emotions, values, actions, and conflicts. However, the theme of Job experience was changed to accommodate relevance in the final reflection and terminal objective. The biblical readings changed to show Job suffering because of the death of his family. The second part of the readings relates to what Job learns from suffering. The third reading relates to how Job ultimately benefits from his suffering. These readings were more literal compared to the previous readings, which revolved around Jobs relationship with God. The post-test final personal reflection topic changed to: How have you suffered? What did you learn from this suffering? How did you ultimately benefit from going through the experience of suffering, learning from suffering and benefiting from suffering in the end? How did your emotions, values and actions change because of the meaningful experience? Students improved their terminal objective because they were able to connect their personal experience with the reflection topic. The reflections were longer, more thoughtful, and conveyed a more meaningful, personal experience. Students seemed more freely to express their personal experiences, like divorces, deaths in the family, personal problems of family members, and other meaningful life experiences. Because of the intimate nature of writing personal reflections, it was important to the designers to have a suitable writing environment. This was important in regards to the emotional comfort of the teacher, students, and school setting, which ultimately contributed to the test subjects willingness to write and to share their
HOW TO WRITE A SELF-REFLECTION personal experiences. If the setting had been unfamiliar where the test subjects didnt have a
17
feeling of emotional comfort and safety the test subjects would probably have chosen topics that werent as intimate and personally and emotionally meaningful. Small Group Demographics The data from the demographics survey revealed that the majority of the respondents were mostly female, with 10 females and 4 males. Half of the respondents were 14 years old and the other half were 15 years old, which is to be expected for 9th graders. The data revealed that the 9th graders are comfortable with thinking and reflecting about meaningful life experiences and writing in journals. However, when it comes to religious content, very few have heard of the story of Jobs suffering or have read the Bible on their own. There seems to be a connection with this data and the module, as students did express their concern that they could not connect with the story of Job. Small Group Data Test Analysis There were no entry-level tests because it was observed in the peer evaluations and in the one-on-ones that test takers all took more than one hour. In the interest of time, all of the entrylevel test skills were omitted. It was in the entry levels that alternative forms of assessment were contained. There were word matching and word connotations and there was no reading of extensive passages. All alternative forms of assessment were eliminated and the module contained only multiple-choice questions, short-answer exercises, and an essay, which made the module monotonous and repetitive. While this may not have had an effect on the test results, the students verbally expressed a negative, emotional response after having to take the test three times.
HOW TO WRITE A SELF-REFLECTION San Marciano high school has a year-round schedule. The school has two weeks for
18
spring break so the components of the module could not be done during that time. When school started again, it was close to the due dates of module results, and so the design team had to give the tests back-to-back for the pre-, embedded, and post-tests. The students complained about the scheduling. Upon hearing the complaints, the teacher decided not to have all the students take the post-test. Only the students being used for the test data had to take the post-test. These students voiced some level of displeasure in having to take the test three times while their classmates were given another assignment that did not entail writing. Many of the test subjects did not focus while taking the post-test and turned in answers that were significantly shorter compared to the embedded test. The teacher decided to redistribute the post-test at a later class day to avoid three straight days of testing. The test redistribution allowed for half an hour and for students to add to their existing answers. The teacher felt that the initial test scores were not indicative of their learning, but an expression of emotional frustration for taking the test while others did not have to. Upon redistribution of the test at a later time, the complaints were minimal and students seemed rejuvenated to finish their post-test. The designers had a total of 14 test subjects. However, one of the test subjects had to leave midway through the post-test and was not able to finish. While her test score was a respectable 20 out of 30 points, it was a corrupted indication of her progression through the module. For data accuracy, her pre-, embedded, and post-test scores were omitted. The designer were left with 13 test subjects from which to work from. A summary comparing pretest and post-test scores by percentage are displayed in Figure 2.
19
Figure 2. Instructional analysis comparing pre-test and post-test scores by percentage. The summary statistics in Figure 3 details the 30 test questions and are grouped under one of six objectives: change in emotions, change in values, change in actions, demonstrating a personal conflict similar to Job, demonstrating a relationship similar to Job, and finally, the terminal objective of writing a personal, self-reflection. The reflection must demonstrate a personal conflict and relationship that is similar to Job that resulted in a change in emotions, values, and actions. Each of these topics was worth five points: one for the multiple-choice and four points for each of the short answer responses. The terminal objective of writing a selfreflection essay did not have a multiple choice question. The test results are as follows: 23.4 out of 30, or 70% for the pre-test; 26.2 out of 30, or 87% for the embedded test; 27.5 out of 30, or 92% for the post-test. The percentage increase from the pretest to the embedded test is 9%, the percentage increase from the embedded to the post-test is 15%.
20
In examining the line graph in Figure 4, the test results varied across the board. The most important of the test data is the terminal objective of the final self-reflection. The five grading criteria were as follows: conveyed a personal experience; writing length was -1 page in length; showed a change in emotion, values, and actions; showed a connection with the story of Job from the Bible; and was about a meaningful, personal experience. In the pre-test, the five grading criteria in percentages were as follows: 54%, 92%, 77%, 46% and 77%, which averages to 69% overall. The embedded tests result in the five grading criteria were as follows: 85% (31% increase over the pretest), 100% (8% increase), 92% (15% increase), 69% (23% increase), and 77% (0% increase), which averages to 85% with a 16% increase over the pre-test. There were significant increases in four out of the five areas. However, there was not an increase in the area concerning a connection with the Bible story. This is probably due to the aforementioned topic that a one-on-one had found where it was difficult to relate a personal experience to Jobs. The post-test results in the five grading criteria were as follows: 100% (15% increase over the embedded test), 100% (0% increase), 100% (8% increase), 92% (23% increase), 92% (15% increase), which averages to 97%. The successful completion of the terminal objective shows that the thirteen test subjects mastered the objectives. The average increase for the terminal objective was as follows: 69% for the pretest, 85% for the embedded and 97% for the post test. The designers were very pleased with the terminal objective results. A positive response towards the module was also found in the attitudinal survey (see Appendix G and F). From an educational and attitudinal account, the instructional module of writing a reflective journal had a positive impact on students accomplishing the terminal objective.
21
Figure 3. Individual scores by objective with summary statistics.
22
Figure 3 continued. Individual scores by objective with summary statistics.
23
Figure 3 (continued). Individual scores by objective with summary statistics.
24
Figure 3 (continued). Individual scores by objective with summary statistics.
As indicated in Figures 4 and 5, the one anomaly was multiple-choice question 1a in the post-test about change in emotions. The distracter was similar to the correct answer found in the pre- and embedded tests, which 12 of the 13 students answered. This was the only test question that displayed a significant drop in the post-test when compared to the pre- and embedded tests. It was observed that the students were economizing their reading and chose the answer that was similar to the previous tests, even when they were told that the post-test was different.
25
Figure 4. Line graph representation of test scores by objective.
Figure 5. Bar graph representation of test scores pertaining to change in emotions.
HOW TO WRITE A SELF-REFLECTION Anecdotal Data
26
One of the major things that all three designers observed was the frequent turning of test pages. Students were obviously not reading the directions and the reading material. Students would often ask the teacher for clarification, rather than reading the module. Since the module was conducted in a familiar classroom to the students, there was no impact on test scores because of unfamiliarity or discomfort. The students knew one another and there was no obvious interpersonal anxiety between the students. During the small group it was observed in the beginning that students were complaining about doing the test again. The designers thought that the scores would be lower because of their negative attitudes. However, as the module progressed, the students were able to complete the reflective journal. The post-test scores even increased when it was given at a later time. The environment played a crucial role in the success of the module. Students have been in this Catholic school setting for about 7 months. They seemed to feel comfortable sharing a meaningful experience, which made it conducive to journal writing. This observation and students completing the terminal objective in the post-test with 90% accuracy seems to indicate that the environment was ideal for writing a reflective journal. Attitude Survey Data From the responses to the Likert scale questions, the majority of the students (see Figure 6) felt that the module was easy to understand and the level of difficulty was suitable to their abilities. However, when asked if the test questions were well-written and straightforward, the majority had a neutral response.
27
Question
Strong ly Disagr ee 0% 0% 0% 0% 0% 0%
Disagr ee
Neutr al
Agre e
Strong ly Agree 15% 53% 23% 46% 31% 15%
N/ A
Total
1. The lesson was clear and easy to understand. 2. The level of difficulty was about right for me. 3. The length of the lesson was manageable. 4. The examples in the lesson were helpful. 5. I learned a lot from the lesson. 6. This lesson will help me to write more meaningful spiritual reflections. 7. The questions and information in the lesson helped prepare me for the post-test. 8. All of the test questions were well-written and straightforward. 9. I did better on the post-test because I worked through the lesson.
15% 23% 23% 8% 8% 15%
23% 8% 23% 15% 38% 23%
46% 8% 31% 31% 23% 46%
-8% -----
99% 100% 100% 100% 100% 99%
0%
8%
23%
38%
31%
--
100%
0% 0%
0% 8%
46% 54%
23% 23%
31% 15%
---
100% 100%
Figure 6. Likert scale attitude survey responses in percentages. The majority strongly agreed that the examples were helpful, and that the module will help them to write more meaningful spiritual reflections. An apparent contradiction is that the majority had a neutral response when asked if they had learned a lot.
HOW TO WRITE A SELF-REFLECTION There seems to be minimal correlation between the attitude survey and the test data. Although all the students improved in the post-test, 54%, or the majority of students, were neutral when asked if they did better on the post-test because they worked through the lesson.
28
Another example that demonstrates a lack of correlation is the performance of student #6, who was overall dissatisfied with the effectiveness of the module. She had marked dissatisfied on all of the Likert scale questions, and her open-response survey questions also indicated dissatisfaction. However, this students scores on the objective portion of the test improved from 78% on the pre-test to 92% on the post-test; on the pre-test she had left 3 of the open-response questions blank, but was able to complete them on the post-test. Conclusion Recommended Revisions Content. The designers found that the student reflections frequently lacked clarity. Some students made little distinction between the different clusters; their writing and the examples they chose were generalized and vague. Additions to content, such as deconstruction of the examples and non-examples, direct instruction about writing a spiritual reflection, and an explicit definition of spiritual reflection, would sharpen the distinction between weak and strong writing. Student responses could have been more focused if they were given direct instruction; specific key phrases and wordings could have been highlighted to demonstrate the difference between a strong reflection and a weak one. In both the one-on-one and small group evaluations, some subjects found it difficult to connect a personal experience to Jobs experience. The designers felt that the enormous drama of Jobs story, as well as the ambiguity of the storys meaning, made it difficult for modern teens to relate or find a personal connection to the story. Since the focus of the instruction is on
HOW TO WRITE A SELF-REFLECTION writing a reflection, and not on analyzing the Bible story, the designers felt that the module
29
would be better served if they chose a more direct scripture reading. For example, the definition of love in First Corinthians may have been more relevant to teens and their concerns. Process. The major issue discovered was student fatigue, which was most likely due to the amount of writing and reading. This was demonstrated by the length of time it took students to complete the module and the rushed quality of the answers. The designers also observed students skimming quickly through the pages of text. Varying the testing practice, such as asking for scenario or sentence completion, analysis, or scenario revision, would reduce the amount of writing. In addition, by varying the practice, the skills will scaffold and build upon each other. In this way, students will be groomed for the most sophisticated skill, the actual writing of the reflection. The designers also noted some lack of engagement, and one student was disgruntled about repeating the material three times. This could be resolved by spacing the pre-, embedded, and post-tests out over a longer period of time. In addition, the module should be broken into larger chunks of four to five test items. The small chunk size of one test item may have contributed to student fatigue; the frequent switching between instruction and testing modes may have taxed their engagement. Format. The designers observed that several students flipped through the reading, and appeared to be intimidated by the large blocks of text. This could be mitigated by changing the delivery instruments. Videos of the scenarios being enacted, or podcasts of an actor reading the stories might further peak students interest.
30
Component Name Content: Goals, objectives, instructional analysis, test items, strategy, instruction, examples, nonexamples, practice, feedback, etc.
Problem Identified Students seemed to be unclear about what precisely made an example weak or strong.
Changes to be Made Deconstruct examples. Point out why an example is strong or weak; highlight, bold or circle key phrases. Compare and contrast strong and weak examples. Place examples with practice items. Add instruction to writing a personal reflection.
Evidence Student reflections lacked clarity. Also from instructor comments.
Content: Goals, objectives, instructional analysis, test items, strategy, instruction, examples, nonexamples, practice, feedback, etc.
Confusion between instruction and practice; missing instruction on final personal reflection. Students didnt receive direct instruction and had to glean information from the examples. Meaning of Jobs story is ambiguous and difficult for 9th graders to grasp. The scope of his suffering is well beyond their personal experience and was difficult for them to find personal experiences that were analogous to Jobs experiences. Some students were not able to relate to the story and the example.
One subject in the one-toone pointed out that spiritual reflection was not fully and explicitly defined. Also from instructor comments.
Content: Goals, objectives, instructional analysis, test items, strategy, instruction, examples, nonexamples, practice, feedback, etc.
Choose a scripture that is more direct and less ambiguous. For example, the definition of love in first Corinthians.
Students in both the oneon-one and in the small group found it difficult to isolate the meaning of the story. They also couldnt easily grasp the relationship between God and Job and the meaning of the story.
Content: Goals, objectives, instructional analysis, test items, strategy, instruction, examples, nonexamples, practice,
Possibly chose a different scripture that is more accessible for students. Write examples that use common teen
Students in both the oneon- one and in the small group found it difficult to find a personal experience that is parallel to the experiences in the

feedback, etc. Content: Goals, objectives, instructional analysis, test items, strategy, instruction, examples, nonexamples, practice, feedback, etc. Instruction: Some students were unclear about what they were expected to write, and uncertain about the content of a spiritual reflection. experiences. Shift the focus of the module to defining what a spiritual reflection is, and analyzing examples and non-examples. Have skills build upon each other: vary the practice rather than have students write throughout the module.
31
examples and to the story. One subject in the one-toone pointed out that spiritual reflection was not fully and explicitly defined. Some students had difficulty writing, particularly in the section about Jobs relationship with God, as the story and examples were far removed from their personal experience. Also from instructors comments.
Figure 7. Summary of recommended revisions.

Content: Goals, objectives, instructional analysis, test items, strategy, instruction, examples, nonexamples, practice, feedback, etc. Process: Pace, length, scheduling, chunk size, motivation, prerequisite skills, amount of practice, environment, written directions in the tests, instruction or practice, data collection procedures No clear distinction between final brainstorming and writing the final reflection. Clarify the purpose of the last brainstorming activity. From instructor comments.
Length: Amount of reading intimidated some students.
Instructions should be shortened and made more explicit. Readings should be shortened.
Small group students expressed amazement at the size of the module. Students flipped through the readings and just began writing. Students observed to be skimming through readings, particularly the introduction and examples. The student who scored highest took 1 3/4 hours to complete. Many answers seemed rushed and demonstrated little reflection. Also, from instructor comments.
Process: Pace, length, scheduling, chunk size, motivation, prerequisite skills, amount of practice, environment, written directions in the tests, instruction or practice, data collection procedures Process: Pace, length,
Length: Amount of writing extended completion time to over an hour.
Limit the amount of writing required. Vary the testing practice; ask students to complete or revise or analyze scenarios or sentences, rather than always creating them.
Students were not
Give the pre-,
Students skimming

scheduling, chunk size, motivation, prerequisite skills, amount of practice, environment, written directions in the tests, instruction or practice, data collection procedures Process: Pace, length, scheduling, chunk size, motivation, prerequisite skills, amount of practice, environment, written directions in the tests, instruction or practice, data collection procedures invested into the process as they knew other classes didnt take the module; they knew it wouldnt figure prominently into their grade. embedded, and posttests to all 80 students. Acknowledge their efforts with refreshments during the posttest (we had given it to them the day after). Space the pretest, module, and post-test further apart so students find it less repetitive. through module; comments in attitude survey.
32
Pre-, embedded, and post-tests were run too closely together.
Some students noted that they could just repeat the answers they had written the day before; some students were initially argumentative about doing the same activity again. Some students expressed confusion about having taken the pre-test. Students observed to be skipping the reading portions. One comment in attitude survey expressed dissatisfaction with the spacing.
Figure 7 (continued). Summary of recommended revisions.
Process: Pace, length, scheduling, chunk size, motivation, prerequisite skills, amount of practice, environment, written directions in the tests, instruction or practice, data collection procedures Process: Pace, length, scheduling, chunk size, motivation, prerequisite skills, amount of practice, environment, written directions in the tests, instruction or practice, data collection procedures Format: Spelling, grammar, vocabulary, table of
Some problems identified in the one-toone were not remedied because of limited time.
Run the one-to-one earlier so that there will be time for more extensive revisions.
Some comments in the one-on-one, for example, the lack of definition and example for the term spiritual reflection, required a more comprehensive revision, which we didnt have time for.
Chunks were too small (1 item each).
Chunks should be 4-5 questions long. Put all of the questions at the end of the cluster.
Researchers observed lack of engagement from students; may have had difficulty switching from reading to testing several times throughout the module. Also from instructor comments.
The module was too dependent on reading.
Add multimedia such as audio or video. Scenarios
Students flipped through the reading.

contents, page numbers, placement of practice or feedback, color, symbols, media, etc. could be dramatized or read by an actor.
33
Format: Spelling, grammar, vocabulary, table of contents, page numbers, placement of practice or feedback, color, symbols, media, etc. Format: Spelling, grammar, vocabulary, table of contents, page numbers, placement of practice or feedback, color, symbols, media, etc.
Instruction was visible when students were being tested.
Put instruction on the previous page.
One of the students reflections was very close to the examples, and may have been following the examples too literally. Also from instructor comments.
Lack of distinction between sections (for example, between change in values and change in action).
Clarify changes in sections by highlighting, bolding, or circling key words. Reword so that the key words are more prominent.
Indistinct differences between sections in students brainstorming. Also from instructor comments.
Format: Spelling, grammar, vocabulary, table of contents, page numbers, placement of practice or feedback, color, symbols, media, etc.
Numbers in table of contents were not aligned in right margin.
Use decimal tab to line them up.
From instructor comments.
Format: Spelling, grammar, vocabulary, table of contents, page numbers, placement of practice or feedback, color, symbols, media, etc.
Gray boxes with Theme to Ponder not carried throughout the module.
Use the gray boxes consistently to set off quotes.
Format: Spelling, grammar,
Layout of self-checks are crowded.
Add a space after the checkbox.

vocabulary, table of contents, page numbers, placement of practice or feedback, color, symbols, media, etc.
34
Discussion One of the challenges is that in a K-12 setting is that one subject usually overlaps aspects of another subject. While the testing environment was a religion class, reflective journaling requires knowledge of English to implement. The designers did not have knowledge of these students performance in their language arts courses. Without formal research, the designers thought that the test subjects would be able to write two pages of journal writing in one hour. The SME, the religion teacher, has had the students for seven months, and had previously given them essay tests of this length. The test subjects would have been able to do this, but three consecutive days of testing taxed their stamina. In future K-12 instructional design modules, designers should take into account the other academic subjects that the module can address or are dependent on. In a design study it is important to know the audience as thoroughly as possible by conducting extensive focus groups. In the one-on-ones, the designers observed their actions and asked what they thought at the end. In the future, the designers will consider interviewing the subjects as they work through the module. This may result in more specific and accurate data collection.
35
In examining the post-test scores, the main anomaly in the test results was 1a, which had only one student choosing the correct answer. Upon examination of the test question it was observed that the distracter that the students chose could be reasoned to be correct. This multiple- choice answer was confusing and the differences between the choices were miniscule; this was the only anomaly in the test results. When the module test was given to all of the religion students at San Marciano, the SME noticed several students reflecting for a while after reading the examples. When asked why they werent writing, students said that they could write something, but that it wasnt meaningful. After looking at their results, the designers realized that more time should be allocated for reflection. Reflection implies being given time to reflect. This assignment was given an hour time limit; however, if time is not constrained, then more time would be ideal. While the classroom environment was ideal for structure and a pen and paper module, under different circumstances it would be ideal to let them reflect in a forest surrounded by solitude and nature to optimize the reflective writing environment. There may be a discrepancy in the attitude data due to the wording of the questions. Students may be comfortable indicating their personal experiences (they found the module easy to understand) but not comment on the writing or the craft of the module. They may be hesitant to judge the technical qualities of the module, especially if their responses reflect on your teacher. Another apparent contradiction in the attitude data is that the majority of learners agreed that the examples were helpful, yet the majority refrained from a negative or positive response when asked if they had learned a lot. Perhaps this question is too general, and it is difficult for
HOW TO WRITE A SELF-REFLECTION students to gage how much they have learned, whereas what they are now able to do is much more easily defined for them. Although all the learners improved in the post-test, they didnt seem to recognize that they did improve. This may be due to the effect that Dr. Fulford has mentioned: once learners
36
have completed a module they are more cognizant of what they dont know and they judge their performance with higher standards and expectations. Interestingly, as evidenced by learner #6s responses, it is possible to learn even if you might be dissatisfied or critical of the effectiveness of the learning material.
HOW TO WRITE A SELF-REFLECTION References
37
Dahl, R. E. (2004). Adolescent Brain Development: A Period of Vulnerabilities and Opportunities. Keynote Address. Annals of the New York Academy of Sciences, 1021(1), 1-22. doi:10.1196/annals.1308.001
Gagne, R., & Wager, W., Golas, Katharine C., Keller J. (2005). Principles of Instructional Design. (5th ed). Thomson-Wadsworth.
38
Appendices
HOW TO WRITE A SELF-REFLECTION Appendix A. Formative Evaluation Plan Test Population The test population consists of the following learners: 1. Three dyads or three individual students, roughly 13 to 15 years of age. These dyads or individual students will test our module. Their input and comments will be used to revise our module prior to administering our module to the small group of 15 learners. 2. A class of roughly 15 to 17 students, roughly 13 to 15 years of age. These students are currently in the ninth grade religion class at San Marciano High School. Strategies for Meeting Times The pre-test, module and post test will be presented during the students normal religion class time. The normal class time lasts one hour and the school operates on a six day cycle. During that six day cycle, the students have religion class three times. Timeline
39
The instruction will be implemented in three separated class sessions. The module will start on April 6 and 7. The post test will be administered on April 14 and 15. Types of Data Collected The types of data that will be collected will be student demographics; student responses from all pre-, embedded-, and post-tests; attitudinal surveys that will be administered after the meeting; any marginal notes that students make in their module; interviews; and any observable student behaviors during the instructional meeting time. Types of Evaluation We will gather both quantitative and qualitative data. The scores from the pre-test, embedded test and post-test will be quantifiable. Student feedback in the form of interviews, written comments including margin notes, and oral comments during the module will be recorded and analyzed. In addition, the researchers will observe and take field notes during the administration of the module during the small group testing. Methods for Gathering Information Data will be collected from the pre-test, embedded test, and the post-test. Each student will have their own module in which they can write comments on. In addition, the researchers will observe test participants and take notes on student comments or areas of difficulty. Script for Describing the Project Hello class. Ive mentioned throughout the course of the year that I am a student just like you, but I am getting my Masters in Educational Technology. In one of the classes that I am taking, we are learning how to design instructional lessons. There are three of us working on the instructional lessons, Mrs. St. John, Ms. Huynh and myself are creating a instructional lesson
HOW TO WRITE A SELF-REFLECTION that is designed to help students how to write a reflective journal inspired by the book of Job. This module is not to test you as students, but to test us and how well we designed the instruction.
40
In the next few weeks you will be learning about the Book of Job and how to write a reflective journal using your past experiences. We will be giving you a pre-test before we begin to see if we correctly assessed what you already know, you will then go through the instruction. After the instruction we will be giving a post test to see what you learned after using our instruction module. Your scores do not affect the grade you will be receiving in class and if you do not do well, it is the fault of the lesson design. All we ask is that you try your best and concentrate to the best of your ability, since reflective writing is key to developing writing skills. You are helping us design our instruction and will have the opportunity to grade us on how well the instruction was designed. We will use your answers to help us make our instruction better for other freshmen students. Thank you for your participation. Lets begin by meeting the thinker that will help you along the way.
41
Appendix B. Small Group Demographic Survey How to Write a Self-Reflection Demographics Survey Your answers to this survey will be anonymous. The information will be used for the purposes of our instructional design project for the Masters in Educational Technology program at UH Manoa. Thank you for your participation! 1. Gender (circle one): Male (4) 2. Age: 15 (7) 14 (7) Female (10)
3. How many times have you heard the story of Job? (circle one) 0-1 (10) 2-3 (3) 4-5 (0) 6 or more (1)
4. How many years of religion classes have you taken? (circle one) 0-1 (3) 2-3 (6) 4-5 (1) 6 or more (4)
5. How many times have you read the Bible on your own? (circle one) 0-1 (9) 2-3 (1) 4-5 (1) 6 or more (3)
6. Have you ever written in a diary or free-write journal? (circle one) Yes (12) No (2)
7. Have you ever thought and reflected about experiences that are meaningful and recorded the experience in a diary, journal, poem, song or art? (circle one) Yes (12) No (2)
8. How comfortable are you with writing self-reflections? (circle one) very comfortable (1) comfortable (13) not comfortable (0) not at all comfortable (0)
42
Appendix C: Pre-Test Questions Theme 1: Identify a Change in Emotions Choose from the following scenarios to answer the question below. Emotion A: When I got a bad grade for a test, I remember being angry at the teacher because I thought his test was unfair. Emotion B: After I got my terrible haircut that made me look like a poodle, I remember thinking how horrible my hairstylist was, so I didnt tip her. Emotion C: After being heartbroken for the first time, I felt like I was not myself and I hated my boyfriend for what he said. Emotion D: After opening her birthday presents, she thought about shopping because she did not get the gift that she wanted. 1. Which scenario shows a change in emotion? A. Emotion A B. Emotion B C. Emotion C D. Emotion D Demonstrate a Personal Change in Emotions Think of a time when your emotions changed because of a meaningful experience. Describe this in 2-3 sentences. Strong Example: After seeing my first dog getting hit by a car. I remember feeling extreme sadness at the loss of my dog. Weak Example: After dropping my plate on the floor. I remember thinking that I need to get another plate of food. Self-Check: Are your 2-3 sentences: in the first person narrative (uses I)? relaying a personal experience? describing meaningful personal change in emotions?
HOW TO WRITE A SELF-REFLECTION Theme 2: Identify a Change in Values Choose from the following scenarios to answer the question below.
43
Values A: After being told about how a boy cheated on his girlfriend, Ive thought that guys can be immoral. Values B: After seeing my Mom struggle with her health because of her eating habits, Ive vowed to make exercise an important component of my life. Values C: After speaking to the mobile phone salesmen, I was thought that I will need an iPhone. Values D: After hearing some amazing live jazz, I wanted to download some contemporary jazz music off of iTunes. 2. Which scenario shows a change in values? A. Values A B. Values B C. Values C D. Values D Demonstrate in Writing a Change in Values Think of a time when youve had a change in values because of a crucial life experience. Describe this in 2-3 sentences. Strong Example: Ive never valued the arts and personal expression. However, after using art and personal reflection as a way to deal with my parents divorce, Ive valued the enormous importance of art and personal expression to get through troubling emotions. Weak Example: I never used to value pets and animals. However, after being around my friends dog, I thought that I wanted a dog. Self-Check: Are your 2-3 sentences: in the first person narrative (uses I)? relaying a personal experience? describing meaningful personal change in values?
HOW TO WRITE A SELF-REFLECTION Theme 3: Identify a Change in Actions Choose from the following scenarios to answer the question below. Action A: He would always go to McDonalds to eat whenever he had the chance. However, after seeing the movie Super Size Me, his love for hamburgers lessened. Action B: She would go on Facebook everyday. However after seeing negative messages, she thought she shouldnt go on Facebook as often.
44
Action C: He never exercised. However, after seeing his Mother die of a heart attack, he started to exercise for an hour a day. Action D: She never travelled to Asia before. However, after watching a Japanese Manga on the Cartoon Network, she had an appreciation of Japanese drawing. 3. Which scenario shows a change in action? A. Action A B. Action B C. Action C D. Action D Demonstrate in Writing a Change in Actions Think of a time when you had a change in actions. Describe this event and how your actions changed in 2-3 sentences. Strong Example: I never used to do any volunteer work. However, I travelled abroad to Peru and saw the poor living conditions of children around the world. Since coming back Ive spent my free time trying to raise money for charities that help poor children. Weak Example: I used to go shopping whenever I got paid. However, after seeing some homeless people going through the garbage can I thought about whether or not I should give some money to the homeless person. I then thought that they should get a paycheck like I did. Self-Check: Are your 2-3 sentences: in the first person narrative (uses I)? relaying a personal experience? describing meaningful personal change in actions?
HOW TO WRITE A SELF-REFLECTION Theme 4: Jobs Conflicts Choose from the following scenarios to answer the question below.
45
Conflict A. He didnt believe the warnings and threats from his father, so he was surprised when his father finally took away all of his privileges. Conflict B. She was a good daughter and always obedient, so she didnt understand why her mother left her. Conflict C. She didnt care about what her parents wanted of her--there was nothing they could do or take from her that would change her. Conflict D. He didnt respect his parents or their values--he thought they were foolish and irrelevant to his life. 4. Which conflict between the parent and child is similar to the conflict between God and Job? A. Conflict A B. Conflict B C. Conflict C D. Conflict D Answer on page 13. Demonstrate a Personal Conflict Think of a time when you had a conflict with a parent, teacher, or guardian that was similar to Jobs conflict with God. Describe this time. Strong Example: I began 6th grade really wanting to do well, but my social studies teacher, Ms. Tanaka, always gave me low marks on my essays. The other kids said that the Tank just had it in for me and I should complain to the counselor, or have my parents complain to the principal. Instead, I tried to listen carefully in class and to do my best on all of my assignments. One day, Ms. Tanaka called me up to her desk and said that she recognized how hard I tried, and she appreciated my attitude. I remember feeling proud of myself and, looking back, I realized how much I learned that year in her class, not just about history, but about myself. Weak Example: I always ask my dad whenever I want something because he usually says yes. My grandmother, my dads mother, complains that I have him wrapped around my finger, but I know that my dad just loves me and wants me to be happy. As far as conflict goes, I dont have much conflict with my dad. I have a little conflict with my grandmother who lives with us, but it doesnt really matter what she says since it is our house.
HOW TO WRITE A SELF-REFLECTION Self-Check: Is your paragraph: in the first person narrative (uses I)? 2-3 sentences long? relaying a personal experience? describing a conflict that is similar to Jobs conflict with God ? Theme 5: Jobs Relationship with God Choose from the following scenarios to answer the question below. Relationship A. He told a lie to a friend, and was surprised when his father found out and grounded him. Relationship B. When her mother embarrassed the girl in front of her friends, the girl knew her mothers weaknesses and how she could get back at her. Relationship C. He loved his mother, even when his mother was angry and punished him. Relationship D. She knew that if she just behaved the way her father wanted her to, he would give her everything she wanted. 5. Which relationship between the parent and child is similar to the relationship between God and Job? A. Relationship A B. Relationship B C. Relationship C D. Relationship D Answer on page 13. Demonstrate a Relationship Think of a time when your relationship with a parent, teacher, or guardian was similar to Jobs relationship with God. Describe this time. Strong Example: Once at recess I got into a fight with a boy. The principal called me in, and then sat quietly and listened to my whole story, and then sent me back to class. I still did get punished, but I loved that principal for listening to me. Weak Example: Id say that God and I are not on speaking terms. Sometimes I asked for important things, like wanting my grandma to get better, or my dog to live. God didnt answer me, or maybe he just didnt listen to me. In any case, Im not talking to Godwhy should I?
46
HOW TO WRITE A SELF-REFLECTION Write your paragraph on the lines below: Self-Check: Is your paragraph: in the first person narrative (uses I)? relaying a personal experience? describing a conflict that is similar to Jobs conflict with God? describing a relationship that is similar to the relationship Job had with God during his struggles? Theme 6: Personal Reflection Write a spiritual reflection. How is Jobs message meaningful to you, and how is it similar to your own personal life experience?
47
Strong Example of a Meaningful Spiritual Reflection: When I was nine, my little brother was three. He wasnt at all like mehe was always curious and kind of fearless. I like staying inside. My mom says that when I was Jareds age, I wouldnt talk to anyone. Jared, on the other hand, talked to everyone. He was sort of an escape artist too. My mother put those child-protect locks on the door, but Jared figured out how to open them. One evening my mom was busy cooking dinner and I was playing on the computer. All of a sudden she said, Wheres Jared? We realized that we hadnt seen him for about half an hour. We looked all around the townhouse and then went outside to call for him. We told all the neighbors and walked up and down the streets. Some of the neighbors went out looking too. We went back home and my mom was about to call the police, when our phone rang. It was the townhouse manager. He had found Jared, floating in the pool face down. He probably jumped in and thenI dont even like to think about what happened then. The pool had a gate but somebody had left it open that evening. In fact, someone had deliberately propped it open with a slipper. I was really sad and angry. I was angry at the person who left the pool open, I was mad at the townhouse manager for not checking the gate, and I was angry at my mom for not keeping better track of Jared. I was also even a little mad at Jaredhe should have stayed in the house like he was supposed to. Deep down, though, I knew it wasnt anyones fault. It was just what happened. I imagine that God felt just like I did. That God cried and was sad with all of us. I picture God caring for us, and that helps me to know that Jared is being taken care of and loved and watched over. Some time after the funeral, I went with my mom to talk about the pool gate at the townhouse meeting. She started to cry, so I said, Everyone should follow the rules. You shouldnt prop it open for your friends because little kids could get in and get hurt. Before this, I wouldnt have said anything, especially to a roomful of adults. My mom said she was proud of me. Sometimes people dont realize what might happen when they break rules. Because of Jared, I try to speak up whenever I see someone doing something they shouldnt do.
HOW TO WRITE A SELF-REFLECTION My Personal Reflection Self-Check Is your reflection: in the first person narrative (uses I)? one-page long (300-400 words)? meaningful? related to and to similar to the biblical message? about a change in thoughts, actions, values, and/or emotional feelings? Answer Key for Pre-Test Questions 1. 2. 3. 4. 5. The correct answer is C. The correct answer is B. The correct answer is C. The correct answer is B. The correct answer is C.
48
HOW TO WRITE A SELF-REFLECTION Appendix D. Embedded Test Questions
49
50
51
52
53
54
55
56
57
58
59
60
Appendix E: Post-Test Questions Theme 1: Identify a Change in Emotion Choose from the emotions below to answer question #1. Emotion A: When I got a bad grade for a test, I remember being angry at the teacher because I thought his test was unfair. Emotion B: After I got my haircut, I remember being very angry with the hair stylist so I didnt tip her. Emotion C: After being heartbroken for the first time, I will always think about that event before getting into a relationship. Emotion D: When it was her birthday, she was disappointed because she did not get the gift that she wanted. 1. Which emotion shows a change? A. Relationship A B. Relationship B C. Relationship C D. Relationship D Demonstrate a Personal Change in Emotions Think of a time when your emotions changed because of a meaningful experience. Describe this in 2-3 sentences. Strong Example: After seeing a movie about hunger, I began to appreciate every snack and meal that I had. Weak Example: I never like cats, and after my brother got a pet cat, I was certain that I didnt like cats. Self-Check: Are your 2-3 sentences: in the first person narrative (uses I)? relaying a personal experience? describing meaningful personal change in emotions?
61
Theme 2: Identify Change in Values Choose from the situations below to answer question #2. Situation A: After being told about how a boy cheated on his girlfriend, Ive thought that guys can be immoral. Situation B: After seeing my Mom struggle with her health because of her eating habits, Ive vowed to make exercise an important component of my life. Situation C: After speaking to the mobile phone salesmen, I was convinced that I will need an iPhone. Situation D: After hearing some amazing live jazz, Ive valued the contemplative nature of jazz so Ive downloaded some contemporary jazz music off of iTunes. 2. Which situation shows a change in values? A. Situation A B. Situation B C. Situation C D. Situation D Demonstrate a Change in Values Think of a time when youve had a change in values because of a crucial life experience. Describe this in 2-3 sentences. Strong Example: I used take my friends for granted. But after I moved to a new school, I really valued people who made the efforts to include me as a friend. Weak Example: I was always overweight as a little kid. When I entered middle school I lost a lot of weight, but I still have to think about what I eat. Self-Check: Are your 2-3 sentences: in the first person narrative (uses I)? relaying a personal experience? describing meaningful personal change in values?
62
Theme 3: Identify a Change in Action Choose from the scenarios below to answer question #3. Scenario A: She used to go to the Art Academy to view the Renaissance art. However, after going to China she developed an appreciation for Chinese and other Asian art. Scenario B: He used to like Egyptian history. After reading about royal Egyptian families and how brothers and sisters would marry one another, he thought that it was an uncivilized and gross practice to marry your sister. He still loved other aspects of Egyptian history especially the art. Scenario C: He always liked the ocean. After going diving in Australia on the Great Barrier reef and seeing dangerous marine life, he started to develop a fear of marine life. Scenario D: She was never really a good student who would go home to study. After reading a biography of her favorite actress who got straight As and went to Harvard, she started to study after school to get good grades. 3. Which scenario shows a change in actions? A. Scenario A B. Scenario B C. Scenario C D. Scenario D Theme 4: Identify a Conflict similar to Job's Conflict Chose from the following conflicts to answer question #4. Conflict A. She never paid any attention to her parents and what they wanted: she generally just followed her impulses. Conflict B. Whenever her father scolded her, she pretended to be sorry and he always forgave her. She was annoyed when her father finally grounded her. Conflict C. He argued constantly with his father; he opposed his father on every issue. Conflict D. Sometimes his mother seemed angry with him and he wasnt sure why; still he loved her and tried to obey her.
63
4.
The conflict between Job and God is most similar to A. Conflict A. B. Conflict B. C. Conflict C. D. Conflict D.
Theme 5: Identify a Relationship that is similar to Job's Relationship with God Chose from the following relationships to answer question #5. Relationship A. She didnt understand her father, but she always loved him. Relationship B. He knew that if his father said no to something he wanted, all he had to do was to go and ask his mother and shed get it for him. Relationship C. She rarely even thought about her parents and what they wantedshe was focused mainly on her friends and boyfriend. Relationship D. Her relationship with her mother was very easy: she was like a best friend more than a parent. 5. The relationship between God and Job is most similar to A. Relationship A. B. Relationship B. C. Relationship C. D. Relationship D.
Demonstrate a Relationship Time to Brainstorm! Think of a time when your relationship with a parent, teacher, or guardian was similar to Jobs relationship with God. Describe this time on the next page. Strong Example: My grandmother didnt speak English, and I didnt speak Japanese, so we couldnt talk to each other. Even though I didnt understand her at all, I knew that she loved and cared for me. Weak Example: I wanted to talk to the Principal about some issues I had with the school, but she was always too busy and wouldnt meet with me. Her secretary told me I should go talk to the counselor.
64
Self-Check: Are your 2-3 sentences: in the first person narrative (uses I)? relaying a personal experience? describing a relationship that is similar to Jobs relationship to God? Theme 6: Personal Reflection Theme to Ponder: What meaningful experience do I have that is similar to Job's experience of suffering? Time to Brainstorm! Think of a personal life experience that is both meaningful and similar to Jobs story of suffering. How did you learn from suffering? What benefit did you gain from suffering? Describe this experience in a short (2-3 sentences) paragraph. Strong Example: I'm going to write about when my grandfather passed away. I suffered through sadness and the pain of missing him, but I learned to value my family. I benefited because I learned to make time for my family and friends, and my life has been more joyful from this. Write your paragraph on the lines below: Self-Check: Is this personal experience: meaningful? related to or similar to the biblical message? about a change in emotions, values, or actions?
65
Time to Write! Write a spiritual reflection on the next page. Answer the following questions: How have you gone through suffering in life? What did you learn from suffering? Describe a beneficial change in thought, emotion, value and action because of the experience of suffering. Strong Example of a Meaningful Spiritual Reflection: Growing up I would always visit my grandparents. My grandfather would always take me fishing and he would always show me love and patience. As he grew older my grandfathers health slowly started to deteriorate. It got so bad that he needed to be flown to the hospital. I was the first person to see him in the hospital. Once he recognized my face, he grabbed my hand and held it as hard as he could. I remember seeing him suffer in the hospital. Unfortunately my grandfathers condition worsened in the coming months and he eventually passed away. This left me devastated at the loss of my grandfather and I learned about the importance of life, and that our time spent with people whom we love is temporary and can be taken away. Since my grandfather passed away, I started to treasure the time that I had with my three other grandparents. My actions changed because I tried to learn as much as I can from them and do what I could to make them feel loved. I started to value the time that I had with them and went out of my way to try to spend time with them. This change in actions and values also ended up extending to my family and friends. Since learning about the importance of time and life through the experience of suffering, my life has improved. The relationships with those who are most important to me have blossomed. While I need to be reminded of this every once and a while, this was the first time in my life that suffering made me learn about life. My life has ultimately become richer because of the experience. Remember that your personal reflection should: be meaningful. relate to or be similar to Jobs story of suffering. show how you learned from suffering. show how you ultimately benefitted from suffering. show a personal change in either emotions, values, and actions.
66
Appendix F: Attitude Survey Attitude Survey For The Writing a Spiritual Reflection Lesson Thank you for participating in our instructional design project! We would really appreciate one last thing: please complete this survey about the lesson and tests. Please read each statement carefully. Then, circle the appropriate number that corresponds to your belief. The survey is based on a 5-point scale, where 1 is strongly disagree, and 5 is strongly agree: 1 2 3 4 5 Strongly Strongly Disagree Agree Questions about the Lesson 1. The lesson was clear and easy to understand. 1 2 3 4 5 2. The level of difficulty was about right for me. 1 2 3 4 5 3. The length of the lesson was manageable. 1 2 3 4 5 4. The examples in the lesson were helpful. 1 2 3 4 5 5. I learned a lot from the lesson. 1 2 3 4 5 6. This lesson will help me to write more meaningful spiritual reflections. 1 2 3 4 5
67
Questions about the Tests 7. The questions and information in the lesson helped prepare me for the post-test. 1 2 3 4 5 8. All of the test questions were well-written and straightforward. 1 2 3 4 5 9. I did better on the post-test because I worked through the lesson. 1 2 3 4 5 ** Please answer the following questions by writing in complete sentences. 10. Based on what you learned in the lesson, do you feel more confident in writing spiritual reflections?
11. If you were designing this lesson, what would you change? What would you keep the same?
12. Any other suggestions or comments?
Thank you again for completing our lesson!
68
Appendix G: Attitude Survey Comments
Student 1
Question 10 No, I feel the same.
Question 11 I would make the writing No. shorter but I would make the questions the same. I wouldnt change anything.
Question 12
Yes, only certain spiritual reflections.
It was an interesting test. I never took a test like this before.
Not really because I dont I would make sure to put the No. know much spiritual things. story. Everything else I would I dont know things from keep the same. the Bible. I feel a little more comfortable now. I kind of feel confident because it gave us examples. Nope, I dont feel any different and dont think doing it a 3rd time makes a difference. I wouldnt change anything. A very interesting test, this was.
4 5
I would keep everything the I dont like the Job parts. same besides the Job sections. I would make it so we know if Not to do it 3 times! the reflections we wrote was good. We werent told if it was good or not so we dont know if were doing it correct or we need to improve.
I do feel more confident in I would explain the examples Left blank. writing reflections. more clearly because they are kind of hard to understand.
Yes, I do because learning the things I did the way I did helped me better understand how to write a spiritual reflection.
I would put more examples to Left blank. make it a little easier to understand. Other than that, everything else should be kept the same.
69
I do feel more confident when writing.
I would make it a little No. shorter, but everything is the same.
10
Yes, I feel I have more If I were to design this lesson Left blank. confidence in writing I wouldnt change anything. I spiritual reflections. This thought it was well designed. lesson helped me to better understand myself. Yes. I would keep it the same. Left blank. Left blank.
11 12
Maybe, I didnt really I wouldnt use the module understand all of it and Im concept. Id just make it a not good at writing long straightforward test. reflections. I am almost confident when I would change the length of writing the lesson. the post-test. I would leave the examples of the lessons the same. Yes, I feel that I can write more confident in writing spiritual reflections. Being a part of the learning experience of this lesson helps me understand the difference between a good and bad example. Also, I can put more feeling into my writing to help explain what I am trying to say.
13
Left blank.
14
I would integrate the lesson of Finish the movie Prince of loss and forgiveness because Egypt and that's about it. we lose things all the time. Good lesson plan and very Sometimes we lose it or it's insightful. taken from us. Though not everyone has forgiven people that has caused harm to them, so I think that would be a good lesson to write about. Everything else was fine.
70
Appendix H: Attitude Survey Summary Statistics Survey Average Student 1 2 3 4 5 6 7 8 9 10 11 12 13 Average Question 1 5 5 3 4 2 2 4 3 4 4 4 3 3 3.54 Q. 2 5 5 NA 4 2 2 2 5 5 5 5 3 3 3.83 Q.3 4 4 3 4 2 2 2 5 3 5 3 2 4 3.31 Q.4 5 3 4 5 3 2 5 4 5 5 4 4 4 4.08 Q.5 3 3 3 5 3 2 5 5 4 4 4 3 3 3.62 Q.6 3 4 3 4 2 2 4 4 5 4 4 3 3 3.46 Q.7 3 5 4 4 3 2 5 5 4 5 4 2 3 3.77 Q.8 5 4 3 5 3 3 3 5 3 5 3 2 4 3.69 Q.9 3 5 3 3 3 2 3 4 5 4 4 4 3 3.54
Likert scale attitude survey responses in frequencies Question Strongl Disagre y e Disagre e 0 0 0 0 0 0 2 3 3 1 1 2 Neutr al Agr ee Strongl y Agree 2 7 3 6 4 2 N/ A Tot al
1. The lesson was clear and easy to understand. 2. The level of difficulty was about right for me. 3. The length of the lesson was manageable. 4. The examples in the lesson were helpful. 5. I learned a lot from the lesson. 6. This lesson will help me to write more meaningful spiritual reflections. 7. The questions and information in the lesson helped prepare me for the post-test. 8. All of the test questions were well-written and straightforward. 9. I did better on the post-test because I worked through the lesson.
3 1 3 2 5 3
6 1 4 4 3 6
-1 -----
13 12 13 13 13 13
--
13
0 0
0 1
6 7
3 3
4 2
---
13 13

Etec 613 Formative Evaluation Draft 8

Transféré par

Informations du document

Description originale:

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Etec 613 Formative Evaluation Draft 8

Transféré par

Droits d'auteur :

Formats disponibles

Running head: HOW TO WRITE A SELF-REFLECTION

Formative Evaluation How to Write a Self-Reflection Based on a Biblical Message

Susan St. John

HOW TO WRITE A SELF-REFLECTION Table of Contents

HOW TO WRITE A SELF-REFLECTION

HOW TO WRITE A SELF-REFLECTION Appendices

Figure 10. Revised hierarchy chart.

HOW TO WRITE A SELF-REFLECTION

HOW TO WRITE A SELF-REFLECTION

HOW TO WRITE A SELF-REFLECTION

HOW TO WRITE A SELF-REFLECTION

Figure 3. Individual scores by objective with summary statistics.

HOW TO WRITE A SELF-REFLECTION

Figure 3 continued. Individual scores by objective with summary statistics.

HOW TO WRITE A SELF-REFLECTION

Figure 3 (continued). Individual scores by objective with summary statistics.

HOW TO WRITE A SELF-REFLECTION

Figure 3 (continued). Individual scores by objective with summary statistics.

HOW TO WRITE A SELF-REFLECTION

Figure 4. Line graph representation of test scores by objective.

Figure 5. Bar graph representation of test scores pertaining to change in emotions.

HOW TO WRITE A SELF-REFLECTION Anecdotal Data

HOW TO WRITE A SELF-REFLECTION

Strong ly Agree 15% 53% 23% 46% 31% 15%

15% 23% 23% 8% 8% 15%

23% 8% 23% 15% 38% 23%

46% 8% 31% 31% 23% 46%

99% 100% 100% 100% 100% 99%

HOW TO WRITE A SELF-REFLECTION

Evidence Student reflections lacked clarity. Also from instructor comments.

HOW TO WRITE A SELF-REFLECTION

Figure 7. Summary of recommended revisions.

Length: Amount of reading intimidated some students.

Length: Amount of writing extended completion time to over an hour.

Students were not

Give the pre-,

HOW TO WRITE A SELF-REFLECTION

Pre-, embedded, and post-tests were run too closely together.

Figure 7 (continued). Summary of recommended revisions.

Chunks were too small (1 item each).

The module was too dependent on reading.

Add multimedia such as audio or video. Scenarios

Students flipped through the reading.

HOW TO WRITE A SELF-REFLECTION

Instruction was visible when students were being tested.

Put instruction on the previous page.

Numbers in table of contents were not aligned in right margin.

Use decimal tab to line them up.

From instructor comments.

Use the gray boxes consistently to set off quotes.

From instructor comments.

Figure 7 (continued). Summary of recommended revisions.

Format: Spelling, grammar,

Layout of self-checks are crowded.

Add a space after the checkbox.

From instructor comments.

HOW TO WRITE A SELF-REFLECTION

Figure 7 (continued). Summary of recommended revisions.

HOW TO WRITE A SELF-REFLECTION

HOW TO WRITE A SELF-REFLECTION References

HOW TO WRITE A SELF-REFLECTION

HOW TO WRITE A SELF-REFLECTION