RESEARCH METHODOLOGY

TOPIC 5

TOPIC 5:

RESEARCH METHODOLOGY
Assoc. Prof. Dr. Saat Md. Yasin

INTRODUCTION
This topic deals with the general concept in carrying out a research. It deals with the appropriate steps in carrying out research methodology to ensure that a new researcher would not fall into grave errors that could waste a lot of precious time and funding. This topic guides you in the understanding of the process to fulfil certain objectives of this course.

OBJECTIVES
By the end of this topic, you should be able to: 1. plan a research in education successfully; 2. state the differences between qualitative and quantitative research designs; 3. describe a population and a sample; 4. select sample using random sampling; 5. develop research instruments; 6. conduct a pilot study; 7. explain how to establish the reliability and validity of the research instruments; 8. analyse quantitative and qualitative data; and 9. explain the differences between descriptive and inferential statistics.

58

OUM

TOPIC 5

RESEARCH METHODOLOGY

TOPIC MIND MAP

5.1

WHAT IS RESEARCH?

Research is a formal and systematic application of the scientific and disciplined inquiry to the study of problems while methodology is a way or means of doing something such as the way of making a scientific inquiry into a problem. Since the research methodology in educational research is scientific and disciplined in nature, it has to adhere to certain procedures that need to be strictly followed. First of all, any research to be carried out needs to be carefully planned to ensure the validity of the findings. We have to consider at least three important factors to ensure that the research is properly implemented.

5.2

RESEARCH PLANNING

After identifying the area or the topic for the research, you need to formulate the purpose, objectives, questions and hypotheses of the research. You have to draw out the methods and the procedures of the intended research. Therefore, you require expertise in a number of

OUM

59

RESEARCH METHODOLOGY

TOPIC 5

different areas. The research plan describes the nature of the participants in the research, the variables, the kind of data, the instruments to be used, the condition for data collection and the techniques to analyse the data. When identifying the area of study, you have to consider the type of research to be utilised. Is it a quantitative or qualitative research? Which one is more suitable? Planning a research using the quantitative method is different from planning a qualitative research. You have to consider many different factors when applying either the quantitative or the qualitative methods.

1. 2.

What is the expertise needed in planning a research successfully? What are the factors that you need to consider when applying either the quantitative or the qualitative methods?

5.2.1 Research Questions and Data

Research questions are the questions that we ask to achieve the objectives for our research. These questions can be answered through collecting of data and analysing them. After the research questions have been established and the topic of the study has been identified, you have to describe the following procedures in executing the research. (a) Describe and execute procedures to collect information. The procedures include identifying research participants, the strategies to collect data and activities describing, how, when, why and from whom the data were collected. The detailed description would enable you to smoothly carry out the research without major hitches if there were any. The flaws in carrying out the research might jeopardise the validity of the research findings. (b) Analyse the collected data. The data would be analysed differently depending on the nature of the research. If the research was quantitative, then the data statistical analysis might be needed to achieve the desired findings. On the other hand, if the research was qualitative in nature, then the data analysis requires a descriptive explanation of the findings. Lengthy notes might be scrutinised and summarised. The observations made might be transcribed in coherent and easily understood explanations. (c) After the data have been analysed, the result or the findings of the study has to be presented coherently. They must be clearly stated and the implications of the findings should also be included. The findings might be useful to interested parties for various reasons. The government might want to improve whatever shortcomings pointed out by the findings. The school principals might want to use them for school improvement.

60

OUM

TOPIC 5

RESEARCH METHODOLOGY

RESEARCH DESIGN

Quantitative 1. Experimental 2. Quasi-experimental 3. Correlational 4. Survey

Qualitative 1.Grounded Theory 2. Ethnographic 3. Narrative

Combination of Quantitative and Qualitative 1. Mixed Method 2. Action Research

Figure 5.1: Research design

5.2.2 Selection of Research Design

A research design is a plan to carry out an intended research. According to Creswell (2002), there are eight different research designs which are frequently used in educational research. They are either quantitative or qualitative in nature or in some instances, the combination of both. (a) Quantitative Research Designs The experimental and quasi-experimental designs will be explained separately. (i) Experimental Designs They are usually referred to as group comparison studies because researchers compare different groups in terms of one or more outcomes. Experiments are also called intervention studies because you intervene or manipulate the variables. The experimenters design and conduct experiments where they maximise the influence of the independent variable on the dependent variable, and at the same time minimise the influence of the extraneous factors. There are five key characteristics of experimental research designs that are of importance to you: Participants are selected and assigned at random to different groups, namely control and experimental groups. Intervention is applied to one or more groups Outcomes are measured at the end of the experiment Procedures are designed that address potential threats to validity Statistical comparisons of different groups are conducted. In the educational scene, experimental research could be used in various situations. One such situation is to find out the effectiveness of a new method of teaching over a conventional one. Two groups of students were randomly selected. One group is used as the experimental group using the new method of teaching, while the other is the control group using the conventional method of teaching. The topic of the lesson is chosen from the syllabus and the different treatments are given. After the

OUM

61

RESEARCH METHODOLOGY

TOPIC 5

treatments had been completed, the outcome of the experiment is measured. The outcome might be in the form of the academic achievement through the test results of the students in the control group and the experimental group. This statistical analysis would provide the answer whether the experimental group performed significantly better than the control group. (ii) Quasi-Experimental Design Most of the situations in schools need you to use intact groups such as in studying a new English programme. This study may involve the students in the same grade as respondents where one class is considered the control group while the other is considered as the experimental group. Randomly assigning students to the two groups would disrupt classroom learning. Because teachers often use intact group in experiments, quasi-experimental designs are usually used.

(iii) Correlational Designs A science teacher wants to know whether the assumption that students who are good in mathematics are also good in science is true or false. In order to find out, she compares the examination results of her students in the two subjects, the mathematics and science. She correlates the mathematics test results and the science test results of the students. If the students who scored high marks in mathematics also scored high marks in science, we can conclude that there is a positive correlation between the mathematics and the science results, and if this finding is consistent with other research findings in the same area, we could say that the assumption is correct. The above anecdote describes one of the correlational designs used in the educational field. Correlational research in the above case uses a statistical procedure. Now, with sophisticated applications and explicit procedures of correlations, such as regression, path analysis, and causal modelling, correlational research is entrenched in the family of quantitative research. Correlational research as a design, has two primary forms, namely explanation and prediction. Correlational research helps to explain the association between two or more variables or to predict an outcome. (iv) Survey Designs Suppose the Ministry of Education (MOE) wants to know what the parents and the students feel about having one session schools where students begin school at 8.00 a.m. and ends at 4.00 p.m. The research question might be: What are the opinion of the students and their parents toward one session school and lengthening of the school session? By using a survey design, the MOE would gather the opinions of the students and their parents. The survey design is an economical way of doing research especially when the respondents are large. The MOE would randomly select the respondents from the two groups: students and their parents, and send them a mailed questionnaire, analyse the results and generalised results from of sample to the population. This is a survey that is commonly used in the school system. Survey designs are also frequently used to: describe trends; determine opinions about policy issue, such as giving loans to all students in tertiary institutions; background of lecturers from private and public colleges and universities; understand important attitudes and beliefs towards dating among secondary school students;
62

OUM

TOPIC 5

RESEARCH METHODOLOGY

identify practices, such as carrying weapons to schools and gangsterism; and evaluate programmes, such as using English for mathematics and science subjects. Survey designs are mainly of two types: cross-sectional and longitudinal. Crosssectional designs are used to collect data that reflect current attitudes, opinions or beliefs. Longitudinal designs are used to study individuals for a period of time such as studying people who were high achievers and low achievers in schools.

1. 2. 3.

What research design do you use when you want to know the effectiveness of your new method of teaching oral English? Explain how you implement the design. What are the differences between Experimental Design and Quasiexperimental Design?

(b) Qualitative Research Designs The qualitative types will be explained in the following paragraphs. (i) Grounded Theory Designs A teacher who had been teaching for the past twenty years had the experience of teaching students from different socio-economic backgrounds. He found out that most of the high achievers were from the higher socio-economic background and most of the low achievers were from the lower socio-economic background. As a result, he developed his own theory that students from the higher economic background have parents who are more concerned that their children must have good grades. Their emphasis on discipline and spending more time in doing their homework enables their children to excel academically. On the other hand, students from the lower socio-economic background spend less time on studying, lack of parental supervision, and have non-conducive home environment for learning which do not help them to perform well. The teacher wanted to make a study on this phenomenon which is grounded on a theory. He wanted to test his hypothesis that includes the following: Students from a higher socio-economic background spend more time studying. Parents from a higher socio-economic background send their children to tuition classes. Students from a higher socio-economic background have relatives who are high achievers as their models. The research that is most likely to be used in this situation is of qualitative type in nature. The research based on theory, as in the above case, requires observation as well as interviews. The data collected would be lengthy and qualitative in nature. The data would be analysed and a report of the findings has to be made. To know how this Grounded Theory Designs work, let us examine one published grounded theory studies (Creswell , 2002):

OUM

63

RESEARCH METHODOLOGY

TOPIC 5

Richie, Fassinger, Linn and Johnson (1997), studied the career development process of 18 high-achieving African American-Black and white women by generating a theoretical model. They explored critical influences in the careers of these 18 women. From the interview data, they developed a theoretical model that visually portrayed how sociocultural conditions, personal backgrounds, such as socio-economic background, influence the womens strength and passion for work and interconnectedness with others in their careers.

(ii)

Ethnographic Designs The term ethnography literally means writing about groups of people (LeCompte & Schensul, 1999: 21). Using this design, you can identify a group of people to study, visit the setting of the group, note how they behave and interact with others, how they think and develop a general picture of the group. It can provide a day to day detailed picture of their activities. As an example, let us look at, The life of residential school students in Malaysia. This study requires you to observe the students and make notes on their daily activities. You could become a participant observer in order to know and understand the details of their daily life. By being with them for a certain length of time, you would have an ethnographic view of the students. Ethnographic designs are qualitative research procedures for describing behaviour, beliefs, and language that they developed over a period of time. In the case of residential school students in the example mentioned, they have their own pattern of behaviour and their shared values which are likely to be different from other students in non-residential schools. Van Maanen (1988); Denzin (1997); LeCompte, Preissle and Tesch (1993) had classified nine (9) types of ethnographic designs as shown in Table 5.1.
Table 5.1: Nine Types of Ethnographic Designs

Realist ethnography Confessional ethnography Life history Auto-ethnography Micro-ethnography Ethnographic case study Critical ethnography Postmordern ethnography Ethnographic novels

An objective scientifically-written ethnography. A report of the ethnographers fieldwork experiences. A study of one individual situated within the cultural context. A reflective self examination by an individual set within his cultural context. A study focused on a specific aspect of a cultural group. A case analysis of a person, event, activity, or process. A study of women and the cultural practices that serve to disempower and oppress them. An ethnography written to challenge the problems in our society. A fictional work focused on cultural aspects of a group.

64

OUM

TOPIC 5

RESEARCH METHODOLOGY

The different ethnographic designs stated above share similar characteristics when reporting the findings. If an investigator intends to utilise one of the ethnographic designs above, he/she has to read the subject matter further to ensure the most appropriate approach for the research. (iii) Narrative Research Designs Narrative research is a design that is used by individuals to tell their personal, firsthand accounts to researchers. In narrative research, investigators describe the lives of individuals, collect and tell stories about peoples lives and write narratives of individual experiences (Connelly & Clandinin, 1990). As a distinct form of qualitative research, a narrative usually focuses on studying a single person, gathering data from the collection of stories. Narrative research can assume multiple forms. Narratives may include a variety of research practices as was compiled by Casey (1995, 1996). Examples of types of narrative research forms are: Autobiographies. Biographies. Life Writing. Personal Accounts. Narrative Interviews. Personal Documents. Documents of Life. Life Stories and Life Histories. Oral Histories. Ethnohistories. Ethnobiographies Autoethnographies. Ethnopsychologies. Person-centred Ethnographies Popular Memories. Latin-American testimonies. Polish memoirs. For teachers in Malaysian schools, not all of the above types of narrative research could be applied. Only some of the most commonly used, such as autobiographies and biographies, are usually utilised in research. (c) A Combination of Qualitative and Quantitative Research Designs The following paragraphs will elaborate on the combined qualitative and quantitative research designs. (i) Mixed Method Designs A mixed method research design is a procedure of collecting both qualitative and quantitative data in a single study. Since the 1930s, educational and social science
65

OUM

RESEARCH METHODOLOGY

TOPIC 5

researchers have combined research methods of data collection in their studies (Sieber, 1973). Sieber suggested the combination of in-depth case studies with surveys, creating a new style of research where quantitative and qualitative methods were used within a single study. The rationale for carrying out the mixed method designs in research is mainly to ensure that majority of the factors that influence the situation could be taken care of. Majority of the variables have been looked into using quantitative methods, but variables that could not be taken into account through quantitative research, could be studied through qualitative research. If a nationwide research is to be carried out where variables of various kinds, such as the family, the school system, the curriculum, the students, and funding were to be studied, the mixed method design would suit the purpose. Take study habits as an example. When the details on how a student manages his/her time and the method of how he/she learns, qualitative research such as ethnographic research might be better than just quantitative data, such as the number of hours spent on studying. Other types of data that could easily be carried out quantitatively, such as family income, number of siblings, and academic background should be collected quantitatively. (ii) Action Research Designs Action research addresses a specific, practical issue and seeks to obtain solutions to the problem. Similar to the mixed method research, action research uses data collection based on either quantitative or qualitative methods. Action research provides educators an opportunity to carry out research in the classroom to reflect their teaching practices. Action research offers a means for staff development; for teachers development as professionals, and simultaneously addressing schoolwide problems. As an example, an English teacher found out that her students were weak in English from the students test results and her interaction with them. So, she decided to enquire from the students regarding their background such as their families socio-economic background, the number of books they read in English, whether the family members communicate in English with each other and their attitude toward the English language. These variables could be researched quantitatively as well as qualitatively to collect valid data and find out the variables that influenced the students command of the English language. The findings of this action research would help the English teacher to know some of the reasons why most of her students are weak in the English language. This would help her to design a remedial programme to improve the students command of the English language. If teachers are able to carry out action research to improve their teaching, overall quality of the students learning would improve.

1.

If a researcher wants to find out the reasons why some students achieved excellent academic results, while others failed to do so even though the initial academic potentials seem to be equal, what research design do you suggest a researcher could choose? Explain. What action research can a teacher do in school to improve his/her teaching?

2.

66

OUM

TOPIC 5

RESEARCH METHODOLOGY

5.2.3 Research Population and Sample

RESEARCH POPULATION AND SAMPLE

Selecting Sample

Simple Random

Stratified

Cluster

Systematic

Figure 5.2: Research population and sample

What is the definition of population? Gay and Airasian (2003), defined population as The population is the group of interest to the researcher, the group to which the results of the study will ideally generalized. To determine a population is important in an educational research. The population determines how far and how much data should be collected and analysed. The interpretation and inference of the data is also limited to the population of the study. The population also determines how much time and energy should be devoted to solve a problem. Population also influences how much fund is needed to carry out the research. As an example, a study on the needs of the rural students to improve their academic performance in a certain school is different from a study on the needs of rural students in Malaysia. The populations of both samples are different in size which also determine the types of research designs to be carried out. Another important factor for the researcher to consider is accessible or available population. In most studies, accessible population is generally a realistic one compared to the target population which is more of an idealistic one. Although we can use the whole population for our research, it is not practical because it requires a large amount of funding, manpower and time to carry it out successfully. Therefore, we have to devise a means that would do the same job without distorting the findings. This is done through random sampling. Selecting a sample is a very important step especially when conducting quantitative research. A good sample is one that is representative of the population from which it is selected. The steps in sampling can be summarised as follows: identify the population; determine the required sample size; and select the sample. Take an example in selecting a sample from the Form1 students in a school. You as a teacher, first of all, have to identify the population, that is the Form 1 students in the school. Afterwards, you need to determine the sample size. Out of the 200 students for example, you need to sample 60 students; 30 percent of the population. You need to select randomly from the cohort of Form 1 students of the school. You can use one of the sampling techniques. These sampling techniques are all based on randomness in the selection of the sample. There are four basic random sampling procedures, namely:

OUM

67

RESEARCH METHODOLOGY

TOPIC 5

simple random sampling; stratified sampling; cluster sampling; and systematic sampling. (a) Simple Random Sampling Random sampling is a process of selecting a sample in such a way that all individuals in the selected population have an equal chance to be selected into the sample. Random sampling is the best way to obtain a representative sample. The procedures in simple random sampling involve the following specific steps: (i) (ii) Identify the population and define it. Determine the sample size.

(iii) List the names of all members of the population. (iv) Assign all individuals on the list a consecutive number from zero to the required number, for example, 000 to 200 where each individual has the same number of digits as the others. (v) Select an arbitrary number from the table of random numbers. As an example, by closing your eyes while pointing to a number in the list.

(vi) Go to the next number until the desired number of individuals has been selected for the sample. If 30 percent of the 200 members in the population are to be selected, then 60 students in the sample will fulfill the desired percentage. (b) Stratified Sampling This is a process of selecting a sample in such a way that identified sub-groups in the population is represented in the sample in the same proportion that they exist in the population. For example, if we want to find out the academic qualification of teachers teaching various subjects, the population of the teachers that are to be studied is not uniform because some of these teachers have degrees while the others do not have them. Simple random sampling cannot be applied in this situation due to the fact that it does not represent the strata that exists in the population. Besides, simple random sampling to the above population will increase sampling error. Therefore, simple stratified sampling should be used to reduce sampling error. The first step in carrying out the study using stratified sampling is determining the strata to be studied such as by categorising the graduate and non-graduateteachers. Then from every stratum, respondents are selected at random to represent the teachers population. Random sampling for every stratum could be generated using the same strategy as simple random sampling. The allocation of sample size between strata can be done proportionately. If 60% of the teachers are non-graduate and only 40% are graduates, then the sample should consists of 60% non-graduate teachers and 40% for graduate teachers. (c) Cluster Sampling Cluster sampling is a process of using a sample whenever the sampling unit is not the unit that represents the population but represents a group or a cluster within the population. For example, we dont have a complete list of teachers. We can use cluster sampling by determining the number of clusters such as the teachers based on the states they come from. Sampling based on clusters could be done randomly by using the table of
68

OUM

TOPIC 5

RESEARCH METHODOLOGY

random numbers. When the cluster that represents the population has been randomly selected, we must use every unit in the cluster. After we have randomly selected the teachers from one school district such as Putrajaya, all members within the cluster must be used as the respondents for the study. For a clearer illustration of cluster sampling, note the following steps: (i) (ii) Identify and define the population. Determine the required sample size.

(iii) Identify and define a logical cluster (school, teachers in the district, primary school teachers). (iv) List all clusters that make up the population of clusters. (v) Estimate the average number of population members by clusters.

(vi) Determine the number of clusters needed by dividing the sample size by the estimated size of a cluster. (vii) Randomly select the required number of clusters. (viii) Include all members in the selected cluster in your study. (d) Systematic Sampling It is a method of sampling where individuals are selected from a list taking every Nth name. If N here represents 5, then every 5th name will be selected as the sample for the study until the required numbers are met. This method of sampling though is not used very often. The major difference between systematic sampling and the other types of sampling is that all members of the population do not have an independent chance of being selected for the sample. Once the first name is selected, all the rest are automatically determined for the sample. Therefore, systematic sampling is often not as good as random sampling. There is a possibility that students from a particular ethnic group having names that are concentrated on certain letters could be excluded from the research through omission by using systematic sampling. Systematic sampling involves the following steps: (i) (ii) Identify and determine the population Determine the desired sample size.

(iii) List the names of the population. (iv) Determine the N by dividing the size of the population by the desired sample. (v) Start at some random place in the population list. Close your eyes and point your finger on a name.

(vi) Starting at that point, take every Nth name until the required sample size is reached.

1. 2. 3.

What do you understand by accessible population and target population? When do you utilise systematic sampling? Why is random sampling preferred over other sampling techniques?

OUM

69

RESEARCH METHODOLOGY

TOPIC 5

5.3

RESEARCH INSTRUMENTS

DEVELOPING RESEARCH INSTRUMENTS

Determining Objectives

Determining Contents

Figure 5.3: Developing research instruments

We are familiar with the terms test, examination, assessment and measurement. A test or examination is a formal, systematic procedure for gathering information about peoples cognitive, psychomotor and affective characteristics. A standardised test or examination is administered, scored, and interpreted in the same way no matter where or when it is administered. These standardised tests and examinations are developed by experts. For example, the Ujian Penilian Sekolah Rendah (UPSR) or Primary School Assessment Test, Penilaian Menengah Rendah (PMR) or Lower Secondary Assessment, Sijil Pelajaran Malaysia (SPM) or Malaysian Certificate of Education and the Sijil Tinggi Pelajaran Malaysia or Malaysian Higher School Certificate are all standardised to ensure that all test - takers experience the same conditions when taking them. Assessment is a broader term than test or instrument, and encompasses the process of collecting, synthesising and interpreting information. Tests are a subset of measurement, as are observations and interviews. Measurement is the process of quantifying or scoring ones performance.

5.3.1 Developing Research Instruments

Before you carry out a research, you have to develop the research instruments. If it is to be a test, it should be planned and developed well to suit the purpose and objectives of the research. A well-developed test would measure the variable of the research accurately. The accurateness of the collected data is very important to ensure that the collected data could be analysed and interpreted correctly. (a) Determining the Objectives The test should be constructed according to its objectives. The behavioural objectives to be measured should be realistic and stated clearly. Objectives should also be stated behaviourally and not in the form of learning activities. The statement of the objectives should be linear whereby each and every statement should be related to only one process.

70

OUM

TOPIC 5

RESEARCH METHODOLOGY

(b) Determining the Contents The contents of the area to be measured should be determined first. An effective test is a test which has a high content validity. A test should cover all contents of the research area. To ensure the test used covers all contents, you should first determine the contents of the research area. The opinion of the expert in the area could be utilised to ensure the test to be developed has high reliability and validity. Usually the content of the area of study is wide, so you could use sampling techniques of the content as representatives of the content. You need to determine content sampling to represent the behaviour to be measured. If content sampling is used, you need to ensure that they are representatives of the behaviour to be measured. If this method is used, the researcher should ensure that the chosen contents could represent the overall contents that are to be measured.

5.3.2 Reliability and Validity of Research Instruments

Both reliability and validity of research instruments are vital in any research. Validity is the most important characteristic of a test or a measuring instrument. Validity is concerned with the appropriateness of the interpretations made from test scores. When we test, we test it for a purpose. For example, you may administer a questionnaire to find out the participants opinions on the new methodology used in teaching mathematics compared to the traditional method where talk and chalk were used. A key question for these test users is, Does this test allow me to make the interpretation I wish to make? That is, will responses to the questionnaire allow you to make appropriate interpretations about the respondents opinions? Validity is important to all forms of research and all types of tests and measures. There are four types of test validity: content validity; criterion-related validity; construct validity; and consequential validity. (a) Content Validity Content validity is the degree to which a test measures an intended content area. It requires both item and sampling validity. Item validity is concerned with whether the test items really measure the content area. Sampling validity is concerned with how well the test samples the total content area being tested. Content validity is particularly important for achievement tests. A test score cannot accurately reflect a students achievement if it does not measure what the student was taught and supposed to learn. The validity of the content is determined by expert judgment. Experts in the topic covered by the test are asked to assess its content validity. The judgment is made by comparing what was taught and what is being tested. The validity would be very strong when the two coincides. (b) Criterion-Related Validity Criterion-related validity has two forms, namely concurrent and predictive validity. (i) Concurrent validity is the degree to which scores on one test correlate to scores on another equivalent test. For example, a test is developed that claims to do the same thing as some other tests, except easier or faster. One way to determine
71

OUM

RESEARCH METHODOLOGY

TOPIC 5

whether the claim is true or otherwise is to administer the new and the old test to a group of students and compare the scores. If there is a high correlation between them, the concurrent validity of the new test is established and it may be utilised to replace the older test. (ii) Predictive validity is the degree to which a test can predict how well individuals will do in a future situation. Predictive validity is very important for tests that used to classify or select individuals. In Malaysia, we use the test results of UPSR for selection into residential schools. These test results are considered as having high predictive validity. The selected students admitted into these schools are expected to perform well in the next public examination three years later, namely the PMR. No test would have perfect predictive validity, and therefore predictions based on the scores of any test will be imperfect. However, predictions based on a combination of several test scores will be more accurate. In order to establish the predictive validity of a test, the criterion has to be carefully defined, which is the valid measure of the performance to be predicted. Once the criterion has been identified, the procedure for determining predictive validity is as follows: Administer the test which is considered as the predictor to a group. Wait until the behaviour to be predicted occurs. Obtain measures of the criterion for the same group. Correlate the two sets of score. Evaluate the results. Once the two results are correlated, and if the coefficient is high, the test has good predictive validity. (c) Construct Validity Construct validity is the most important form of validity because it asks the question: Wha does this test measure? Constructs are non-observable traits, such as anxiety, intelligence and honesty. You can only observe its effect. Constructs can explain certain differences between individuals. As an example, it was observed that some students learn faster than others or learn and remember more than others. To explain these differences, a theory on intelligence that is related to learning was developed. This theory explains that students who have the above qualities are more intelligent than others. They also perform better in examinations. Tests were developed to measure how much intelligence a person has. The construct or theory explains that students who score more tend to do better in school and other learning environment than those who have less of it. Those with high scores are said to be more intelligent than others. Research involving a construct are valid only to the extent that the instrument used actually measures the intended construct. Determining construct validity is not the easy and it usually involves gathering a number of pieces of evidence to demonstrate the validity. For example, if some psychologists construct an intelligence test instrument called Smart Ali, the validity of the instrument should be subjected to various tests. First of all, we could see whether students who scored high on the Smart Ali test learned faster, learn more and remember more than low scorers. We could correlate the scores of the students at the beginning of the school year and at the end of the school year. We could also correlate performance on the Smart Ali test with other well established intelligence tests to observe whether the correlations were high. Scholars should also be invited to examine whether the test items represent the typical topics in the field of

72

OUM

TOPIC 5

RESEARCH METHODOLOGY

intelligence. In addition to this confirmatory evidence, we need to seek disconfirmatory validity information. For example, we do not expect this intelligence test to be highly correlated to self-confidence or anexiety tests.

RELIABILITY

Reliability as Stability

Reliability as Equivalence

Reliability as Internal Consistency

Figure 5.4: Three types of reliability

(d) Definition of Reliability Reliability is a synonym for consistency and replicability over time, over instruments and groups of respondents (Creswell, 2002). It is about precision and accuracy. A reliable research must show that if it were to be done on a similar group of respondents in a similar context, than similar results would be obtained. There are three types of reliability, namely; stability, equivalence and internal consistency. (i) Reliability as Stability Reliability as stability is a measure of consistency over time and over similar samples. It means that a reliable instrument will achieve similar data from similar respondents over time. In the experimental model, for example, this would mean that a test and then a retest were taken within an appropriate time span, similar results would be obtained. Correlation coefficients can be calculated for the reliability of the pre-and post-tests. Reliability as Equivalence Reliability could be achieved through using equivalent or alternative forms of a test or data gathering instrument. If the equivalent forms for the test is devised and produces similar results, then the instrument could be said as having this form of reliability.

(ii)

(iii) Reliability as Internal Consistency Internal consistency demands that the instrument be run only once unlike the other two mentioned earlier (stability & equivalence). Internal consistency could be demonstrated by running the test once only through the split-half method. Let us suppose that a test is to be administered to a group of students. Here, the test items are divided into two halves matching in terms of difficulty and content. If the test is to show split-half reliability, then the marks obtained in each half should be correlated highly to the other half.

OUM

73

RESEARCH METHODOLOGY

TOPIC 5

1. 2. 3.

Why are the reliability and validity of research instruments vital? What is the difference between reliability as equivalence and reliability as internal consistency? Explain the difference between predictive validity and concurrent validity.

5.3.3 Pilot Study

A pilot study is a pre-research study undertaken on similiar set of respondents: in terms of location, nature of setting and profile of respondents (age, gender, etc). This is done to ensure the validity of instruments and to resolve possible problems before the actual research is done. Pre-testing of the instruments to be used in a research is of vital importance. A pilot study has several functions, mainly to increase the reliability, validity and practicability of the questionnaire (Oppenheim, 1992; Morrison, 1993; Wilson and McLean, 1994). Some of the objectives of piloting the questionnaire are (Cohen, Manion & Morrison, 2000): To check the clarity of the questionnaires including instructions and layout. To gain feedback on the validity of the questionnaire items. To eliminate ambiguities. To gain feedback on the type of question and its format. To gain feedback on response categories for closed questions. To gain feedback on the attractiveness of the questionnaire. To gain feedback on the layout of the questionnaire. To check the time taken to complete the questionnaire. To identify redundant questions. To identify commonly misunderstood items.. To try out the coding system for data analysis. After the pilot study, you can refine the questionnaire based on the responses of the respondents in the pilot study. This refinement would have eliminated all the shortcomings of the instruments. Then, you can proceed with the actual data collection for the research.

5.4 DATA ANALYSIS

Data analysis becomes very crucial after all the required data were collected. Data analysis has to be done before we could make any inference. Numbers have to be crunched using statistical analysis and statistical values have to be interpreted. Quantitative and qualitative data have to be treated differently using guidelines provided in this module. Let us look at more information regarding data analysis.

74

OUM

TOPIC 5

RESEARCH METHODOLOGY

5.4.1 Measuring Research Variables

Data are collected from the respondents through a questionnaire or from test results. The questionnaire items were generated from research variables. In educational research, the typical variables are as follows: Leadership styles of school principals. Academic achievement of students. Interpersonal communication skills. Attitudes toward school. Teaching styles of teachers. Socio-economic background of parents. Time spent on studying and doing home-work. Reading habits of students. These variables are the main ideas that researchers seek to address in their studies. A variable is a characteristic or attribute of an individual or an organisation that you can measure or observe. It also varies among individuals or organisations. When we talk about a variable as a characteristic we mean personal aspects about the individuals, such as their age and income level. An attribute represents how an individual in an organisation, feels, thinks or behaves. When we discuss about individuals having self-confidence or displaying leadership traits, we are talking about their attributes. Researchers measure these characteristics and attributes. Measurement means recording information from individuals through: Observation; or asking questions. In either case, student scores vary. The variables vary means the scores will have different values depending on the respondents characteristics. For example: SPM Examination results vary by nine possible scores: 1 being the highest, which is equivalent to a distinction, and 9 being the lowest which is equivalent to a failure. Gender varies by two possible scores: male = 1 and female = 2. In a quantitative study, most researchers used to record the scores by using the Likert Scale on a 5 point scale. For example, we may state a statement: The teacher always uses examples to explain difficult concepts. The 5 point scale used would most probably be: 5 = strongly agree 4 = agree 3 = moderately agree 2 = disagree 1 = strongly disagree

OUM

75

RESEARCH METHODOLOGY

TOPIC 5

The respondents would choose one of the values to state their agreement or disagreement to the given statement. The high scores denote that there is a strong agreement to the statement and vice versa. When a research variable in a construct it is usually measured by taking the sum or average of scores of all related items in the questionnaire. On the other hand, a qualitative study requires a different method of description. Instead of utilising numbers to represent some ideas, the researcher might have to write lengthy descriptive narrations, which later on have to be summarised and classified under different selected variables. The analysis of qualitative and quantitative data will be explained in more details later.

5.4.2 Analysing Qualitative Data

Qualitative researchers use to analyse data by using the following steps: Preparing and organising the data for analysis. Describing and developing themes from the data. Reporting the findings. Interpreting the findings. Validating the accuracy and credibility of the findings. In a summary form, the qualitative process of data analysis was vividly presented by Creswell (2002:258).
Code the Text for Description to be Used in the Research Report Code the Text for Themes to be Used in the Research Report

Iterative Code the Data (i.e., locates text segments and assigns a code label to them)

Simultaneous

Read Through Data (i.e., obtains a general sense of material)

Prepare Data for Analysis (i.e., transcribes field notes)

Collect the Data i.e., a text file, such as field notes, transcriptions, or optically scanned materials)

Figure 5.5: The Qualitative Process of Data Analysis 76

OUM

TOPIC 5

RESEARCH METHODOLOGY

Figure 5.5 summarises the general process of analysing qualitative data. Qualitative researchers collect data first and then prepare it for data analysis by transcribing field notes. Your next step is to read through the data to have a general sense of the material. Since there is a lot of data, you have to code the data for easy referral later on. Qualitative researchers need to analyse their data by reading through several times and conduct an analysis each time. In doing so the researcher would have sifted through a lot of information which could be useful in writing the final report. A useful conceptualisation is to think about the process as both simultaneous and iterative, which means going back and forth a few times. Therefore, there is not one single way to analyse qualitative data. It is a process in which you try to make sense of the information.

5.4.3 Analysing Quantitative Data

Before quantitative data could be analysed, a researcher would have to prepare and organise the data for analysis. The following steps would be useful as a guide: (a) First of all, you have to assemble all data by transforming it into numeric score or value to each response category for each item. Before conducting an analysis of scores, you should consider what types of scores to be used. For example, a single item score is an individual score to each item for each participant in your study. The next one is summed scores which is the scores of an individual added over several items that measure the same variable. The other is the net or difference scores which are scores in a quantitative study that represent a difference or a change for each individual. (b) The next step is selecting a statistical programme. With the presence of the computer age, selecting a suitable programme for analysing quantitative data is really not a big problem. If you are a student in a university, use the programme that is available at the university. (c) The third step is inputting data. This is a process when you transfer the data from the responses on instruments to a computer file to be used for analysis. (d) Exploring and Descriptively Analysing the Data is the fourth step. By doing this, you engage in data analysis. This procedure consists of: exploring and describing the data; and conducting statistical tests on the data. By exploring the data, you examine for errors and missing data so that decisions can be made about the missing data. (e) Cleaning and Accounting for Missing Data You must clean the data because of errors in scoring that you make when inputting the data. This is a process of visually inspecting the data for scores outside the accepted range. This will alert you on the errors which were made especially after a tiring process of inputting data. Sometimes data to individual items are missing. Missing data is information that is not supplied by participants to specific items. Missing data will reduce the statistical options available for your statistical analysis. Therefore, how would you handle this problem? The obvious approach is to avoid having missing data by providing items such that respondents are willing and able to answer. If your database contains missing data, there are several technical procedures that can help you. One procedure is eliminating respondents with missing scores from the data analysis, but this practice may reduce the number of overall respondents. Some researchers recommend

OUM

77

RESEARCH METHODOLOGY

TOPIC 5

substituting values for the missing cases. The researcher can have the computer programme to substitute a value for each missing score, such as the group mean (George & Mallery, 2001). (f) Calculating Descriptive and Inferential Statistics Descriptive statistics present information that helps you in assessing the general trend in the data and in answering descriptive research questions in the studies. These objectives can be achieved by running the frequency distribution and its statistics. To understand the statistics, you need to have some knowledge about them. These statistics are the numbers derived from formulas to measure aspects of a set of data such as: Measures of central tendency which are summary numbers that represent a single value in a distribution of scores (Vogt, 1999). They are expressed as an average score (the mean), the middle of a set of scores (the median) or the most frequently occurring (the mode). Measures of variability indicate the spread of the scores in a distribution (Creswell, 2002). Variance, standard deviation, and range all indicate the amount of variability of scores. Measures of relative standing are statistics that describe one score relative to a group of scores (Creswell, 2002). The percentile rank and z-score are two frequently used statistics. The other basic types of statistics are inferential statistics which are briefly explained here. Inferential statistics refer to statistics that enable a researcher to draw conclusions, inferences, or generalisations from a sample to a population. Inferential statistics are concerned with determining how likely it is that results based on a sample are similiar to these obtained from the population (Gay & Airasian, 2003). The question that guides inferential statistics is whether expected differences are real, significant ones or only the result of sampling errors. Inferences concerning populations provide only probability statements where you are never certain when making an inference about a population. In inferential statistics, there are many other factors that need to be learnt. A course on statistics would provide sufficient knowledge regarding inferential statistics. Other terms that are important for inferential statistics are standard error, the null hypothesis, tests of significance, the t-test, multiple regressions, chi square, and other more advanced statistical analysis which can be found in statistics texts. 1. 2. 3. 4. Why should you conduct a pilot study? What would you do with missing data? In qualitative research, how would you transcribe field notes? What are the skills needed in analysing data from quantitative research?

78

OUM

TOPIC 5

RESEARCH METHODOLOGY

SUMMARY
The topic on Research Methodology gives a brief explanation on the process of how to implement a research successfully. This topic needs to be read with understanding with other topics written in this module. Some of the terms and procedures would be better understood through reading other related topics. Questions are also given to help you understand the concepts and the process of the methodology of research. In order for you to comprehend this topic well, you need to read other topics because all topics are inter-related with each other.

OUM

79