REPORT ON CLASS PRESENTATION

Topic of presentation:

Validity in business research methods

Report Submitted to:

Sir Mazhar Manzoor

Report prepared by:

Mohammad Irfan (MBA3 SEC.A R.NO#51)

Subject of presentation:

Business Research Methods

FEDERAL URDU UNIVERSITY KARACHI GULSHAN CAMPUS

Validity is the extent to which a test measures what it claims to measure. It is vital for a test to be valid in order for the results to be accurately applied and interpreted. Validity isnt determined by a single statistic, but by a body of research that demonstrates the relationship between the test and the behavior it is intended to measure. There are three types of validity:
Content validity:

When a test has content validity, the items on the test represent the entire range of possible items the test should cover. Individual test questions may be drawn from a large pool of items that cover a broad range of topics. In some instances where a test measures a trait that is difficult to define, an expert judge may rate each items relevance. Because each judge is basing their rating on opinion, two independent judges rate the test separately. Items that are rated as strongly relevant by both judges will be included in the final test.
Criterion-related Validity:

A test is said to have criterion-related validity when the test has demonstrated its effectiveness in predicting criterion or indicators of a construct. There are two different types of criterion validity:
y

Concurrent Validity occurs when the criterion measures ar e obtained at the same time as the test scores. This indicates the extent to which the test scores accurately estimate an individuals current state with regards to the criterion. For example, on a test that measures levels of depression, the test would be said to have concurrent validity if it measured the current levels of depression experienced by the test taker.

Predictive Validity occurs when the criterion measures are obtained at a time after the test. Examples of test with predictive validity are career or aptitude tests, which are helpful in determining who is likely to succeed or fail in certain subjects or occupations.

Construct Validity:

A test has construct validity if it demonstrates an association between the test scores and the prediction of a theoretical trait. Intelligence tests are one example of measurement instruments that should have construct validity. Test validity for a pre-employment is a study undertaken and directed by the test publisher in accordance with certain professional st andards to measure what it is

supposed to measure. There are two main methods to demonstrate a tests validity, one is criterion validity and the other is content validity. Criterion validity assesses whether a test reflects a certain set of abilities. For instance, when an employer hires new employees based on normal hiring procedures like interviews, education, and experience. This method demonstrates that people who do well on a test will do well on a job, and people with low score on test will do poorly on a job. Content validity represents job function testing. The procedure here is to identify necessary tasks to perform a job like typing, design, or physical ability. In order to demonstrate the content validity of a selection procedure, the behaviors demonstrated in the selection should be a representative sample of the behaviors of the job. Another method that is used rarely because it is not very sophisticated is face validity. Face validity is the property of a test intended to measure something. It is based only on the appearance of the measure and what it is supposed to measure, but not what the test actually measures. Face validity is often contrasted with content validity because it is not validity in the technical sense. Face validity is a simple form of validity in which researchers determine if the test seems to measure what is intended to measure. Essentially, researchers are simply taking the validity of the test at face value by looking at whether a test appears to measure the target variable. On a measure of happiness, for example, the test would be said to have face validity if it appeared to actually measure levels of happiness. Obviously, face validity only means that the test looks like it works. It does not mean that the test has been proven to work. However, if the measure seems to be valid at this point, researchers may investigate further in order to determine whether the test is valid and should be used in th e future.

VALIDITY

In general, VALIDITY is an indication of how sound your research is. More specifically, validity applies to both the design and the methods of your research. Validity in data collection means that your findings truly represent the phenomenon you are claiming to measure. Valid claims are solid claims. Validity is one of the main concerns with research. "Any research can be affected by different kinds of factors which, while extraneous to the concerns of the research, can invalidate the findings" (Seliger & Shohamy 1989, 95). Controlling all possible factors that threaten the research's validity is a primary responsibility of every good researcher.

INTERNAL VALIDITY is affected by flaws within the study itself such as not controlling some of the major variables (a design problem), or problems with the research instrument (a data collection problem).

"Findings can be said to be internally invalid because they may have been affected by factors other than those thought to have caused them, or because the interpretation of the data by the researcher is not clearly supportable" (Seliger & Shohamy 1989, 95).
Here are some factors which affect internal validity:
y y y y y y y

Subject variability Size of subject population Time given for the data collection or experimental treatment History Attrition Maturation Instrument/task sensitivity

EXTERNAL VALIDITY is the extent to which you can generalize your findings to a larger group or other contexts. If your research lacks external validity, the find ings cannot be applied to contexts other than the one in which you carried out your research. For example, if the subjects are all males from one ethnic group, your findings might not apply to females or other ethnic groups. Or, if you conducted your research in a highly controlled laboratory environment, your findings may not faithfully represent what might happen in the real world.

"Findings can be said to be externally invalid because [they] cannot be extended or applied to contexts outside those in whic h the research took place" (Seliger & Shohamy 1989, 95).
Here are seven important factors affect external validity:
y y y y

Population characteristics (subjects) Interaction of subject selection and research Descriptive explicitness of the independent variable The effect of the research environment

RESEARCH VALIDITY

Validity is the extent to which a question or scale is measuring the concept, attribute or property it says it is. For example, if one is measuring an attribute of a product such as "Brand Recognition", how do we know that the question (or questions) used to measure Brand Recognition are valid? Usually, this is a result of what we know of the "perceived meaning" of the question from previous times it has been used. Validity can also be optimized by caref ul pretesting of alternative questions designed to measure the same concept. In the academic sphere, validity can take many forms. Construct validity for example refers to the ability of a measure to relate meaningfully to other similar measures used before. In commercial market research, many companies rely on "face validity" - the extent to which the respondent "knows" what is being measured and it seems sensible to them. Validity can be differentiated from Reliability, another property of "good" research . In practice, validity can also refer to the success of the project in retrieving "valid" results. There are many sources of error that can reduce the validity of a project including poor sample selection and resultant bias, simple coding errors, misunderstanding of management and research questions by the researchers and misunderstanding of the investigative questions by the respondents. Other errors include asking "leading questions", unconscious non -verbal prompts on "good answers", vindictive respondents, or inappropriate methodologies used to analyze the raw data. Competent market research agencies emphasize validity of results in their research conduct, even over presentation and colorful reports, which can sometimes obfuscate the actual validity of a market research survey.
y y y

Researcher or experimenter effects Data collection methodology The effect of time

Validity: the best available approximation to the truth of a given proposition, inference, or conclusion The first thing we have to ask is: "validity of what?" When we think about validity in research, most of us think about research components. We might say that a measure is a valid one, or that a valid sample was drawn, or that the design had strong validity. But all of those statements are technically incorrect. Measures, samples and designs don't 'have' validity -- only propositions can be said to be valid. Technically, we should say that a measure leads to valid conclusions or that a sample enables valid inferences, and so on. It is a prop osition, inference or conclusion that can 'have' validity. We make lots of different inferences or conclusions while conducting research. Many of these are related to the process of doing research and are not the major hypotheses of the study. Nevertheless, like the bricks that go into building a wall,

these intermediate process and methodological propositions provide the foundation for the substantive conclusions that we wish to address. For instance, virtually all social research involves measurement or observation. And, whenever we measure or observe we are concerned with whether we are measuring what we intend to measure or with how our observations are influenced by the circumstances in which they are made. We reach conclusions about the quality of our measures -conclusions that will play an important role in addressing the broader substantive issues of our study. When we talk about the validity of research, we are often referring to these to the many conclusions we reach about the quality of different parts of our research methodology. We subdivide validity into four types. Each type addresses a specific methodological question. In order to understand the types of validity, you have to know something about how we investigate a research question. Because all four validity types are really only operative when studying causal questions, we will use a causal study to set the context.

The figure shows that there are really two realms that are involved in research. The first, on the top, is the land of theo ry. It is what goes on inside our heads as researchers. It is we keep our theories about how the world operates. The second, on the bottom, is the land of observations. It is the real world into which we translate our ideas -- our programs, treatments, mea sures and observations. When we

conduct research, we are continually flitting back and forth between these two realms, between what we think about the world and what is going on in it. When we are investigating a cause-effect relationship, we have a theory (implicit or otherwise) of what the cause is (the cause construct). For instance, if we are testing a new educational program, we have an idea of what it would look like ideally. Similarly, on the effect side, we have an idea of what we are ideally trying to affect and measure (the effect construct). But each of these, the cause and the effect, has to be translated into real things, into a program or treatment and a measure or observational method. We use the term operationalization to describe the act of translating a construct into its manifestation. In effect, we take our idea and describe it as a series of operations or procedures. Now, instead of it only being an idea in our minds, it becomes a public entity that anyone can look at and examine for themselves. It is one thing, for instance, for you to say that you would like to measure self-esteem (a construct). But when you show a ten-item paper-and-pencil self-esteem measure that you developed for that purpose, others can look at it and understand more clearly what you intend by the term self -esteem. Now, back to explaining the four validity types. They build on one another, with two of them (conclusion and internal) referring to the land of observation on the bottom of the figure, one of them ( construct) emphasizing the linkages between the bottom and the top, and the last (external) being primarily concerned about the range of our theory on the top. Imagine that we wish to examine whether use of a World Wide Web (WWW) Virtual Classroom improve s student understanding of course material. Assume that we took these two constructs, the cause construct (the WWW site) and the effect (understanding), and operationalIized them -- turned them into realities by constructing the WWW site and a measure of knowledge of the course material. Here are the four validity types and the question each addresses: Conclusion Validity: In this study, is there a relationship between the two variables? In the context of the example we're considering, the question might b e worded: in this study, is there a relationship between the WWW site and knowledge of course material? There are several conclusions or inferences we might draw to answer such a question. We could, for example, conclude that there is a relationship. We mi ght conclude that there is a positive relationship. We might infer that there is no relationship. We can assess the conclusion validity of each of these conclusions or inferences.

Internal Validity: Assuming that there is a relationship in this study, is the relationship a causal one? Just because we find that use of the WWW site and knowledge are correlated, we can't necessarily assume that WWW site use causes the knowledge. Both could, for example, be caused by the same factor. For instance, it may be tha t wealthier students who have greater resources would be more likely to use have access to a WWW site and would excel on objective tests. When we want to make a claim that our program or treatment caused the outcomes in our study, we can consider the internal validity of our causal claim. Construct Validity: Assuming that there is a causal relationship in this study , can we claim that the program reflected well our construct of the program and that our measure reflected well our idea of the construct of the measure? In simpler terms, did we implement the program we intended to implement and did we measure the outcome we wanted to measure? In yet other terms, did we operationalize well the ideas of the cause and the effect? When our research is over, we would like to be able to conclude that we did a credible job of operational zing our constructs -- we can assess the construct validity of this conclusion. External Validity: Assuming that there is a causal relationship in this study between the constructs of the cause and the effect; can we generalize this effect to other persons, places or times? We are likely to make some claims that our research findings have implications for other groups and individuals in other settings and at other times. When we do, we can examine the external of validity

these claims. Notice the that validity how each type

question

addresses

presupposes an affirmative answer to the previous one. This is what we mean when we say that the validity types build on one another. The figure s hows the idea of cumulativeness as a staircase, along with the key question for each validity type. For any inference or conclusion, there are always possible threats to validity -reasons the conclusion or inference might be wrong. Ideally, one tries to reduce the plausibility of the most likely threats to validity, thereby leaving as most plausible the conclusion reached in the study. For instance, imagine a study examining whether there is a relationship between the amount of training in a specific tech nology and subsequent rates of use of that technology. Because the interest is in a relationship, it is considered an issue of conclusion validity. Assume that the study is completed and no significant correlation between amount of training and adoption ra tes is found. On this basis it is concluded that there is no relationship between the two. How could this conclusion be wrong -- that is, what are the "threats to validity"? For one, it's possible that there isn't sufficient statistical power to detect a r elationship even if it exists. Perhaps the sample size is too small or the measure of amount of training is unreliable. Or maybe assumptions of the co relational test are violated given the variables used. Perhaps there were random irrelevancies in the stu dy setting or random heterogeneity in the respondents that increased the variability in the data and made it harder to see the relationship of interest. The inference that there is no relationship will be stronger -- have greater conclusion validity -- if one can show that these alternative explanations are not credible. The distributions might be examined to see if they conform with assumptions of the statistical test, or analyses conducted to determine whether there is sufficient statistical power. The theory of validity, and the many lists of specific threats, provides a useful scheme for assessing the quality of research conclusions. The theory is general in scope and applicability, well -articulated in its philosophical suppositions, and virtually impossible to explain adequately in a few minutes. As a framework for judging the quality of evaluations it is indispensable and well worth understanding.
Links
http://coles.kennesaw.edu/drbob/dr.bob/Introduction%20to%20Validity.pdf http://www.asiamarketresearch.com/glossary/validity.htm http://psychology.about.com/od/researchmethods/f/validity.htm