Research Methods - Mar 2013

Q2. Explain what we mean by the concept of statistical significance and draw on a published research paper to illustrate an application of the concept
A test of statistical significance is a tool of inference used by researchers to assess the evidence in favour for or against some claim about a population from which a smaller sample has been drawn. The concept of statistical significance is intimately linked with the concept of null hypothesis testing where these claims are formulated as null H0 and alternative Ha hypotheses which are statements related to certain population parameters (most commonly the mean). There are numerous forms of significance testing e.g. Z test , Student t-test, F-Test , etc but the key concept is that a these tests assume a certain distribution of the test parameter under a null hypotheses and measure the likelihood (p-value) of the observed results occurring by chance alone ASSUMING the null hypothesis is true. This statistic is then compared against a level of signifance that is decided prior to the test (typically 0.05 or 0.01) where if the pvalue is less than or equal to the significance level the null hypotheses should be rejected in favour of the alternative hypotheses. (Thompson (1994) as quoted in Miller & Salkind (2002)) The result of the significance test is always given in terms of the null hypothesis where the options would be to either reject H0 in favor of Ha" or "do not reject H0". (Note this is not exactly the same as saying accept the alternative hypotheses). The test of significance is therefore essentially a means for researchers to avoid saying that there is a relationship where in fact there is none. Flyvberg et al (2002) conducted a research study focused on making a statistically significant study of cost escalation in transportation infrastructure projects based on a sample of 258 different project with the aim of demonstrating that there is a systematic under-estimation of cost at the point of project approval. The researchers first established a way of defining a metric around which the hypothesis could be developed. This metric was the cost escalation % of a project which is the completed cost of the project divided by the budgeted cost. From this metric, a null hypotheses was then proposed that there is equal chance of under-estimation and overestimation where the alternative hypotheses is that under-estimation is more common than over-estimation. A twosided test using the binomial distribution which is a test of deviations from a theoretically expected distribution of observations into only two categories (In this example was represented by a binomial of n=258 and p=50% which was checked against the observed data) Based on this test, the null hypothesis was rejected as the p-value was less than the significance level of 0.001. Secondly another null hypotheses was that the size of the under-estimation and over-estimation are similar. A MannWhitney test that is used to assessing whether one of two samples of independent observations tends to have larger values than the other was applied to check the p-value of the observed data. The null hypotheses was then rejected as the p-value was less than the significance level of 0.001.

The paper goes further to perform analyses of sub-sets of data within the total 258 data points to understand if there is a statistically significant difference between them in terms of categorical data e.g. types of project , locations of project, year of project decision and completion. This was performed through the use of a F-test in one-way analysis of variance which is used to assess whether the expected value of a quantitative variable within multiple pre-defined groups differ from each other. In this set of results, the null hypotheses could be rejected for project type and location but not for year of sanction/completion. The authors appear to have avoided the pit-fall of only presenting the significance level in absence of the effect size information as they have also relied on simple histograms to highlight the magnitude of the differences. For example the binomial two tailed test was reported alongside the statement that the likelihood of actual costs being higher than estimated costs was 86% (as opposed to 50% - which is the expected figure if there were no difference in the direction of the under/over -estimation). Another example would be that Mann Whitney test was complemented with the reporting of the actual completed costs were reported as being on average 28% higher than estimated costs. [719 vs Target 500 Words]

References
Flyvbjerg, Bent, Mette K. Skamris Holm, and Sren L. Buhl, 2002, "Underestimating Costs in Public Works Projects: Error or Lie?" Journal of the American Planning Association, vol. 68, no. 3, Summer, pp 279-295 Miller, Delbert C., and Neil J. Salkind, 2002, eds. Handbook of research design and social measurement. Sage Publications, Incorporated. p.p 384-287 Thompson, Bruce. "The concept of statistical significance testing." Practical Assessment, Research & Evaluation 4.5 (1994): 5-6.