Chapter 11.

Data analysis and report writing

11.1 OVERVIEW

11.1 OVERVIEW 11.2 INTRODUCTION 11.3 METHODS 11.3.1 Study design 11.3.2 Subjects 11.3.3 Data collection 11.3.4 Data analysis 11.4 RESULTS 11.4.1 Descriptive statistics 11.4.2 Analytical statistics 11.5 DISCUSSION 11.6 CONCLUSIONS

Once you have developed your study idea, developed your study plan, and executed your study successfully, the fun part begins! Now is the time to discover the truth behind your study questions. Is treatment A better than treatment B? Does it depend on what group of patients received the treatment (eg, young versus old)? Are patients really better off receiving this new plate or surgical technique? Is it possible that the old way is the best way? Is it possible that it doesnt matter what technique you use? Finding out these answers is the motivating force behind performing a clinical study. If you paid attention to detail in your study planning and worked hard in ensuring the quality and validity of your data collection methods, there is no reason these questions cannot be answered, at least in your study population. Many investigators reach this point in the study so exhausted that they put the data aside and forget about it for a while. Its not uncommon to leave such a tremendous effort behind by not getting to the data analysis. Discipline in sticking to the effort until the end is paramount for this phase of your study. Unfortunately, we cannot just push a button and all our results, tables, and figures pop up on our computer screen. Performing the data analysis takes a combination of expertise, discipline, and patience in carefully handling the data in accordance with the data analysis plan you set forth in the study protocol [Part 2; Chapter 6] . This goes back to assembling a complete research team. Though resources may limit you, it pays to get an epidemiologist or biostatistician involved in this phase of your study, if even as a consultant. Ideally, this person was involved in helping you develop the analysis plan in the protocol. They can help you with basic elements of the data analysis to be sure you are handling your data in an efficient and effective manner. Furthermore, if available to you, they can perform the whole analysis in a complete and robust manner which will get the best out of your hard efforts up to this point. On the other hand, some investigators enjoy analyzing

their own data and have the expertise to do it. If so, go for it! If not, you can learn a lot by working along side a person who does the analysis for or with you. Like we did in Part 2 of this module where we discussed study design principles following the outline of a protocol [Part 2: Developing the Study Plan] , we will discuss data analysis and reporting principles following the outline of a study report or manuscript for publication, Table 11.1. Keep in mind that each sponsor (if applicable) and journal may have different requirements [Part 5, Chapter 12] and therefore the outline may change slightly; however, these basic elements should be included in any study report or manuscript for publication.

11.2 INTRODUCTION

You can borrow from your Background and Significance section of your protocol when you write this section. It will be important to be relatively brief, likely far fewer characters than your protocol. The key is to inform the reader as to why your topic is important. It is not necessary to spend a tremendous amount of effort reiterating what others have already discussed in previous studies on your topic. Try to make it novel and to the point. The introduction should end with a clear and concise description of your study purpose. If you had more than one purpose or a secondary purpose, list them separately. For example, you can state something like this: There is a tendency to change your objectives based on findings you have become aware of since the study. Be careful with this. It is ok to report secondary findings that you did not anticipate, but it is important that they are reported in their context and not as primary objectives. Bottom line - list the objectives you set out to answer. They should be consistent with the specific aims from your protocol [Part 2; Chapter 3] . Statistically non-significant findings are still significant!

11.3 METHODS

This is where you tell your reader what study design you chose to use to answer your primary and secondary objectives. It also gives you the opportunity to describe the institution(s) in which you recruited your subjects. You get to tell your story of how you identified and recruited subjects, the characteristics of these subjects, how many participated and how many you lost to follow-up.

11.3.1 Study design

This section is short and to the point. Here you tell the reader whether your study was a randomized or quasi-randomized controlled trial, cohort study, case-control study, case-series and whether you used additional methods such as matching, block randomization, stratified randomization, etc. [Part 2; Chapter 5].

11.3.2 Subjects

It is very important to clearly and thoroughly describe your study populations. Readers need to know if your results and conclusions may be generalized to their population. Additionally, it is important to place your study in time. Technological advances,

changes in patient care procedures, and differences in reporting practices at different times can affect the outcomes and interpretation of your study1. The following items should be included in this section: Explanation of inclusion and exclusion criteria. Institutions in which you identified and recruited your patients. Time period in which you collected your data. For retrospective studies, give the dates during which the data were originally collected as opposed to when you abstracted them from the study.

11.3.3 Data collection

It is important in this section to describe clearly and concisely the process by which you identified and recruited subjects and whether they finished the study. Full accountability of every subject should be attempted. When possible, provide a schematic summary of the study showing the number and disposition of participants at each stage, Figure 11.1. The schematic summary should include: The total number of subjects approached. The total number of subjects found eligible and ineligible. The number of subjects who agreed to participate and signed informed consent. The number of subjects who did not complete the study (eg, lost to follow-up, withdrew, died). The number of subjects who completed the study and whose data are included in the final analysis. Make every attempt to describe the following subjects:

Those evaluated who did not meet the study criteria. Those subjects who were approached but declined to participate. Those subjects who were enrolled but were withdrawn or dropped out. Those subjects who were enrolled but were lost to follow-up. Thos subjects who were enrolled and completed the study. Subjects who agree to participate may differ in important ways than those who decline to participate. For example, they may be more likely to take risks or more motivated to get better and comply with the treatment protocol. Patients may be withdrawn or drop out willingly for various reasons, some of which may bias the interpretation of the results. Subjects who are lost to follow-up may be those subjects who are least satisfied with the results of their treatment or may be doing so well they do not feel they need to be evaluated any longer. Either scenario could bias the results. Studies with low follow-up rates (ie, <90%) should be interpreted cautiously for this reason. Finally, the final sample should be described completely. This is generally done in the results section.

Describe the nature and duration of follow-up effort. This should come directly from your protocol assuming you followed these procedures. For example, you may state something like this:

Upon study entry, we obtained the following:

Patients mailing address, telephone number, and email address. The name and address of their primary care physician. The name, address and phone number of three people at different addresses with whom the patient does not live who are likely to be aware of the patients location.

Study personnel called patients to remind them of upcoming study visits and patients were contacted no less frequently than once every three months to maintain contact and planned change in residence.

Finally, describe the baseline prognostic factors and outcome measures that were collected. If special instruments were used to define these, identify them, provide any background info as to their validity, reliability, and responsiveness, and if possible provide a copy of the instrument in the paper. A clear description of the instrument s content, scale, and interpretation with appropriate references should be included whether you provide an actual copy of the instrument or not.

Figure 11.1. Example of a cohort study schematic summary.

11.3.4 Data analysis

The Uniform Requirements for Manuscripts Submitted to Biomedical Journals summarizes the overall intent of statistical reporting: Describe statistical methods with enough detail to enable a knowledgeable reader with access to the original data to verify the reported results2. The statistical comparisons you make here are the same as the ones you specified in your protocol before any data were collected1. Therefore, this section should restate what that plan was. Assuming you followed that plan, this section can be a mirror image of the protocols section with some slight edits as needed. It is understandable that the choice of specific statistical tests may depend on the quality of the data in the end (eg, whether continuous data were normally distributed or not), so the specific test often cannot be stated in advance. In the end, this section should provide a clear and concise description of how you analyzed your data so that someone could repeat it. If you performed a randomized controlled trial, you should be clear as to whether the statistical analysis was on the basis of intent-to-treat [Part 2; section 5.1] . Furthermore, it is a good idea to reiterate you power calculations so as to validate whether or not you achieved the necessary number of patients in your study that you had planned for. Finally, identify the statistical package or program (with its corresponding reference) used to analyze the data. Results may very slightly from package to package due to discrepancies in validation and updating.

11.4 RESULTS

The results section should be quick and to the point. In other words, there should not be too much explaining or justifying the findings. This is left for the discussion. The flow of the results section should begin with descriptive statistics of study groups, findings with respect to measured risk factors or outcomes (analytical statistics) and accompanying tables and figures as necessary. 11.4.1 Descriptive statistics The presentation of descriptive data on the study population is important for the following reasons: It enables you to determine the comparability of study groups at baseline and evaluate the likelihood of any selection bias[Using didactic] or confounding [Part 2; section 5.6.2.] . The baseline characteristics of the study population can help in determining the generalizability [Using, External Validity] of the results of study population to other study populations. This is typically the first table in your report or manuscript, Table 11.2

Table 11.2. Baseline data for hip fracture patients treated with a Dynamic Hip Screw (DHS) or Percutaneus Compression Plating (PCCP)

DHS No. or mean Age (years) AO Classification A11 stable A12 stable A13 stable A22 unstable A22 unstable A23 unstable 5 15 5 6 10 3 11 34 11 14 23 7 9 6 1 2 14 7 83 % or range 64-98

PCCP No. or mean 82 % or range 65-96

23 15 3 5 36 18

From this table, we note that both groups are relatively equal with respect to age; however, if we combine fracture classifications into stable versus unstable fractures, we find that 44% of those treated with the DHS were unstable fractures compared with 59% of those treated with the PCCP. This may put the PCCP group at an unfair disadvantage if unstable fractures are more likely to give rise to a poorer outcome. An unequal distribution between treatment groups of an independent factor that is also associated (negatively or positively) with the outcome constitutes a potential confounding variable [Part 2; section 5.6.2] . Potentially confounding variables should be dealt with in the analytical analysis. [see example below]. 11.4.2 Analytical statistics A thorough description of the purpose of analytical studies and hypothesis testing is outlined in the Developing the Study Plan part of this module [Part 2; section 6.2] . This section will focus on reporting your results. Your outcomes will either be categorical or continuous or a combination of both [Part 2; section 6.2] . This section will discuss how to present both types of outcomes. Statistics can be as much of an art as it is a science. Furthermore, there are myriads of ways to display your data. Going through all these options is beyond the scope of this module; however, we will present some important evidence-based medicine concepts ways in which data reporting should have a significant impact on your audience and allow them to apply your findings clinically. As some say, the p-value is overrated!.

Categorical outcomes Often surgeons want to know the proportion of patients who experience a certain outcome, either negative (e.g. non-union) or positive (e.g. those who have an excellent clinical outcome). In many studies, the authors simply report these dichotomous outcomes as proportions for the treatment and control groups and a pvalue to tell us if the results are significant. A common way of reporting these outcomes is presented in Table 11.3.

Table 11.3 Hypothetical outcomes comparing the LCP to the standard plate Treatment group LCP (N=119) Outcome Union Nonunion Complications n 107 12 12 % 90 10 10 Standard plate (N=112) n 90 22 11 % 80.0 80.0 10 0.04 0.04 0.92 P-value*

*Hypothesis testing using the chi-square test. RR is simply the proportion of patients with the outcome in the treatment group divided by the proportion of patients with the outcome in the control group. In this case, 0.10/0.20 = 0.50 The RRR is |1-RR| x 100, or in our case, (1-0.5) x 100 = 50%. A relative risk reduction of 50% means that the LCP reduced the risk of non-union by 50% compared with the control (standard plate) treatment. If the treatment increases the risk of a bad event, we call that relative risk increase (RRI). Furthermore, when a treatment increases the probability of a good event, the term we use is relative benefit increase (RBI) (Table 11.4). The RD is the absolute difference between the proportions, 0.20-0.10 = 0.10 or 10%. The NNT represents the number of patients one would need to treat in order to prevent a negative outcome (or allow a positive outcome, depending on which outcome is being evaluated). The formula is 1/RD. In our example, 1/.10 = 10; therefore, for every 10 patients treated with the LCP, one non-union can be prevented compared with a standard plate (Table 11.5).

Table 11.4. The relationship among relative and absolulte risk reduction, risk increase and b

Relative 1-|T/C| Treatment reduces the risk of bad event vs.control Relative risk reduction (RRR) Relative risk increase (RRI) Relative benefit increase (RBI)

A |

Treatment increases the risk of bad event vs.control

Treatment increases the probability of a good eventvs. control

Table 11.5. Summary of the ways to report non-unions for fictional data comparing the LCP with a standard plate. Treatment (T) (n=30) Control (C) (n=30) No. (%) 6(20) RR RRR RD NNT

No. (%) 3(10)

T/C 0.50

1-(T/C) 0.50

C-T 0.10

1/(C-T) 10

As discussed in Part 2 (section 6.2) , rates can also be reported and a statistical test like the chi-square test performed which will provide a p-value. While the p-value may be statistically significant (p<0.05), reporting the data in this fashion provides little additional value with respect to applying your findings in a practical sense. The measures above are relatively simple to perform and are highly recommended when comparing proportions. You may also choose to use statistical software to perform multivariate regression. Regression methods such as Cox, negative binomial, and logistic regression also provide a point estimate like the relative risk. They have the added benefit of allowing you to control for potential confounding factors and therefore are a very useful tool when reporting your results. You should have collected all factors that you though may be potential confounder and have a plan for controlling for them in your study protocol. An example of how confounding may effect your conclusion if you dont use a method to control for these factors (eg, stratified analysis or regression) is given below. When evaluating LCP versus a standard plate, one may want to adjust for smoking status (a potential confounder) in this way (Table 11.6):

Table 11.6. Fracture non-union among smokers and non-smokers for fictitious data. LCP (N=100) n Smokers Nonsmokers Total 11/15 3/85 % (73.3) (3.4) Standard (N=100) n 31/42 2/58 % (73.8) (3.3) PR 0.94 1.02

14/100

(14.0)

33/100

(33.0)

0.42

RR = relative risk (cast/ORIF)

In this example, the proportion of patients treated with an LCP who end up with a nonunion is 14% compared with 33% receiving a standard plate; a relative risk of 0.42, which translates to a 58% reduction in risk. However, there happen to be far less smokers in the LCP group than the cast group, and smokers have a higher non-union rate no matter what treatment they receive, whether LCP or a standard plate (73% non-union). When the data are stratified, one can quickly see that there really is no difference in non-union between the LCP and standard plate groups in either smokers or non-smokers (73% nonunion in each group among smokers, and about 3% in each group among non-smokers). The difference is that more smokers ended up in the standard plate group. When we adjust for smoking, the relative risk of nonunion among those receiving an LCP is 0.99 compared with a standard plate. Continuous outcomes Continuous outcomes are very common in orthopedic research. Examples include operative time, blood loss, range of motion, visual analog pain scales, and various outcome scores that often fall on a 0 to 100 point scale [Part 2; section 6.2] . As discussed in Part 2 (section 6.2) these outcomes are typically presented as a mean with a standard deviation, Table 11.7. Hypothesis testing is performed using Students ttest or the Pearsons product moment-correlation coefficient. Table 11.7. Hypothetica Foot and Ankle Outcome Scores*. Treatment LCP N 119 Mean 88 SD 17.5 Min 41 Max 98 P-value P<0.001

Standard Plate

112

76

10.2

34

91

*Scores normalized to a total possible of 100 points. Two-sample t-test. If there are potential confounding variables that need to be controlled for such as age, fracture severity, smoking status, etc. you can use linear regression or analysis of variance. These are discussed superficially in Part 2 (section 6.2) but a thorough discussion of multivariate methods is beyond the scope of this module.

11.5 DISCUSSION

Since the results section is intended to report your finding without interpretation, the discussion section allows you to put your findings in context. However, the discussion section has a tendency to take on a life of its own. It is very common for this section to become long and difficult to read. It is also common for this section to serve as a platform for one to give their opinions with respect to the treatment intervention being evaluated. Some go as far as to discuss public policy changes in their discussion. There are no standard guidelines for this section which is probably why authors take the liberty to write everything they were not able to write in the previous three sections of the manuscript. Be careful with this approach. It is very refreshing to read a discussion that is clear and concise. If you present your methods and results in an effective way, then the discussion allows you to interpret those results in such a way as to allow the reader to make up his/her own mind in the end. In light of this background, here are some guidelines to consider when writing the discussion section which will prevent you from falling into the trap of writing too much with too little substance. Consider addressing these issues in the order presented below: 1. Discuss the implications of the primary analyses first. Secondary analyses should be discussed later and described as explanatory. 2. Distinguish between statistical and clinical significance. Statistical significance relates to how likely the observed effect is due to chance (ie, sampling variation). Clinical significance relates to the magnitude of the observed effect. Lets look a little closer at these concepts. Statistical significance depends on three parameters: Sample size (the larger the sample size, the easier to demonstrate statistical significance) Variability in patient response, either by chance or by non-random factors (the smaller the variability, the easier to demonstrate statistical significance) Effect size, or the magnitude of the observed effect between group (the greater the size of the effect, the easier to demonstrate statistical significance) The size of the effect is what really interests us, and these are the values reported as

relative risk (RR) relative risk reduction (RRR) risk difference (RD) (see 11.3.2). These values are given as a single number, and therefore they are referred to as point estimates. A range of values around the point estimates can be calculated, which in turn gives us an interval estimate. The interval estimate is also known as the confidence interval; a 95% confidence interval tells us that an experiment conducted 100 times would result in the point estimate falling inside the confidence interval 95 times. For example, if the RRR = 0.45 and its 95% confidence interval = 0.35 0.55, that means that a repeat of the experiment conducted 100 times would yield a RRR somewhere between 0.35 and 0.55 ninety five times.

If the confidence interval does not include the number one for RR (or the number zero for RRR), the point estimate will be statistically significant. As a hypothetical example, consider the case that nonunion occurred in 14 of 100 cases of open fracture receiving external fixation and 26 of 100 cases in those receiving ORIF. The RR would be 0.46, and the 95% confidence interval would be 0.30 to 0.97. Here, the confidence interval excludes 1.0 and this tells us that the p-value will be significant at the 0.05 level; in fact, in this example, p=.034, telling us that the point estimate of 0.46 is statistically significant. Now take a look to see what happens when we change the rate in the ORIF group slightly from 26 of 100 cases of nonunion to 24 of 100 cases. The RR would be 0.58 and the 95% confidence interval would be slightly wider, 0.32 to 1.06, p=0.07. Here, the number 1.0 is included in the confidence interval telling us that the point estimate of 0.58 is not statistically significant. As you can see, the confidence interval is closely related to the p-value. With the use of confidence intervals, you can use your clinical experience to judge if the size of the effect is large enough to be important clinically.

3. Discuss any weaknesses and strengths? in your research design or problems with data collection, analysis, or interpretation. Disclosure of weaknesses or limitations is often difficult; however, honesty is the cornerstone of clinical research. Readers will often identify these anyway, so discussing them gives you an opportunity to defend them and/or recommend ways to overcome them when others attempt to answer similar questions. 4. Discuss the results in the context of the published literature Describe the similarities and differences of your work with that of other authors who have done similar studies. Make an attempt to explain why your findings may be similar and why they may be different. Be cautious not to attempt to review the whole body of literature on this topic. Again, brevity is important so be selective in which studies you choose to review.

5. Discuss the generalizability of the results The purpose of your study is to produce results that the entire orthopedic community can apply to their practice. The ability to generalize your findings is dependent on your study population, its inclusion and exclusion criteria, and other factors such as your follow-up rate.

11.6 CONCLUSIONS

Limit your conclusions to only those supported by the results of your study. Unsupported conclusions are very common in scientific research. Consider the following principles1:

You should provide equal emphasis on positive and negative findings. Results of secondary or post hoc analyses should be presented as explanatory. Conclusions should be based on fact and logic, not supposition or speculation. Studies using surrogate endpoints (eg, muscle strength, range of motion, perhaps even bony union) should be interpreted with caution. In other words, just because a patient has good shoulder strength and range of motion does not necessarily mean they have a good final outcome if they cannot perform activities of daily living.