Digital Data Integrity QA for Multi-Institutional Clinical Trials

William L. Straube1, Walter R. Bosch1, John W. Matthews1, and Roxana J. Haynes1, James A. Purdy1,2 1Washington University School of Medicine, St. Louis, MO and 2UC Davis School of Medicine, Sacramento, CA Supported by NCI Grant U24 CA 81647

Abstract
Purpose: The Image-guided Therapy QA Center (ITC) as part of the NCI-sponsored Advanced Technology QA Consortium (ATC) has nearly 15 years experience in performing data integrity QA review for multi-institutional advanced technology clinical trials that require digital data submission. This presentation will report on that experience. Method and Materials: Participants in some advanced technology multi-institutional clinical trials must be able to submit imaging data as well as RT objects (CT, RT Structure Set, RT Dose, and RT Plan) to the ITC for protocol compliance QA review of contoured volumes and dose coverage/heterogeneity. Data are sent via FTP or on media. However, prior to that QA review, experienced personnel at the ITC carefully review each digital data set in regard to completeness of protocol required elements, format of data, and possible data corruption. Results: Thus far over 4000 data sets have been submitted to ITC. Unfortunately, data often need resubmission due to problems discovered by ITC. Errors in submission can be divided into four categories: 1) misuse of UI of treatment planning system (TPS), 2) misunderstanding of protocol requirements, 3) user error with digital data transfer software, and 4) updated TPS software, whose data export feature no longer is ATC compliant. Statistics of number of resubmissions required as well as specific details of these problems will be presented. Conclusion: Digital data submission of complete 3D data set is essential for QA of advanced technology clinical trials. However, collection of these data requires review and troubleshooting by experienced personnel to ensure subsequent protocol compliance QA and later still, quality data analysis. A significant portion of the ITC workload involves digital data integrity QA to ensure quality of submitted digital data.

Participating Institution
Data Submission Workstation FTP Client/ DICOMpiler DICOM Pt 10 files

ITC
Data Integrity QA Tools RRT Web Server

Study Chair

RTOG / DICOM

SFTP

RTP System

SFTP Server

Web-based RRT

TPS Files

CD-R
RTOG format

Tape
Fed-Ex, etc.

RTOG/ DICOM Data Import Web-based RRT

RTOG HQ

C
Web-based RRT

Figure 1. Diagram illustrating the flow of data from the institution through the ITC to the reviewers. The ITC performs digital data integrity QA to ensure that the data are reviewable. Figure 2. Tool used for renaming of structures to follow a uniform naming convention. Uniform structure names permit comparison of DVHs among subjects enrolled on a clinical trial protocol. While standard structure names for each of the ATC-supported protocols are posted on the ATC website (http://atc.wustl.edu), submitted data often differ from the standard. Correct interpretation of submitted structure names may require visualization of contours, especially for head and neck cases.

Introduction
The Image Guided Therapy Quality Assurance (QA) Center (ITC) has been accepting, processing and reviewing digital data submissions for support (QA and analysis) of advanced technology protocols for the past 12 years. For the past 7 years the ITC has been a part of the NIH funded Advanced Technology Consortium (ATC) which consists of national cooperative groups and QA centers. Over 4000 case data sets have been submitted and processed for review. For protocols supported by the ATC, institutions are required to submit the complete 3D treatment planning data set from their treatment planning system. Many of the commercial treatment planning systems in use have implemented digital data export in a standardized format (either DICOM or RTOG data exchange) that can be processed by the ITC and made available to reviewers via a web based Remote Review Tool (RRT) that allows the reviewer to assess the dose volume statistics and structures as planned by the institution and compare these to protocol guidelines. The receipt of the data by the ITC requires review by the personnel at the ITC for completeness and integrity of the data. We refer to this review as digital data integrity QA. Often data does not come to the ITC in a reviewable form, and the ITC must intervene and investigate issues that need resolution before the data can be processed and reviewed. Thus, at present, the submission and review of digital data is not a totally automated process and requires human intervention to make possible the review of a large number of the cases that are submitted to the ITC. The QA tools and procedures developed by the ITC have made practical the processing of large amounts of protocol data for review and analysis. Nevertheless, the receipt of reviewable digital data is often an iterative process that requires repeated correspondence with the submitting institution.

Figure 5. This figure shows three examples of data requiring intervention by ITC personnel before they can be reviewed by study chairs. Figure 5A shows an instance where non square CTs were submitted and the numbers of rows and columns were incorrectly specified. Figure 5B is an example in which a CT gantry tilt was used to scan a patient, resulting in a misalignment between CTs and structures (arrows indicate urethra contour and actual location of urethra). Figure 5C shows the displacement of the dose grid (isodose of 0.1 Gy in lower right corner) and the CT and structures of data set submitted for a phantom irradiation.

Results
RTOG Protocol 0126 0413 0232 0522 0236 0321 0117 0521 # of submissions 183 2100 37 26 27 73 16 18 # Cases requiring ITC intervention 56 570 7 9 3 5 3 7 % Cases requiring ITC intervention 32 % 27 % 19 % 35 % 11 % 7% 19 % 39 % Interval for which statistics are given
100

Discussion
Rate of most common errors requiring resubmission

Percent of Errors Encountered

1/2006 6/2006 2/2005 6/2006 1/2006 6/2006 1/2006 6/2006 1/2006 6/2006 1/2006 6/2006 1/2006 6/2006 1/2006 6/2006

90 80 70 60 50 40 30 20

Our previous report showed that the calculation of DVHs can vary among treatment planning systems, and that, for the sake of consistency in quality assurance and dose-volume analysis of submitted data, it was necessary for the ITC to recalculate DVHs (Straube, et al., DVH Analysis: Consequences for Quality Assurance of Multi-Institutional Clinical Trials. Presented at the 48th Annual Meeting of the AAPM, Seattle, WA, 2005.).
% disagreement between DVH volumes calculated at the ITC and those submitted from various TPSs
30

PTV - 75 cc
1.2
20

PrecisePlan CMS Eclipse Helax Pinnacle

1
10

0 Missing protocol Required elements Digital Data Transfer problems (FTP, SFTP) Misuse of Treatment Planning System Export User Interface

Relative Volume

0.6

ITC Institution

% disagreement

10

0.8

0 0 -10 100 200 300 400 500 600

TOTAL

2480

660

27 %

0.4

Type of error
0.2
-20

Methods And Materials

The ITC has been receiving digital data for advanced technology protocols for 12 years utilizing what is called Method 1. Figure 1 shows a flow diagram which illustrates the path of the data from submission to review utilizing this data submission method. Data is converted by the institutions treatment planning system to either DICOM or RTOG data exchange files which are then sent to the ITC via SFTP or Media. The ITC reviews the digital data at the time of receipt to ensure that it is complete and ready to be processed so that it can be reviewed using the ATCs RRT. Once the digital data integrity review is complete and the data are deemed ready for processing, the data are extracted into a proprietary file format using tools developed by the ITC, so that it can be viewed via the RRT. Included in this processing is the renaming of structures (Figure 2) to a standard naming convention that allows the recalculation of DVHs and the later analysis of dose volume statistics among subjects in a clinical trial. A significant portion of the data submissions are incomplete or cannot be processed or reviewed for a number of reasons. These problems require human intervention to request resubmission of the data or to resolve issues with the data before it can be made available for review.

Table 1. This table shows the rate of problems requiring intervention by the ITC staff for each RTOG protocol supported by the ITC. 2100 submissions were received for the 0413 protocol (since the protocol was activated), a large phase III study involving partial breast irradiation. Overall for the data collected on 2480 submissions, 660 or 27% required intervention by the ITC staff. Often this intervention included iterative communications with personnel at the institution submitting the data.

Figure 3. This chart shows the rate of specific errors that are seen on a daily basis at the ITC. Overall 27% of cases submitted require human intervention by the ITC due to errors in submission of the data. 44% of these errors are caused by a misuse of the treatment planning system export user interface.

0 0 20 40 60 80 100
-30

Dose (Gy)

Volume (cc)

The results shown here indicate that digital data integrity QA and the intervention of ITC personnel are essential to permitting a sizable fraction of submitted data to be reviewed in ATC-supported cooperative group trials. The ITC is thus a critical resource in maintaining the quality of these data.

Conclusions
The processing of digital data for the review of advanced technology clinical trials is not a totally automated process. A focused review of the data collected over the past 6 months or longer shows that approximately 27% of the data submitted requires human intervention in order to obtain correct, reviewable digital data (Table 1). Procedures and tools developed by the ITC have made possible the collection of data for these studies. The most common sources of problems in digital data submission are the following (Figure 3): Errors in the use of digital data submission software on the treatment planning device. Errors in the understanding of the required protocol elements. Errors in the use of FTP and SFTP software. The receipt of reviewable data is often an iterative process that requires repeated, direct communication (Email and Telephone) with an institution.

Categories of Submission Problems

Over the years several issues have been seen consistently which require intervention by the ITC personnel. 1. Misuse of Treatment planning system data export capabilities. 2. Missing protocol required elements or mistakes in protocol understanding. 3. Error in use of digital transfer software 4. New release of treatment planning system with inability to correctly submit ATC compliant data. Problems in categories 1,2, and 3 are seen on a daily basis. Category 4 occurs much less frequently, but is much more complicated to resolve because it requires software changes by the vendor.

Figure 4. A commonly observed problem is the incorrect setting of the grid margin (3D calculation volume) and the dose grid resolution on a treatment planning system that submits data for Mammosite treatment plans on a partial breast irradiation protocol. Note the breaks in the isodose lines (indicated by the arrows) and coarseness of the isodose lines in this example.