STATE OF CALIFORNIA

DEPARTMENT OF MOTOR VEHICLES

An Evaluation of the Validity of California's Driving Performance Evaluation Road Test

By Patricia A. Romanowicz and Robert A. Hagge

August 1995

Research and Development Section Division of Program and Policy Administration California Department of Motor Vehicles RSS-95-154

REPORT DOCUMENTATION PAGE

Form Approved OMB No. 0704-0188

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 222024302, and to the Office of Management and Budget, Paperwork Reduction Project (0704-0188), Washington, DC 20503.

1. AGENCY USE ONLY (Leave blank) 4. TITLE AND SUBTITLE

2. REPORT DATE

3. REPORT TYPE AND DATES COVERED 5. FUNDING NUMBERS

August 1995 An Evaluation of the Validity of California's Driving Performance Evaluation Road Test
6. AUTHOR(S)

Patricia A. Romanowicz and Robert A. Hagge

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

California Department of Motor Vehicles Research and Development Section P. O. Box 932382, F-126 Sacramento, CA 94232-3820
9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)

8. PERFORMING ORGANIZATION REPORT NUMBER

RSS-95-154
10. SPONSORING/MONITORING AGENCY REPORT NUMBER

11. SUPPLEMENTARY NOTES

12a. DISTRIBUTION/AVAILABILITY STATEMENT

12b. DISTRIBUTION CODE

13. ABSTRACT (Maximum 200 words)

This report presents findings of an evaluation of the validity of the Driving Performance Evaluation (DPE) road test that was piloted in 30 California Department of Motor Vehicles field offices. The study represents the fourth stage in a four-stage project to develop an improved competency-based drive test for possible statewide implementation. The DPE was found to have construct validity as demonstrated by experienced good drivers having had significantly lower fail rates and mean point scores than did inexperienced drivers and drivers with physical or mental disabilities that affected their driving. The evaluation also found the DPE to be more difficult than the current drive test, with fail rates of 45.6% and 26.2% for the two tests, respectively. The DPE was also found to take 11 minutes longer to administer than did the current drive test. The impact on test validity of several modifications to shorten the DPE test time was also evaluated.

14. SUBJECT TERMS

15. NUMBER OF PAGES

drive test, driver competency, fail rates, disqualifications, test time, modifications
17. SECURITY CLASSIFICATION OF REPORT 18. SECURITY CLASSIFICATION OF THIS PAGE 19. SECURITY CLASSIFICATION OF ABSTRACT

37
16. PRICE CODE

20. LIMITATION OF ABSTRACT

Unclassified
NSN 7540-01-280-5500

Unclassified

Unclassified
Standard Form 298 (2-89)
Prescribed by ANSI Std. Z39-18 298-102

VALIDITY OF CAS DPE TEST

PREFACE

This report presents findings of a validity evaluation of a prototype drive test that was piloted in 30 California Department of Motor Vehicles field offices. The study represents the fourth stage in a four-stage project to develop an improved competencybased drive test for possible statewide implementation. The present report is being issued as an internal monograph of the Department of Motor Vehicles' Research and Development Section rather than an official report of the State of California. The findings and options may therefore not represent the views and policies of the State of California.

ACKNOWLEDGMENTS

This report presents results of a validity evaluation of a prototype drive test that was piloted in 30 California Department of Motor Vehicles field offices. The study was conducted under the general direction of Raymond C. Peck, Research Chief. Mark Harling, Manager III of Field Operations Division's Program Support Staff, acted as a liaison between Research and Development and the study field offices. Special thanks go to Eric Berube, Research Analyst II, for doing the analysis of covariance (ANCOVA) and to Debbie McKenzie, Staff Services Analyst, for producing the tables and graphs, monitoring data collection, and coding data. Thanks are also given to David Hironaka, Information Systems Technician of Headquarters Operations' Data Entry, for writing the data entry procedures. The authors also wish to thank all of the Licensing Registration Examiners Supervising Motor Vehicle Representatives, Driver License Supervisors, and Field Office Managers in the 30 Driving Performance Evaluation offices for their efforts in the data collection during the pilot, especially staff in the Fullerton, West Covina, and Westminster field offices. Thanks are also given to LeeAnn Richardson, Hearing Officer in the Driver Safety Review Unit, for locating and notifying three-accident reexamination subjects for the validity study.

EXECUTIVE SUMMARY

Introduction In 1990 the California Department of Motor Vehicles (DMV) established a program to increase the level of competency of California drivers. One of the components of
i

VALIDITY OF CAS DPE TEST

this program is the development and implementation of an improved drive test, the Driving Performance Evaluation (DPE). The purpose of this study was to evaluate the validity of Stage 4 of the DPE and its effect on drive test fail rate and average administration time. The DPE differs from the current drive test in several important ways. For example, it is around 10 minutes longer, includes an off road skill test and freeway driving, and uses more objective or standardized criteria in which defined maneuvers are observed and scored only at preplanned locations and times during the test.

Methods Six groups of subjectsnovice originals, out-of-state originals, congratulations letter renewals, written-test renewals, three-accident reexamination referrals, and limited-term renewalswere administered the DPE by two Licensing Registration Examiners (LREs) in each of three DMV field offices from September 1994 through January 1995. Additional data on DPE performance and process measures were collected 1 week each month during the same time period for all applicants appearing for a drive test in one of the 30 DPE offices. The validity of the DPE was assessed by comparing the six groups on fail rate and mean point score, by correlating group status with DPE pass/fail result and point score, and by comparing the DPE performance of accident-free and accidentinvolved drivers. Several possible changes to the DPE were evaluated to determine their effects on validity as measured by correlation of group status with DPE score. DPE fail rate and average test time were compared with the fail rate and average test time for the current drive test in the 30 DPE offices.

Results The novice original and limited-term renewal groups had significantly higher fail rates than did the out-of-state original, congratulations renewal, and written-test renewal groups, and the out-of-state original group had a significantly higher fail rate than did the congratulations renewal group. The novice original, three-accident reexamination, and limited-term renewal groups had significantly higher mean point scores than did the out-of-state original, congratulations renewal, and written-test renewal groups.

ii

VALIDITY OF CAS DPE TEST

Validity was also found at the individual-examiner level; all LREs rated the DPE performance of the novice original group as being significantly worse than that of the congratulations and written-test renewal groups. Group status (novice original vs. congratulations renewal) was significantly correlated with DPE pass/fail result and point score. Although the difference was not statistically significant, accident-involved subjects had a higher DPE fail rate than did accident-free subjects. Fail rate increased by 19.4 percentage points, from 26.2% for the current drive test to 45.6% for the DPE. Average test time increased by 11.2 minutes, from 13.6 minutes for the current drive test to 24.8 minutes for the DPE. All simulated modifications to the DPE reduced validity, except eliminating street parking which slightly increased validity. Eliminating freeway driving reduced validity the most, causing the correlation of point score with group status to decrease from -.30 to -.25.

Conclusions The overall results of the analyses provide strong evidence that the DPE is a valid test. The DPE is more difficult than the current drive test, substantially increasing test fail rate. The DPE takes more than 10 minutes longer to administer than does the current drive test. If the DPE must be modified to reduce test length, freeway driving should not be eliminated.

iii

VALIDITY OF CAS DPE TEST

TABLE OF CONTENTS PAGE PREFACE............................................................................................................................... ACKNOWLEDGMENTS..................................................................................................... EXECUTIVE SUMMARY..................................................................................................... Introduction...................................................................................................................... Methods............................................................................................................................. Results................................................................................................................................ Conclusions....................................................................................................................... INTRODUCTION................................................................................................................. METHODS............................................................................................................................. Subjects.............................................................................................................................. Data Collection................................................................................................................. Data Analysis.................................................................................................................... DPE validity.................................................................................................................. Evaluation of DPE modifications .............................................................................. RESULTS................................................................................................................................ DPE validity ...................................................................................................................... Construct validity........................................................................................................ Ultimate criterion validity.......................................................................................... General Process Measures.............................................................................................. DPE difficulty ............................................................................................................... DPE test time................................................................................................................ Evaluation of DPE Modifications................................................................................... DISCUSSION AND RECOMMENDATIONS ................................................................... REFERENCES ....................................................................................................................... APPENDICES NUMBER A B C Driving Performance Evaluation Score Sheet .................................................... Daily DPE Activity Form ....................................................................................... 24 25 i i i i ii ii iii 1 1 1 3 4 4 5 5 5 5 13 14 14 18 19 21 23

Table C-1. Correlation of Item Score and Group Status (Novice Original vs. Congratulations Renewal) for First-Attempt Subjects Completing the DPE (DQs Excluded) in Fullerton, West Covina, and Westminster (N = 1,563).................................................................. 26

iv

VALIDITY OF CAS DPE TEST

TABLE OF CONTENTS (Continued) APPENDICES (Continued) NUMBER PAGE Table C-2. Number of Subjects (n) and Fail Rate for the DPE and Current Drive Tests by Field Office ..................................................................... 29 Table C-3. Number of Subjects (n) and Turn-and-Stop DQ Rate by Field Office.......................................................................................................... Table C-4. Number of Subjects (n) and Mean Test Time for the DPE and Current Drive Tests by Field Office..................................................... LIST OF TABLES

30

31

1 2 3 4 5 6 7 8 9

Number of Subjects in Each Office by First-Attempt Applicant Group...................................................................................................... Number of Subjects (n), Fail Rate, and Mean Score by First-Attempt Applicant Group ............................................................................ Fail Rate (%) and Number of Subjects (in Parentheses) by Office, LRE, and First-Attempt Applicant Group............................................... Fail Rate ANOVA Results...................................................................................... Mean Score and Number of Subjects (in Parentheses) by Office, LRE, and First-Attempt Applicant Group (DQs Excluded)............................... Mean Score ANOVA Results................................................................................. Number of Subjects (n) and Correlation (r) of Pass/Fail Result and Total Point Score with Applicant Group by LRE........................................ Number of Subjects (n), Fail Rate, and Mean Score by First-Attempt Applicant Group (30 Offices)........................................................ Number of Subjects (n), Mean Age, Percentage of Women, and Observed and Statistically-Adjusted DPE Fail Rates for Accident-Free and Accident-Involved Renewal and 3-Accident Reexamination Subjects .........................................................................................

6 6 8 9 10 10 12 13

13

VALIDITY OF CAS DPE TEST

TABLE OF CONTENTS (Continued) LIST OF TABLES (Continued) NUMBER PAGE 10 Number (n) and Percentage of Applicants with Usable Data Reported on the Daily DPE Activity Form by DPE Test Status (30 Offices)................................................................................................................ 14 11 12 13 14 DPE Results for All Applicants (30 Offices) ......................................................... DPE Results for First-Attempt Novice Originals (30 Offices)........................... Fail Rate (%) and Number of Subjects (in Parentheses) by Group and Attempt Number for All Applicants Tested (30 Offices)........................... Total Fails, On-Road DQs, Turn-and-Stop DQs, and Point-Score Fails as Percentages of all Novice Original First-Attempt Applicants by Month for September 1994 through February 1995.................................... Number of Subjects (n) and Correlation of Total Point Score with Group Status by LRE for Various DPE Modifications (DQs Excluded)........... LIST OF FIGURES 1 2 3 4 5 DPE fail rate by first-attempt applicant group ................................................... DPE mean score by first-attempt applicant group (DQs excluded) ................ DPE fail rate by office, LRE, and first-attempt applicant group....................... DPE mean score by office, LRE, and applicant group (DQs excluded)........... Total fails, on-road DQs, turn-and-stop DQs, and point-score fails as percentage of all novice original first-attempt applicants by month for September 1994 through February 1995. ........................................ DPE mean test time by month for September 1994 through February 1995.......................................................................................................... 7 7 9 11 15 16 17

17 20

15

18 19

vi

VALIDITY OF CAS DPE TEST

INTRODUCTION This study evaluated the validity of the Driving Performance Evaluation (DPE) drive test piloted in 30 California Department of Motor Vehicles (DMV) field offices in Los Angeles, Orange, and San Diego counties. The evaluation is the final step (Stage 4) in a four-stage project for developing and evaluating a new drive test for possible statewide implementation in California. The Stage 1 study (Shumaker, 1994) evaluated the reliability of the department's current drive test in six field offices that were selected in a pre-Stage 1 study (Williams & Shumaker, 1994) to be a representation of field offices statewide. A prototype of the DPE was piloted in Bellflower, Laguna Hills, Sacramento, and South Sacramento in Stage 2. The Stage 3 pilot (Hagge, 1994) evaluated the reliability of the DPE in the six field offices used for the Stage 1 study and found it to be much more reliable than the current drive test. It also provided information for further improving the DPE prior to the pilot for the current study. The DPE is based on the driver performance model applied to commercial road tests described in a report by Mackie et al. (1989). A brief review of the relevant drive test literature is included in the Stage 1 report. METHODS This study assessed the validity of the DPE in three field offices, and provides general descriptive statistics on the DPE testing process in the 30 DPE pilot offices. Validity is defined here as the ability of the test to accurately measure driving competency. The higher the validity of the test, the greater the likelihood that the correct licensing decision will be made based on DPE performance. The selection of subjects and the data collection and analysis procedures are described below. Subjects Subjects in the validity study were drawn from applicants administered the DPE in the Fullerton, West Covina, and Westminster field offices. These offices were also involved, along with three other offices, in the Stage 1 and Stage 3 evaluations mentioned above. The subject pool consisted of: (1) novice original driver license applicants taking a California drive test for the first time; (2) original license applicants previously licensed in another state and taking a California drive test for the first time; (3) reentrant license renewal applicants who received a good-driver "congratulations" letter and, therefore, were not required to take an 18-item written test; (4) renewal license applicants who were required to take an 18-item written test; (5) licensed drivers involved in three or more traffic accidents in 1 year;

VALIDITY OF CAS DPE TEST

(6) limited-term license renewal applicants taking a drive test for the first time on the current license renewal application. Only drivers taking the DPE for the first time were selected as subjects. Subjects who failed the DPE were not included in the study again upon taking a DPE retest. Participation in the study was represented as mandatory and applicants were not told they were subjects in a study. If subjects asked whether the results of the test would count against them, they were told that their performance on the DPE would not be used to make a licensing decision. Subjects who performed poorly on the DPE may have been asked to take additional tests of driver competency that could be used as a basis for a licensing action. Applicants who were selected as subjects, but refused to take the DPE, were to be sent to a Driver License (DL) Supervisor for further discussion. Subjects who strongly objected to the drive test were to be informed by the DL Supervisor that DMV has legal authority to give a drive test to a license applicant at any time. Subjects who still refused to cooperate were to be referred to the Field Office Manager for additional discussion and possible release from the study. Renewal subjects who were willing to be tested, but indicated that they were unable to take the drive test at that time, were to be rescheduled for a DPE to be administered within 2 weeks, and were not to be allowed to complete the licensing transaction until they returned to take the DPE in the same office. Novice originals who indicated that they were not prepared to drive on the freeway were not used as subjects on that day. These applicants were asked to practice freeway driving and to later reschedule the DPE when they felt they were ready. Three-accident reexamination subjectsthose with three or more accidents in a yearwere identified by Driver Safety Review Unit and instructed to phone a designated contact person for a DPE appointment in one of the three validation study offices. The contact person was to schedule the DPE early enough for the subject to be given the drive test within 2 weeks. Because the number of limited-term renewals and three-accident subjects was expected to be small, these applicants were to be tested only by the study Licensing Registration Examiners (LREs) and given the highest priority in testing throughout the study. When none of these subjects were available, each study LRE was to try to alternate testing between the other four groups in an attempt to roughly balance the number of subjects in each group. In an effort to minimize subject-selection bias, each study LRE was to choose the next person in line as the next subject to be tested in the specified group. Subjects selected for the development of general DPE performance and process measures consisted of all applicants who appeared for a drive test in one of the 30 DPE offices during the general data collection periods specified below.

VALIDITY OF CAS DPE TEST

Data Collection All LREs in the three validation study offices administered the DPE, but only two designated study LREs in each office administered DPEs for the DPE validity assessment. The validation study used the same LREs, DL Supervisors, and Field Office Managers who participated in the Stage 3 DPE study, except for two LRE replacements. Data were collected from September 19, 1994 through January 27, 1995 for all subjects except the three-accident reexamination group. Data collection was extended through February for the latter group in order to obtain a larger sample size for that group. In addition to the information normally required on the DPE score sheet (see Appendix A), each study LRE was to record on the sheet the subject's study group, the DPE start and end times, and, if applicable, the reason why the subject was not scored on the DPE freeway maneuvers (e.g., freeway closure or subject's refusal). Each study LRE also recorded data on a separate Daily DPE Activity form (see Appendix B) for each subject they tested that day. The information collected included test start and end times, applicant status, drive test attempt number, whether the applicant was tested on the freeway, whether the applicant was restricted to no freeway, and drive test result. The form also provided space for the validation study LREs to explain why an applicant selected as a subject may not have been tested on the DPE (e.g., they refused to participate or did not have auto insurance). At the end of each day, the DL Supervisor was to collect and send to the Research and Development Section (R&D) all DPE score sheets and Daily DPE Activity forms completed by the study LREs. Data for the descriptive analysis were collected from all 30 DPE offices for 1 week each month during September 1994 through June 1995. The data were collected using the Daily DPE Activity form mentioned above. The form was to be completed by each LRE. Each DL window technician was also to complete the form for applicants who appeared for a drive test but for some reason (e.g., no auto insurance) were not referred to an LRE to take the DPE. Data were not collected for phone-in cancellations and no-shows. The completed Daily DPE Activity forms were submitted to R&D at the end of each week of data collection. Score sheets for the department's current drive test (DL 179) were also collected from the 30 offices. LREs were instructed to record test start and end times on all score sheets completed in May 1994, just before the DPE was implemented. A random sample of score sheets was selected for each office to reduce the number of documents needing to be key entered. Sampling ratios were used that were expected to yield within-office sample sizes of 75 or larger. Average drive test time and fail rate for the current drive test, normalized to reflect the same proportional representation of DPE score sheets collected from each office, served as baseline measures for assessing the operational impact of the DPE. All DPE score sheets and Daily DPE Activity forms were screened by R&D and key entered by Data Entry.

VALIDITY OF CAS DPE TEST

Data Analysis DPE validity. The validity of the DPE was evaluated by comparing the six applicant groups on average DPE fail rate and mean DPE point score. If performance on the DPE were a valid measure of driving competency, experienced drivers (e.g., renewals and California originals previously licensed in another state) would be expected to perform better than novice drivers and limited-term renewals drivers with worsening physical or mental conditions that affect their driving. The groups were also compared within each office and LRE to determine whether the validation results varied as a function of who did the scoring. The analysis of variance (ANOVA) statistical procedure was used to test whether group differences on the DPE performance measures were statistically significant (i.e., not likely to have occurred by chance). Two additional DPE validity measures were calculated by correlating subject status (novice original vs. congratulations renewal) with DPE pass/fail result and DPE point score. Item validity measures were also computed by correlating subject status with item scores for subjects who completed the DPE. Subjects who were automatically disqualified (DQued) due to making a serious error that caused the examiner to terminate the test were excluded from the computation. Correlation as measured here refers to the strength of the relationship between subject status and DPE performance. The correlation coefficient theoretically can range from -1 to 1. A large positive or negative coefficient indicates the two variables are strongly related, whereas a / 0 coefficient indicates the two variables are completely unrelated. A positive value means that the variables tend to increase or decrease together, and a negative value means that one variable tends to increase as the other decreases, or visa versa. "Ultimate" criterion validitythe relationship between performance on the DPE and accident riskwas assessed using a contrasted-groups design. Driver records for the 3-year period prior to testing were obtained from the department's automated DL masterfile for renewal subjects (groups 3 and 4) and three-accident reexamination subjects (group 5). These subjects were divided into two groups: accident-free drivers and drivers with two or more accidents. If the DPE has criterion-related validity, the accident-free group would be expected to have a significantly lower fail rate than would the accident-involved group. Analysis of covariance (ANCOVA) was used to test whether the differences in group fail rates were statistically significant. This statistical technique enabled the group fail rates to be statistically adjusted to partial out the effects of other variables (or covariates), in this case age, sex, and LRE. The adjusted means reflect what the relative performance of the two groups would be expected to be had all subjects been of the same age and sex and been tested by the same LRE. Because of the small number of accident-repeaters and other limitations in the use of accidents as a criterion measure, the results of the analysis were not expected to be definitive. A future assessment of the DPE's criterion-related validity will also be made by comparing the driver records subsequent to drive testing for original applicants given the DPE with those for a sample of original applicants given the current drive test in matched northern California offices. The former group will be identified from the Daily
4

VALIDITY OF CAS DPE TEST

DPE Activity forms collected from the 30 DPE offices. Drivers in the latter group will be selected either from the DL masterfile or from score sheets for the current drive test collected from northern offices before Stage 1. The results of this analysis will be presented in a follow-up report to be completed by September 1998. Evaluation of DPE modifications. Several possible changes to the DPE were evaluated to determine the effect of each modification on test validity. The validity measure used for this purpose was the correlation of groups status (novice original vs. congratulations renewal) with DPE total score for subjects who completed the DPE (DQs excluded). Total test score under each DPE-change scenario was simulated by counting only errors made on maneuvers that would remain on the test. It was not possible to compute validity coefficients based on pass/fail results, which would have included all subjects, because data indicating at what point in the DPE a DQ occurred were not available. The following changes to the DPE were assessed: (1) (2) (3) (4) (5) (6) (7) eliminate the turn-and-stop skill test; eliminate street parking; eliminate two intersections (#7 and #8 on score sheet); eliminate one left turn (#4) and one right turn (#4); eliminate two intersections (#7 and #8), one left turn (#4), and one right turn (#4); eliminate straight business and residential items; eliminate freeway driving.

The effect of changing the DPE on total fail rate could be estimated only for the turnand-stop and freeway options due to the inability to match DQs with specific DPE maneuvers on the score sheet. DQ data for the turn-and-stop skill test were available from the DPE score sheets and the Daily DPE Activity forms. The occurrence of DQs on the freeway were determined from LRE comments made on DPE score sheets collected for a future R&D study over 2 weeks during October 1994 in all 30 DPE offices. RESULTS DPE Validity Construct validity. Table 1 presents the number of subjects tested in each validation study office by applicant group. (The counts do not reflect the proportional representation of each group that would be expected for normal operationsi.e., they reflect the LREs selection of subjects rather than actual workload volumes.) As expected, the number of subjects in the three-accident reexamination and limited-term renewal groups was small relative to the number of subjects in the other groups, especially in West Covina where only one three-accident reexamination subject was tested. Novice originals far outnumbered other subjects, comprising over 2/3 of the total subject count.

VALIDITY OF CAS DPE TEST

Table 1 Number of Subjects in Each Office by First-Attempt Applicant Group

Group Novice original Out-of-state original Congratulations renewal Written-test renewal 3-accident reexamination Limited-term renewal Total Fullerton 535 68 107 104 16 44 910 West Covina 720 53 129 167 1 17 1,087 Westminster 1,095 85 107 142 12 67 1,508 Total 2,350 206 343 449 29 128 3,505

The number of subjects who were selected for the study but were not tested due to their refusing to participate or wanting to be rescheduled for testing on another day is unknown. The study procedures called for LREs to select study subjects and record information on any refusals or rescheduled subjects on the Daily DPE Activity form. However, post-study comments made by employees in the study offices indicated that this procedure was not followed. It appears that the selection of subjects was made by technicians at the DL window, who then referred willing subjects to the LRE. Therefore, refusals and reschedules could not be accounted for. Table 2 presents the number of subjects, DPE fail rate, and mean DPE point score (number of errors) for each group in the three offices combined. The performance measures are shown graphically in Figures 1 and 2, respectively. Only subjects who completed the DPE (i.e., who were not DQed) were included in the computation of mean DPE score. The 0.2% of subjects who DQed on the pre-drive check list were excluded from both analyses. Table 2 Number of Subjects (n), Fail Rate, and Mean Score by First-Attempt Applicant Group
Group Novice original Out-of-state original Congratulations renewal Written-test renewal 3-accident reexamination Limited-term renewal Total n 2,350 206 343 449 29 128 3,505 Fail rate (%) 52.6 31.6 14.9 24.1 31.0 48.4 43.7 Mean score 9.9 8.3 7.5 8.3 12.1 10.4 9.2

Note. Groups differed significantly on fail rate (F = 59.97, p<.001, two-tailed) and point score (F = 21.35, p<.001, two-tailed).

VALIDITY OF CAS DPE TEST

60 50

FAIL RATE (%)

40 30 20 10 0
Novice original Out-of-state original Congratulations renewal Written-test renewal 3-accident reexamination Limited-term renewal

APPLICANT GROUP

Figure 1. DPE fail rate by first-attempt applicant group.

14 12 10 MEAN SCORE 8 6 4 2 0
Novice original Out-of-state original Congratulations renewal Written-test renewal 3-accident reexamination Limited-term renewal

APPLICANT GROUP

Figure 2. DPE mean score by first-attempt applicant group (DQs excluded).

ANOVA results indicate that the group differences on fail rate were statistically significant (p<.001). Results of Tukey post-hoc significance tests (familywise alpha = .05) indicate that the novice original and limited-term renewal groups had significantly higher fail rates than did the out-of-state original, congratulations renewal, and written-test renewal groups, and that the out-of-state original group had a significantly higher fail rate than did the congratulations renewal group.

VALIDITY OF CAS DPE TEST

Results of the ANOVA conducted on point scores indicate the groups also differed significantly on average score (p<.001). Tukey post-hoc tests found mean scores for the novice original, three-accident reexamination renewal, and limited-term renewal groups to be significantly higher than the mean scores for the out-of-state original, congratulations renewal, and written-test renewal groups. This pattern of differences generally parallels the results from the fail rate analysis reported above. Two supplemental three-way factorial ANOVAs were conducted to determine whether the pattern of group differences on fail rate and mean point score was the same for different study LREs in different offices. The three-accident reexamination group and the limited-term renewal group were excluded from these analyses due to the small number of these subjects tested by each LRE. The factors or independent variables in each ANOVA were applicant group, office, and. (Only the main effect of group, and the interaction effects that included group as a factor, are of interest here because the other effects represented in the factorial designthe main effects of office and LRE within officedo not directly address the question of the DPE's validity. In addition, effects averaged across applicant groups are not meaningful because the proportional representation of each applicant group was different for each office and LRE.) Table 3 presents DPE fail rate and number of subjects by office, LRE, and applicant group. The nested factorial ANOVA results are presented in Table 4. The main effect of group was statistically significant (p<.001), indicating significant differences in group fail rates averaged across offices and LREs (which essentially duplicates results from the one-way ANOVA presented above). The Group x Office interaction was not significant (p = .20). The Group x LRE interaction within office was significant (p<.05), indicating that at least one pair of LREs (within office) differed in their patterns of group fail rates. However, the LREs' general pattern of scoring, as reflected in Figure 3, clearly supports the validity of the DPE. In particular, all LREs indicated that the novice original group performed worse on the DPE than did both renewal groups. Table 3 Fail Rate (%) and Number of Subjects (in Parentheses) by Office, LRE, and First-Attempt Applicant Group
Fullerton Group Novice original Out-of-state original Congratulations renewal Written-test renewal
LRE 1 LRE 2

West Covina
LRE 1 LRE 2

Westminster
LRE 1 LRE 2

69.9 (286) 38.7 (31) 15.6 (64) 39.2 (51)

53.4 (249) 18.9 (37) 23.3 (43) 28.1 (89)

52.6 (352) 35.0 (20) 8.4 (83) 24.1 (87)

71.2 (368) 42.4 (33) 30.4 (46) 27.5 (80)

33.8 (515) 10.0 (10) 8.2 (61) 19.2 (47)

48.5 (580) 32.0 (75) 10.9 (46) 11.6 (95)

VALIDITY OF CAS DPE TEST

Table 4 Fail Rate ANOVA Results

Source of variation Group Office LRE within office Group x Office Group x LRE within office df 3 2 3 6 9 MS 22.72 2.59 1.18 0.30 0.45 F 106.59 12.16 5.54 1.43 2.11 p .000 .000 .001 .200 .025

Error within cells

3324

0.21

100
Fullerton LRE 1 Fullerton LRE 2 West Covina LRE 2 Westminster LRE 2 West Covina LRE 1 Westminster LRE 1

80

FAIL RATE (%)

60

40 20 0
Novice original Out-of-state original Congratulations renewal Written-test renewal

APPLICANT GROUP

Figure 3. DPE fail rate by office, LRE, and first-attempt applicant group.

Table 5 presents mean DPE point score and number of subjects by office, LRE, and applicant group for subjects in each group who completed the DPE (DQs excluded). Table 6 presents the results of the nested factorial ANOVA conducted on point scores. The main effect of group on point score was statistically significant (p<.001). However, the significant Group x LRE interaction effect within office (p<.05) indicates that the pattern of differences between group means underlying the main effect of group does not represent the scoring of every LRE individually. This interaction effect is evidenced
9

VALIDITY OF CAS DPE TEST

in Figure 4, which shows the group mean for each LRE. However, as was the case for fail rates, their general pattern of scoring clearly supports the validity of the DPE. In particular, all LRE's mean scores indicated that the novice original group performed worse on the DPE than did both renewal groups.

Table 5 Mean Score and Number of Subjects (in Parentheses) by Office, LRE, and First-Attempt Applicant Group (DQs Excluded)
Fullerton Group Novice original Out-of-state original Congratulations renewal Written-test renewal
LRE 1 LRE 2

West Covina
LRE 1 LRE 2

Westminster
LRE 1 LRE 2

12.8 (122) 10.9 (22) 11.4 (61) 11.2 (35)

12.3 (154) 10.1 (34) 10.0 (41) 11.0 (81)

9.8 (188) 3.6 (13) 4.5 (77) 5.6 (67)

12.1 (107) 9.3 (21) 7.8 (33) 9.1 (58)

7.6 (349) 6.1 (9) 4.4 (56) 6.1 (38)

9.4 (331) 7.2 (51) 8.6 (42) 7.2 (84)

Table 6 Mean Score ANOVA Results

Source of variation Group Office LRE within office Group x Office Group x LRE within office Error within cells df 3 2 3 6 9 2052 MS 974.56 1666.91 460.77 91.46 31.31 16.58 F 58.79 100.55 27.80 5.52 1.89 p .000 .000 .000 .000 .049

10

VALIDITY OF CAS DPE TEST

16
Fullerton LRE 1 Fullerton LRE 2 West Covina LRE 2 Westminster LRE 2

14 12 10 8 6 4 2
Novice original

West Covina LRE 1 Westminster LRE 1

MEAN SCORE

Out-of-state original

Congratulations renewal

Written-test renewal

APPLICANT GROUP

Figure 4. DPE mean score by office, LRE, and applicant group (DQs excluded).

Table 7 presents the correlations of DPE pass/fail result and DPE point score with applicant status for each validation study LRE. DQs were excluded from the computation involving point scores. Applicant status was coded / for novice originals 0 and 1 for congratulations renewals. Correlations involving pass/fail result ranged from -.17 to -.43, and averaged -.27. All of these coefficients were statistically significant (p<.01). The point score correlations fluctuated more widely, from -.06 to -.52, and averaged -.30. All but two of these correlations were significant (p<.05). These results indicate that novice originals tended to fail the test more often and make more errors than did congratulations renewals, and thus validate the DPE. The correlations of item scores with group status for first-attempt subjects completing the test (DQs excluded) are presented in Table C-1 in the Appendix. A positive correlation indicates that novice originals tended to perform better on the item than did congratulations renewals. Items without a correlation coefficient entered in the table did not have enough variance (i.e., were failed by too few applicants) to enable this measure to be computed.

11

VALIDITY OF CAS DPE TEST

Table 7 Number of Subjects (n) and Correlation (r) of Pass/Fail Result and Total Point Score with Applicant Group by LRE
Office LRE Fullerton LRE 1 LRE 2 West Covina LRE 1 LRE 2 Westminster LRE 1 LRE 2 Total (avg.) Pass/fail n 350 292 435 414 576 626 2,693 r -.43** -.21** -.35** -.27** -.17** -.20** -.27** n 183 195 265 142 405 373 1,563 Point score r -.13 -.18* -.51** -.52** -.37** -.06 -.30**

Note. Total (average) correlations are not weighted to reflect differences in within-LRE sample sizes. *p<.05. **p<.01.

Additional support for the validity of the DPE is provided by data reported on the Daily DPE Activity forms collected from the 30 DPE offices during September 1994 through February 1995, excluding data collected from the six study LREs. (In September and October only 29 offices reported, in November 27 offices reported, in December and January all offices reported, and in February 28 offices reported.) Table 8 presents the number of subjects, fail rate, and mean DPE score (DQs excluded) for each first-attempt applicant group identified on the forms. The physical/mental group consisted of applicants who were identified as having a driving-related physical or mental condition that warranted their being given a regular (rather than special) drive test. Drivers in the written test failure group were renewals who had to take a drive test because they failed the written test three times. ANOVA results indicate the differences in fail rates to be statistically significant (p<.001). Tukey post-hoc tests found that the renewal group, which consisted mostly of California drivers with expired licenses, had a significantly lower fail rate than did all the other groups. The post-hoc tests also found that the out-of-state original group had a significantly lower fail rate than did the novice original or limited-term renewal groups, and that the novice original and written test failure renewal groups had significantly lower fail rates than did the limited-term renewal group.

12

VALIDITY OF CAS DPE TEST

Table 8 Number of Applicants (n), Fail Rate, and Mean Score by First-Attempt Applicant Group (30 Offices)
Group Novice original Out-of-state original Limited-term renewal Physical/mental renewal Written-test failure renewal License-expired renewal n 18,942 900 717 151 352 169 Fail rate (%) 45.9 36.2 57.9 49.0 41.5 23.1 Mean score 10.8 9.1 12.1 12.4 11.6 9.2

Note. Groups differed significantly on fail rate (F = 22.88, p<.001, two-tailed) and point score (F = 24.06, p<.001, two-tailed).

The groups also had significantly different mean scores (p<.001). Tukey post-hoc tests found that the renewal and out-of-state original groups had significantly lower mean scores than did all other groups, and that the novice original group had a significantly lower mean score than did the limited-term and physical/mental renewal groups (p<.05). Ultimate criterion validity. Table 9 presents demographic information and the observed and statistically-adjusted DPE fail rates for accident-free and accident-involved renewal and three-accident reexamination subjects. The two groups were not significantly different on mean age (p = .08), proportion of women (p = .78), or fail rate (p = .31). The ANCOVA results showed age and LRE to be significant covariates (p<.01), but not sex (p = .12). Homogeneity of slopes tests showed that the contrasted groups did not have different slopes on any of the covariates, therefore a common-slopes ANCOVA model was selected. The adjusted DPE fail rates were farther apart in the direction of validating the DPE than were the unadjusted means, but the difference was still nonsignificant (p = .17). Although the results fail to substantiate the validity of the test in discriminating accident-repeaters from accident-free applicants, the very small sample size for the accident group greatly limited the statistical power for detecting a significant effect. The size and direction of the differences are encouraging. Table 9 Number of Subjects (n), Mean Age, Percentage of Women, and Observed and Statistically-Adjusted DPE Fail Rates for Accident-Free and Accident-Involved Renewal and 3-Accident Reexamination Subjects
Group Accident-free Accident-involved n 654 42 Mean age 46.6 41.9 % women 45.1 42.9 Unadjusted fail rate (%) 19.7 26.2 Adjusted fail rate (%) 19.6 28.2

Note. The accident-involved group consisted of drivers with two or more accidents during the 3 years prior to DPE testing. Covariates used to adjust DPE fail rate were age, sex, and LRE. None of the differences were statistically significant (p>.05, twotailed).

13

VALIDITY OF CAS DPE TEST

General Process Measures The Daily DPE Activity forms collected from the 30 offices contained data on 36,812 applicants scheduled for a drive test, excluding those tested by the six study LREs. Data were not usable for 2,165 cases because the information was not sufficient to determine whether the DPE was administered. Of the remaining 34,647 applicants for which data were usable, 3,175 (9.2%) had the DPE postponed or rescheduled. Table 10 lists the reported reasons for not giving the DPE on the scheduled day and the number and percentage of subjects associated with each one. Nearly two-thirds of the cases were due to the applicant's not having auto insurance or to vehicle mechanical failure. The table also indicates that, of the 31,472 applicants who did take the test, 208 (less than 1%) were not tested on the freeway. Only six of the freeway-waiver cases were due to inability to test on the freeway due to a traffic accident, congestion, or freeway closure; the remainder were due to the applicant's stating that they never drive on the freeway and consequently may have been given a no-freeway license restriction. (An additional 105 applicants given the DPE may not have been tested on the freeway, but this could not be determined with certainty because of ambiguity of data reported for these cases.) Table 10 Number (n) and Percentage of Applicants with Usable Data Reported on the Daily DPE Activity Form by DPE Test Status (30 Offices) Test status DPE postponed/rescheduled no insurance vehicle mechanical failure not ready for freeway no driver training no registration no accompanying driver other DPE administered not tested on freeway no-freeway restriction accident/congestion/closure n 3,175 1,059 884 251 119 102 54 139 31,472 208 202 6 % of total (N = 34,647) 9.2 3.1 2.6 0.7 0.3 0.3 0.2 0.4 90.8 0.6 0.6 0.0

Note. Data were provided for a total of 36,812 driver license applicants, excluding those tested by the six study LREs. However, data were usable for only 34,647 of these cases because information provided for the other 2,165 did not indicate whether the DPE was administered.

DPE difficulty. Table 11 presents the DPE results for all 31,159 applicants tested in the 30 DPE offices during the general data collection periods from September 1994 through February 1995 (excluding applicants tested by the six validation study LREs). Data were collected from September 1994 through June 1995, however only data for September

14

VALIDITY OF CAS DPE TEST

1994 through February 1995 were used to compute the descriptive measures due to key entry delays and time constraints on data analysis. The overall fail rate was 45.5%. The vast majority of the fails were due to DQs, which accounted for 38.8 percentage points of the total fail rate. A total of 11.6% of all applicants were DQed on the turn-and-stop skill test and, consequently, were not tested on the road. Only 6.7% of all applicants failed because of high point (error) scores. Table 11 DPE Results for All Applicants (30 Offices) DPE result Total fails Point-score fails DQ fails DQ pre-drive DQ turn-and-stop DQ on road DQ unidentified Number of applicants 14,182 2,085 12,097 70 3,609 8,118 300 % of total (N = 31,159) 45.5 6.7 38.8 0.2 11.6 26.1 1.0

Note. Applicants tested by the six designated LREs involved in the validation study are not represented.

The percentage of applicants who failed the DPE in each of the 30 offices is presented in Table C-2 in the Appendix. The table also shows the fail rate in each office for the current drive test. The latter estimates are based on test results recorded on the current drive test score sheets that were collected in May 1994 (just before DPE implementation) to capture baseline test time data. Within-office fail rate ranged from 32.0% to 53.3% for the DPE and from 17.7% to 36.3% for the current drive test. To compare the fail rates for the DPE and current drive test across offices, a normalized fail rate for the current test was computed based on the within-office frequencies for the DPE. The normalized fail rate is the average fail rate that would be expected if the same number of current drive tests as the number of DPEs were administered in each office. The San Clemente office was excluded from the comparison because it did not submit current drive test score sheets to R&D. DPE fail rate for the 29 reporting offices was 45.6%. The current drive test's normalized fail rate was 26.2%. These results represent a fail rate increase of 19.4 percentage points following implementation of the DPE. The percentage of total applicants who were DQed on the turn-and-stop skill test in each of the 30 offices is presented in Appendix Table C-3. The estimates ranged from 3.5% to 18.3%. (Laguna Hills [0.9%] substituted a 3-point turn for the turn-and-stop skill test.)

15

VALIDITY OF CAS DPE TEST

Table 12 presents DPE results for novice original first-attempt applicants only. These figures better reflect the inherent difficulty of the DPE because they reflect how well inexperienced drivers performed on their first try. The results are very similar to those presented in Table 11. The total fail rate was 45.8%, with 38.9 of the percentage points being due to DQs. The rate of DQs on the turn-and-stop skill test was 13.2%. Table 12 DPE Results for First-Attempt Novice Originals (30 Offices) DPE result Total fails Point-score fails DQ fails DQ pre-drive DQ turn-and-stop DQ on road DQ unidentified Number of applicants 8,481 1,278 7,203 35 2,451 4,561 156 % of total (N = 18,522) 45.8 6.9 38.9 0.2 13.2 24.6 0.8

Note. Applicants tested by the six designated LREs involved in the validation study are not represented.

Table 13 presents the fail rate and number of subjects by applicant group and test attempt number for the 30 offices (excluding applicants whose applicant group or attempt number could not be identified). Novice originals had about the same fail rate for all drive test attempts. However, the fail rates for other groups tended to increase on successive attempts. (The third-attempt fail rates for physical/mental and renewal applicants should be interpreted with caution because of the small sample sizes for these groups.) It should be noted that some of the second and third drive test attempts may actually have been the first attempt on the DPE. This should not have substantially biased the results because the DPE had been implemented for several weeks before the data collection started and, consequently, the number of drive tests attempts that were not DPEs was expected to be small. To determine the extent to which the results may have been biased by the inclusion of nonDPE drive tests, the fail rates for the three test attempts made by novice originals were calculated for February 1995 only, more than 6 months after DPE testing had begun. The results were nearly identical to those for all months combined.

16

VALIDITY OF CAS DPE TEST

Table 13 Fail Rate (%) and Number of Applicants (in Parentheses) by Group and Attempt Number (30 Offices)
Applicant group Novice original Out-of-state original Limited-term renewal Physical/mental renewal Written test failure renewal License-expired renewal 1st 45.9 (18,724) 36.2 (900) 57.9 (717) 49.0 (151) 41.5 (352) 23.1 (169) Test attempt 2nd 44.8 (6,224) 41.9 (301 58.8 (369) 48.1 (54) 45.6 (114) 44.4 (27) 3rd 46.3 (1,909) 50.0 (90) 64.1 (142) 56.3 (16) 46.9 (49) 54.5 (11)

Table 14 and Figure 5 present the DPE results for novice original first-attempt applicants by month. The general trend appears to be a decrease in total fail rate over time. This decrease parallels the decrease that occurred in the turn-and-stop DQ rates. On-road DQ rate appears to have remained reasonably steady.

Table 14 Total Fails, On-Road DQs, Turn-and-Stop DQs, and Point-Score Fails as Percentages of all Novice Original First-Attempt Applicants by Month for September 1994 through February 1995
Month September October November December January February Total fail rate 49.1 46.2 44.6 45.5 44.7 44.4 On-road DQ rate 25.4 24.6 23.7 24.6 23.9 25.2 Turn-and-stop DQ rate 15.3 14.0 12.4 13.3 13.4 11.4 Point-score fail rate 7.6 6.3 7.7 6.1 6.5 7.0

17

VALIDITY OF CAS DPE TEST

70 % OF TOTAL APPLICANTS
Total fail rate Turn-and-stop DQ rate Point-score fail rate

60 50 40 30 20 10 0 September October

On-road DQ rate

November

December

January

February

MONTH

Figure 5. Total fails, on-road DQs, turn-and-stop DQs, and point-score fails as percentage of all novice original first-attempt applicants by month for September 1994 through February 1995.

DPE test time. The average reported test time was 24.7 minutes for all DPEs given in the 30 offices (excluding DPEs administered by the six LREs in the validation study). The average time reported for completed DPEs (DQs excluded) was 28.4 minutes. As would be expected, the average test time for completed DPEs was greater than that for DPEs with DQ rates excluded because DQs frequently occur early in the test. Table C-4 in the Appendix presents the mean reported test times for the DPE and the department's current drive test for each of the 30 DPE field offices. Within-office mean test time ranged from 20.5 to 28.6 minutes for the DPE and from 10.6 to 18.1 minutes for the current drive test. To compare the mean test time for the DPE and current drive test across offices, a normalized mean time for the current test was computed based on the within-office test frequencies for the DPE. The San Clemente field office was excluded from the analysis because it did not report any data on current drive test time. The mean DPE test time for the 29 reporting offices was 24.8 minutes. The normalized mean time computed for the current drive test was 13.6 minutes. These results represent an average increase of 11.2 minutes per drive test. Figure 6 shows reported mean DPE test time by month. Mean test time increased by almost 1 minute over the 6 months. The reduction in turn-and-stop DQ rate over the

18

VALIDITY OF CAS DPE TEST

same time period (See Figure 5) may have contributed to this reduction in total test time, since the former trend would have been associated with an increase in the percentage of applicants tested on the road.

26.0

MEAN TEST TIME (minutes)

25.5

25.3

25.0 24.6 24.5 24.4 24.2 24.0 September October November

24.8

24.8

December

January

February

MONTH

Figure 6. DPE mean test time by month for September 1994 through February 1995.

Evaluation of DPE Modifications Table 15 presents the correlation of group status with DPE point score for each DPE-modification option (including making no changes) by LRE. Eliminating freeway driving from the DPE had the largest impact on validity, significantly reducing the average coefficient for all LREs from -.30 to -.25 (p<.001). The loss in validity caused by dropping the freeway maneuvers tended to occur regardless of which LRE did the testing. The effect on the validity of the other DPE modifications was much smaller but still statistically significant (p<.05). The one exception was the removing of two intersections, which did not have a significant effect on validity (p>.10). One of the optionseliminating street parkingslightly increased the average validity correlation coefficient, from -.30 to -.32. Had DQs been included in the simulations, the decrease in validity produced by each change option would have been greater.

19

VALIDITY OF CAS DPE TEST

Table 15 Number of Subjects (n) and Correlation of Total Point Score with Group Status by LRE for Various DPE Modifications (DQs Excluded) DPE modification (what to eliminate)
Office LRE Fullerton LRE 1 LRE 2 West Covina LRE 1 LRE 2 Westminster LRE 1 LRE 2 Total (avg.) 405 373 1,563 -.37** -.06 -.30** -.37** -.04 -.28** -.39** -.07 -.32** -.36** -.05 -.29** -.36** -.07 -.28** -.36** -.07 -.28** -.34** -.05 -.28** -.32** -.02 -.25** 265 142 -.51** -.52** -.51** -.51** -.51** -.54** -.48** -.54** -.48** -.48** -.46** -.50** -.51** -.52** -.48** -.45** No change -.13 -.18* Turnandstop -.08 -.18* Street park -.19** -.21** 2 intersections -.14 -.16* 1 L turn 1 R turn 1 L turn 2 inter1 R turn sections -.11 -.20** -.12 -.17* Bus./ resid. -.12 -.17* Freeway -.09 -.14

n 183 195

Note. Total (average) correlations are not weighted to reflect differences in the number of subjects tested by each LRE. *p<.05. **p<.01.

Eliminating the turn-and-stop skill test would be expected to reduce the DPE fail rate by a maximum of 12 percentage points, which represents the percentage of all applicants tested in the 30 offices who were identified on the Daily DPE Activity forms as having been DQed on one of the turn-and-stop maneuvers. The reduction in fail rate that would actually result from this DPE modification, however, would likely be much smaller than this because applicants who were DQed on the turn-and-stop would probably have made a DQ error somewhere else on the test, or failed based on point score, if they had been permitted to continue testing on the road. Eliminating freeway driving from the DPE would be expected to reduce fail rate by at least five percentage points. This represents the percentage of applicants who were identified from LRE comments recorded on the score sheet as having been DQed due to an error made on the freeway. (This is considered a lower bound estimate because some freeway DQs were probably not explicitly identified as such on the DPE score sheet.)

20

VALIDITY OF CAS DPE TEST

DISCUSSION AND RECOMMENDATIONS

The results provide strong evidence that the DPE is a valid test. Inexperienced drivers and drivers with mental or physical conditions that affect their driving tended to perform worse on the DPE than did experienced drivers with no known driving-related debility. The results show that the novice original and limited-term renewal groups had significantly higher fail rates than did the out-of-state original, congratulations renewal, and written-test renewal groups. Comparisons of the point scores (number of errors) yielded similar results. Novice original, limited-term renewal, and three-accident reexamination driver groups had significantly higher mean scores than did the out-of-state original, congratulations renewal, and written-test renewal groups. In addition, the finding that all LREs consistently scored novice originals as being worse drivers than both renewal groups evidences the DPE's validity even at the individual examiner level. The correlational analyses also confirmed the validity of the DPE. The significant correlation of applicant status (novice original vs. congratulations renewal) with pass/fail result and with point score indicate that novice originals failed the test more often and made more errors than did congratulations renewals. Although there was a wide range in the within-LRE correlation coefficients, the LREs were consistent in scoring the congratulations renewal group better than the novice original group. Analyses of data collected from the 30 DPE offices yielded further evidence of the validity of the DPE. The results show that license-expired renewals had significantly lower fail rates than did all other applicant groups, out-of-state originals had a significantly lower fail rate than did novice originals and limited-term renewals, and novice originals and written test failure renewals had lower fail rates than did limited-term renewals. The same pattern of ranking was found for mean scores, but with fewer distinctions: The license-expired renewal and out-of-state groups had significantly lower mean scores than did all other groups, and the novice original group had a significantly lower mean score than did the limited-term renewal and physical/mental renewal groups. It should be noted that the validation results may be biased due to LRE's having known the status of subjects prior to testing, which may have influenced their scoring. Any such criterion contamination would be expected to make the DPE look more valid than it actually is. The ultimate criterion analysis did not find a significant difference in either the unadjusted or adjusted fail rates for accident-free and accident-involved renewals and 3accident reexamination subjects. However, the direction of the difference in group means was consistent with the DPE being a valid test. The failure to detect a significant difference was not surprising considering that statistical power (the probability of finding a significant effect) was only 17% due to the small sample size in the accident21

VALIDITY OF CAS DPE TEST

involved group. In addition, accident involvement per se is not a good indicator of level of driving ability, which is what the DPE is intended to measure. The likelihood of having an accident not only depends upon level of driving competency, but also exposure variables such as the amount and conditions of driving. Had the accident measure used to establish the contrasted groups been adjusted for these exposure variables, the probability of finding a significant difference between the contrasted groups would have been increased. Information collected from the 30 offices revealed the operational impact of the DPE. Implementation of the DPE substantially increased both the length and the difficulty of the drive test. Fail rate increased from 26.2% for the current drive test to 45.6% for the DPE. This 19.4 percentage-point gain in fail rate represents over a 70% increase in the total proportion of applicants who failed. Average test time increased by 11.2 minutes, from 13.6 minutes for the current test to 24.8 minutes for the DPE. These increments in fail rate and test time translate into a large increase in the cost of administering the drive test. The finding that DPE fail rate for original applicants decreased slightly during the first 6 months of the pilot (from 49.1% in September 1994 to 44.4% in February 1995) suggests that, over time, applicants became better prepared for the DPE. It is uncertain whether this decline will continue. The results of the assessment of possible DPE modifications that would reduce test length showed that eliminating freeway driving was the only option that substantially impacted test validity, reducing the correlation of point score with group status from .30 to -.25. All the other changes slightly decreased validity, except eliminating street parking, which slightly increased the correlation coefficient. These results should be interpreted with caution because the most incompetent subjects (DQs) were not included in the simulations and it is uncertain whether the findings can be generalized to them. It is also possible that the simulation results do not accurately represent what would occur had shorter versions of the DPE actually been administered (all subjects in the simulations having completed the full DPE). In spite of these limitations, the relatively large drop in the validity coefficient produced by eliminating freeway driving provides strong evidence that not testing drivers on the freeway is the worst of all of the options considered. The fact that dropping freeway driving would greatly reduce the DPE's face validity also argues against that option. If the DPE had to be shortened to reduce administration costs, consideration should be given to the expected time savings that would be produced by each modification, which was not assessed in this study. Dropping the turn-and-stop skill test would almost certainly save more time than any of the other options (excluding dropping freeway driving). The time that would be saved by eliminating intersections and turns would probably be small, although it would vary according to the route. Dropping the
22

VALIDITY OF CAS DPE TEST

straight business and residential items may save a little more time than would dropping turns and intersections, but the savings would probably still be marginal. Another factor to be considered before changing the DPE is the effect of the modification on test fail rate, which would impact the cost of retesting applicants. Eliminating the skill test would reduce fail rate by at most 12 percentage points (the percentage of applicants DQed on the skill test), however the actual net increase would probably be much lower than this because many of those who DQ on the turn-and-stop would probably also have failed the on-road portion of the test. Not testing on the freeway would reduce fail rate by a maximum of five percentage points, the percentage of applicants identified from LRE comments on the DPE score sheets to have been DQed on the freeway. None of the other change options could be evaluated for their effect on fail rate due to the inability to match DQs with specific maneuvers on the DPE route.

REFERENCES

Hagge, R. A. (1994). The California driver performance evaluation project: An evaluation of a new driver licensing road test (Report No. 150). Sacramento: California Department of Motor Vehicles. Mackie, R. R., Wylie, C. D., Shultz, T., Engel, R., Townsend, M., Lammlein, S. E., & Johnson, S. (1989). Development of a recommended testing program for commercial motor vehicle operators (the CDL system). Lincoln, NE: Nebraska Department of Motor Vehicles and Washington DC: American Association of Motor Vehicle Administrators. Shumaker, N. C. (1994). The California driver performance evaluation project: An evaluation of the current driver licensing road test (Report No. 151). Sacramento: California Department of Motor Vehicles. Williams, R. L., & Shumaker, N. C. (1994). Class C drive test baseline study: Preliminary report. Sacramento: California Department of Motor Vehicles.

23

VALIDITY OF CAS DPE TEST

Appendix A

24

VALIDITY OF CAS DPE TEST

Appendix B

DAILY DPE ACTIVITY

Scheduled or Walk-in Drive Test Applicants Field office number: _________ Today's date (mmddyy) __________ LRE number _________ DL window number _________
Applicant status Test result Automatic DQ turnandstop (check if yes)

Driver license number

Novice original (check if yes)

Out-ofstate original (check if yes)

Limitedterm renewal (check if yes)

Other (specify)

If not Drive tested test on attempt freeway, number give (1, 2, 3) reason

If license restricted to no freeway, give reason

Automatic DQ predrive (check if yes)

Automatic DQ onroad (check if yes)

DPE score

Failed test (check if yes)

DPE start time (hr:min)

DPE end time (hr:min)

Drive test route number

DPE-related customer complaint (describe)

25

VALIDITY OF CAS DPE TEST

Appendix C
Table C-1 Correlation of Item Score and Group Status (Novice Original vs. Congratulations Renewal) for First-Attempt Subjects Completing the DPE (DQs Excluded) in Fullerton, West Covina, and Westminster (N = 1,563)
TURN AND STOP Traffic check Speed Braking Vehicle position PARKING LOT DRIVING Traffic check Speed STREET PARK Traffic check Signal Speed Parking Parallel INTERSECTIONS Through Traffic check Speed Unnecessary stop Stop Traffic check Speed Full stop Gap or limit line Start Traffic check Yield Speed 1 -.01 -.06* .05* 2 -.10* .07* -.02 3 -.08* .05 -.02 E -.11* -.02 -.01 .00 1 -.15* -.00 E .04 .07* -.04 -.02 -.10* 4 -.10* -.01 -.02 5 .02 .01 6 -.03 .03 X -.11* -.02 -.01 -.04 2 -.09* -.03 X .05* -.01 .01 .01

7 .01 -.03 -.02

8 .04 .03 -.01

-.03 -.03 -.05* .05

-.01 -.01 .02

-.04 .05* -.06* .03

-.05 .05* -.03 .07*

-.07* -.03 -.02 -.07*

-.03 .03 .03

-.04 -.02 .01 -.03

-.03 -.01 -.02 -.04

-.07* .00 -.01

-.06* -

-.04 -.02 -.01

-.03 -.05 -.01

-.03 -.06* .03

-.03 -

-.06* -

-.10* -.01 -.02

(Continued on next page)

26

VALIDITY OF CAS DPE TEST

Table C-1 (Continued) TURNS 1 Approach Traffic check Signal Speed Lane Unnecessary stop Stop Traffic check Speed Full stop Gap or limit line Wheels straight Turn/Complete Traffic check Steering control Too wide/short Correct lane Speed Signal -.04 .05* .01 -.03 -.09* 2 -.01 -.02 -.02 -.01 -.01 LEFT 3 -.05* -.04 -.01 -.02 -.04 4 -.02 -.01 .02 -.03 -.03 1 -.02 .01 -.02 -.01 .01 2 .05 -.01 -.03 -.04 -.07* RIGHT 3 -.05 .01 .01 -.03 -.04 4 -.07* -.02 .06 -.03 -.07*

-.02 -.01 .08* -.06*

-.02 .01 .04 -.03

-.02 -.01 .02 .02 -.05

-.02 -.02 .01 -.04

-.01 -.01 -.04 .05

-.01 -.01 -.04 -.03

-.02 -.05* .03

-.02 -.01 -.03 .03

-.05* -.04 -.03 -.02 -.04 -

-.03 -.03 -.01 -.02 -.03 .05*

-.06* -.02 -.05* -.02 -.03 -

-.07* -.05* -.07* -.02 -.04 -

-.05* -.02 -.01 -.02 -

-.02 -.03 .01 -.04 -.03 -

-.06* -.02 -.04 .01 -.03 -

-.03 -.02 .03 .07* -.03 -.01

STRAIGHT BUSINESS AND RESIDENTIAL B R Traffic checks -.12* -.13* Speed -.06* .02 Spacing -.04 .01 Lane position -.03 .03 LANE CHANGE Traffic check Signal Speed Spacing Steering control (Continued on next page) L .14* -.01 -.05* -.04 -.02 R .06* -.01 -.02 -.04 -.04

27

VALIDITY OF CAS DPE TEST

Table C-1 (Continued)

FREEWAY Entering Traffic check Signal Speed Spacing Lane position Merge Traffic check Signal Speed Spacing Lane position Steering control Lane Use Traffic check Speed Spacing Lane position Exiting Traffic check Signal Speed Spacing Lane position Steering control CURVE Entering speed Through speed Lane position

-.15* -.02 -.06* -.01 -.02

.01 -.04 -.15* -.05* -.08* -.06*

-.10* -.02 -.05* .03

-.09* -.04 -.06* -.03 -.05* .01

-.02 -.04 -.03

28

VALIDITY OF CAS DPE TEST

Table C-2 Number of Subjects (n) and Fail Rate for the DPE and Current Drive Tests by Field Office Current drive test n Fail rate (%) 89 18.0 97 26.8 91 28.6 83 21.7 82 22.0 76 38.2 86 29.1 72 31.9 99 20.2 84 22.6 101 34.7 96 17.7 86 19.8 101 29.7 99 21.2 97 24.7 170 34.1 112 21.4 92 23.9 79 22.8 * * 97 26.8 98 87 79 99 70 92 102 75 24.5 21.8 32.9 30.3 24.3 29.4 36.3 25.3 DPE Fail rate (%) 40.7 43.9 51.9 52.5 41.1 52.0 33.9 47.6 46.9 33.7 44.8 32.0 51.6 39.5 44.4 43.7 49.3 47.8 50.8 48.2 35.1 45.6 49.7 44.8 53.3 35.9 52.2 48.6 48.2 47.3

Office Arleta Bell Gardens Bellflower Chula Vista Compton Culver City Escondido Fullerton Glendale Hawthorne Hollywood Inglewood Laguna Hills Lincoln Park Montebello Oceanside Pasadena Placentia Pomona Poway San Clemente San Diego San Diego Clairemont San Pedro San Ysidro Torrance Van Nuys West Covina Westminster Winnetka
*not reported

n 1,655 1,998 1,654 958 1,082 666 717 704 1,780 860 529 626 1,032 1,166 1,039 1,339 1,749 901 738 737 507 645 1,444 913 720 1,033 624 1,277 1,250 957

29

VALIDITY OF CAS DPE TEST

Table C-3 Number of Subjects (n) and Turn-and-Stop DQ Rate by Field Office Office Arleta Bell Gardens Bellflower Chula Vista Compton Culver City Escondido Fullerton Glendale Hawthorne Hollywood Inglewood Laguna Hills* Lincoln Park Montebello Oceanside Pasadena Placentia Pomona Poway San Clemente San Diego San Diego Clairemont San Pedro San Ysidro Torrance Van Nuys West Covina Westminster Winnetka n 1,655 1,998 1,654 958 1,082 666 717 704 1,780 860 529 626 1,032 1,166 1,039 1,339 1,749 901 738 737 737 645 1,444 913 720 1,033 624 1,277 1,250 957 DQ rate (%) 8.4 14.3 20.1 12.3 11.6 3.9 16.9 12.4 13.0 8.0 15.3 4.2 0.9 7.2 14.8 3.5 12.8 13.3 18.3 12.8 7.9 14.3 5.6 16.3 6.4 9.5 9.3 16.5 20.0 6.7

*Laguna Hills adopted a 3-point turn in lieu of the turn-and-stop skill test.

30

VALIDITY OF CAS DPE TEST

Table C-4 Number of Subjects (n) and Mean Test Time (Minutes) for the DPE and Current Drive Tests by Field Office Current drive test n Mean time 80 13.2 94 13.4 86 12.5 82 14.6 80 12.0 70 13.5 84 16.5 56 14.6 97 13.0 80 12.9 91 11.2 94 18.1 79 14.3 98 11.9 98 13.3 86 13.9 163 12.0 108 13.3 85 12.5 77 14.0 * * 91 92 85 79 87 70 91 94 67 13.8 13.0 10.6 12.3 16.6 15.1 14.3 15.5 13.6 DPE n 1,605 1,985 1,619 948 1,068 663 708 704 1,768 831 524 612 1,048 1,140 1,031 1,322 1,724 897 720 726 507 634 1,420 903 713 1,024 619 1,269 1,240 936 Mean time 25.1 24.9 20.9 26.7 23.1 23.2 23.5 24.1 24.4 27.4 25.5 28.4 20.5 23.6 25.6 24.1 25.7 26.5 24.7 22.7 28.6 24.0 22.4 24.3 25.4 28.3 28.4 23.9 24.5 27.3

Office Arleta Bell Gardens Bellflower Chula Vista Compton Culver City Escondido Fullerton Glendale Hawthorne Hollywood Inglewood Laguna Hills Lincoln Park Montebello Oceanside Pasadena Placentia Pomona Poway San Clemente San Diego San Diego Clairemont San Pedro San Ysidro Torrance Van Nuys West Covina Westminster Winnetka
*not reported

31