National Transportation Safety Board

Washington, DC 20594

Office of the Chairman

February 1, 2021

Docket Management Facility, M-30

US Department of Transportation
1200 New Jersey Avenue SE
West Building, Ground Floor
Room W12-140
Washington, DC 20590-0001

Attention: Docket No. DOT-NHTSA-2020-0106

Dear Sir or Madam:

The National Transportation Safety Board (NTSB) has reviewed the National Highway
Traffic Safety Administration (NHTSA) advance notice of proposed rulemaking (ANPRM) titled
“Framework for Automated Driving System Safety,” published at 85 Federal Register 78058 on
December 3, 2020. In its notice, NHTSA requests comments on the development of a framework
for automated driving system (ADS) safety. 1 Specifically, the agency seeks input on its role in
facilitating ADS risk management through guidance, regulation, or both. NHTSA also requests
guidance on how it should select and design the structure of a safety framework and the appropriate
administrative mechanisms for improving safety, mitigating risk, and enabling the development
and introduction of innovative safety technology.

The NTSB recognizes NHTSA’s efforts to develop a framework for ADS safety. However,
we believe that the Department of Transportation (DOT) and NHTSA must act first to develop a
strong safety foundation that will support the framework envisioned for automated vehicles (AVs)
of the future. The foundation should include sensible safeguards, protocols, and minimum
performance standards to ensure the safety of motorists and other vulnerable road users. We also
call for the standardization of AV data collection to better understand automated control systems,
a requirement for safety critical information to be available and evaluated for developmental
ADSs, the development of performance standards to evaluate driver engagement, the improved
oversight of systems that may operate outside a vehicle’s operational design domain (ODD), and
the incorporation of more robust collision avoidance test procedures into the New Car Assessment
Program (NCAP).

1
ADS, as defined by SAE International and as used in the ANPRM, refers to driving automation levels 3, 4, and
5. An ADS is the hardware and software that are, collectively, capable of performing the entire dynamic driving task
on a sustained basis, regardless of whether it is limited to a specific operational design domain.

66531
 2

The ANPRM specifically asks for comments on 25 questions related to the safety
framework, NHTSA research, administrative mechanisms, and the agency’s statutory authority.
The NTSB’s response is not specific to each question, but rather, expresses key safety principles
that underlie the questions, based on knowledge gained from our investigations of crashes
involving vehicles equipped with various levels of automation. Our response first addresses the
importance of incorporating the lessons learned from NTSB crash investigations into the safety
framework. We then discuss the following eight foundational safety issues:

• Collision Avoidance Technologies—Foundational Building Blocks for Safety

• Safety Risk Management Requirements for Testing AVs on Public Roads
• State Oversight of AV Testing
• Risk Mitigation Pertaining to Monitoring Driver Engagement
• Risk Mitigation Pertaining to Operational Design Domain
• NHTSA Enforcement of AV Safety-Related Defects
• Event Data Recorders for AVs
• Enhancements to New Car Assessment Program

Lessons Learned from NTSB Crash Investigations

Although much attention and federal efforts have focused on highly automated SAE
International (SAE) Level 3–5 vehicles, lessons can be learned from the deployment of AVs on
our nation’s highways today. Between May 2016 and March 2019, the NTSB investigated four
crashes—three resulting in fatalities—that involved vehicles operating in partial automation
mode. 2 In addition, in July 2019, the NTSB completed an investigation of a minor crash involving
a highly automated shuttle on its first day of operation in Las Vegas, Nevada. 3 In November 2019,
the NTSB completed its investigation of the first fatal crash involving a test vehicle controlled by
a developmental ADS. That crash, which occurred in Tempe, Arizona, demonstrated the
complexity of ADS testing and highlighted the need for ADS developers, operators, and state and
federal agencies, specifically NHTSA, to play comprehensive and cooperative roles. 4

The lessons learned from the NTSB’s crash investigations contain important information
regarding the safe testing of AVs on public roads; the importance of driver/operator engagement
in AV operation; risk mitigation pertaining to the appropriate ODD for an AV; and other
improvements needed to establish a strong ADS safety foundation. An attachment to this response
lists open safety recommendations pertaining to AV safety that still require action by the DOT,
NHTSA, and others. The recommendations are discussed in more detail below.

2
See our recent reports on crashes in Williston, Florida (Highway Accident Report NTSB/HAR-17/02); Culver
City, California (Highway Accident Brief NTSB/HAB-19/07); Delray Beach, Florida (Highway Accident Brief
NTSB/HAB-20/01); and Mountain View, California (Highway Accident Report NTSB/HAR-20/01).
3
See our report about the Las Vegas crash (Highway Accident Brief NTSB/HAB-19/06).
4
See our report about the Tempe crash (Highway Accident Report NTSB/HAR-19/03).
 3

Collision Avoidance Technologies—Foundational Building Blocks for Safety

Section III of the ANPRM describes the core elements of ADS safety performance as
sensing, perception, planning, and control. 5 Although those functions are necessary for ADS
performance, they are not sufficient to ensure ADS safety, which depends on an array of other
functions and system capabilities and how the system interacts with the humans both inside and
outside an ADS-equipped vehicle. While a mature ADS may avoid many of the human driver
errors or poor choices that lead to crashes, an ADS can still find itself in crash-imminent scenarios
that warrant emergency maneuvering. Crash avoidance will depend on a vehicle’s mechanical
abilities and the underlying crash avoidance technologies. Certain advanced safety technologies,
which will likely serve as foundational “building block” technologies for AVs, have already proven
effective at preventing and mitigating crashes across all modes of highway transportation.

Since 1995, the NTSB has called for installing collision avoidance technology on passenger
cars and trucks. 6 Collision avoidance technologies, especially forward collision warning and
automatic emergency braking systems, have shown safety benefits in reducing the frequency and
severity of crashes. 7 Although the effectiveness of the technologies has been demonstrated, their
incorporation into vehicle fleets remains slow. As a result, in May 2015, the NTSB issued
recommendations to vehicle manufacturers to install the systems as standard equipment in all new
vehicles. 8 In the same report, the NTSB issued recommendations to NHTSA to incorporate a rating
system into the NCAP for forward collision avoidance systems and to include those ratings on the
Monroney label. 9

As NHTSA moves toward an ADS safety framework, it is important that the agency
prioritize the development of minimum performance standards for collision avoidance
technologies and require the systems as standard equipment on all new vehicles. Independently of
whether a vehicle is driven by a human driver or an ADS, NHTSA should focus on performance
standards for collision avoidance systems. The standards could be technology-neutral and would
address NHTSA’s mission to prevent, reduce, or mitigate crashes. In cars with human drivers,
collision avoidance technologies are redundant systems intended to aid drivers in situations where
their performance is not ideal. For an ADS, collision avoidance technologies could similarly
function as redundant systems to avoid or mitigate crashes when the ADS cannot react on its own

5
“Sensing” refers to the ability of an ADS to receive adequate information from the vehicle’s internal and external
environment through connected sensors. “Perception” refers to the ability of an ADS to interpret information about its
environment obtained through its sensors. “Planning” refers to the ability of an ADS to establish and navigate the
route it will take on the way to its destination. The “control” function of an ADS refers to the system’s ability to
execute the driving functions necessary to carry out a continuously updated driving plan by delivering appropriate
control inputs such as steering, propulsion, and braking.
6
In 1995, the NTSB issued Safety Recommendation H-95-44 to the DOT, asking it to begin testing collision
warning systems in commercial fleets. Because of the DOT’s lack of progress, the NTSB classified the
recommendation “Closed—Unacceptable Action” in August 1999.
7
The NTSB discussed the safety benefits of collision avoidance technologies in a special investigation report
published in May 2015 (The Use of Forward Collision Avoidance Systems to Prevent and Mitigate Rear-End Crashes,
Special Investigation Report NTSB/SIR-15/01).
8
The recommendations (Safety Recommendations H-15-8 and -9, currently classified “Open—Acceptable
Response”) were issued in NTSB/SIR-15/01. For more information about NTSB safety recommendations, see the
Safety Recommendation Database at www.ntsb.gov.
9
See Safety Recommendation H-15-6, currently classified “Open—Acceptable Response.”
 4

to a hazardous situation. In the Tempe crash investigation, the NTSB found that Uber Advanced
Technologies Group’s (ATG) deactivation of the Volvo forward collision warning and automatic
emergency braking systems without replacing their full capabilities removed a layer of safety
redundancy and increased the risks associated with testing ADSs on public roads. Uber ATG did
not violate any Federal Motor Vehicle Safety Standards (FMVSSs) because none exist that require
a minimum level of collision avoidance performance. Postcrash, Uber ATG worked with Volvo to
ensure that the Volvo collision avoidance system was independent and functional when the Uber
ATG ADS was operational, thereby adding a layer of safety redundancy.

Widespread deployment of collision avoidance technologies now will help save lives and
can be instrumental in building public confidence in the capabilities of new technologies as higher
levels of automation are introduced.

Safety Risk Management Requirements for Testing AVs on Public Roads

Section II of the ANPRM describes at length NHTSA’s perception of how prototype ADSs
are being tested on public roads. The discussion illustrates NHTSA’s belief that before public road
testing is conducted, companies undertake a rigorous engineering and safety analysis, with
mitigation strategies in place to address potential risks. However, the NTSB has found that
NHTSA’s perception of the safety of ADS testing is probably unrealistic. In the Las Vegas
investigation, the NTSB learned that as part of its declaration for importing a vehicle without
traditional driving controls (such as steering wheels), the shuttle operator (Keolis North America)
stated to NHTSA that drivers (attendants) who had been trained in all aspects of the vehicle’s
operation would be in the vehicle whenever it was operating and that they would be positioned
where they could take control if necessary. 10 The company also reported that the vehicle was fully
equipped for manual operation. Nevertheless, the NTSB determined that the shuttle attendant did
not have easy access to the manual controller, which limited his ability to take control of the vehicle
before the crash.

Further, in our investigation of the fatal crash involving a developmental ADS vehicle in
Tempe, the NTSB found significant deficiencies in the ADS developer’s management of safety
risk, as well as in NHTSA’s and the state’s oversight of ADS testing. The NTSB stressed that
NHTSA needs to require basic information from developers to ensure the safe testing of
ADS-equipped vehicles on public roads. We also argued that NHTSA should make more effective
and broader use of an already established basic framework for safe ADS testing—NHTSA’s AV
policy.

In the second iteration of its AV policy (AV 2.0), NHTSA provided guidance in the form of
12 safety-relevant elements and encouraged ADS developers and operators to submit voluntary
safety self-assessment reports describing their approach to safety. 11 Although these components of

10
Title 49 Code of Federal Regulations Part 591 (“Importation of Vehicles and Equipment Subject to Federal
Safety, Bumper and Theft Prevention Standards”) requires importers to file a declaration about a vehicle’s eligibility
for importation.
11
The 12 safety elements described in AV 2.0 are system safety, operational design domain, object event detection
and response, fallback (minimal risk condition), validation methods, human-machine interface, vehicle cybersecurity,
crashworthiness, postcrash ADS behavior, data recording, consumer education and training, and federal/state/local
laws.
 5

NHTSA’s AV policy are promising, challenges remain―specifically, the lack of a requirement for
mandatory submission of the safety self-assessment reports and the absence of a process for
NHTSA to evaluate their adequacy.

As a result of its investigation of the Tempe crash, the NTSB recommended that NHTSA
require the submission of safety self-assessment reports and establish an ongoing process for
evaluating them, determining whether appropriate safeguards―such as adequate monitoring of
vehicle operator engagement, if applicable―are included for testing a developmental ADS on
public roads. 12 We view such an evaluation as establishing a minimum level of safety for testing
that developers can achieve and that states can use when determining whether to allow ADS testing
in their state. As the NTSB has previously stated to NHTSA, NHTSA’s general and voluntary
guidance of emerging and evolutionary technological advancements shows a willingness to let
manufacturers and operational entities define safety. We urge NHTSA to lead with detailed
guidance and specific standards and requirements. 13

The traditional division of oversight, in which NHTSA regulates vehicle safety and the
states monitor drivers, may not apply to a developmental ADS. It might not be immediately
apparent who controls the vehicle, or whether vehicle control and supervision are shared between
the computer (the vehicle) and the human operator. A lack of appropriate policy from NHTSA and
the states leaves the public vulnerable to potentially unsafe testing practices.

To ensure that testing of AVs on public roads is conducted with minimal risk, meaningful
action from both NHTSA and the states is critical. Additionally, manufacturers must ensure that
the design, development, verification, and validation of safety-related underlying electronics and
software are reliable and safe for the conditions a vehicle is designed to encounter.

State Oversight of AV Testing

In the absence of federal ADS safety standards or specific ADS assessment protocols, states
have begun legislating requirements for AV testing, resulting in a patchwork of laws and state-
level requirements. The development of state-based requirements can be attributed to states’
concerns about the safety risk of introducing ADS-equipped vehicles on public roads. The
requirements vary. Some states, such as Arizona, impose minimal restrictions. Other states have
established requirements that include an in-depth application and review process. In the Tempe
crash investigation, we determined that Arizona’s lack of a safety-focused application-approval
process for ADS testing at the time of the crash, and its inaction in developing such a process after
the crash, demonstrated the state’s shortcomings in improving the safety of ADS testing and
safeguarding the public.

States that have no, or only minimal, requirements related to AV testing can improve the
safety of such testing by implementing a thorough application and review process before granting
testing permits. Because states would benefit from adopting regulations that require a thorough
review of ADS developers’ safety plans, including methods of risk management, we recommended

12
See Safety Recommendations H-19-47 and -48, currently classified “Open—Unacceptable Response.”
13
See NTSB response dated December 20, 2018, to notice of request for comments: Preparing for the Future of
Transportation: Automated Vehicles 3.0 (AV 3.0), Docket No. DOT–OST–2018–0149.
 6

that the American Association of Motor Vehicle Administrators encourage states to (1) require
developers to submit an application for testing ADS-equipped vehicles that, at a minimum, details
a plan to manage the risk associated with crashes and operator inattentiveness and establishes
countermeasures to prevent crashes or mitigate crash severity within the ADS testing parameters,
and (2) establish a task group of experts to evaluate the application before granting a testing
permit. 14

The ANPRM discusses NHTSA’s recent launch of the Automated Vehicle Transparency
and Engagement for Safe Testing (AV TEST) initiative. The NTSB commented on the initiative
and expressed concern about the lack of a requirement for specificity in testing information.15
Because the initiative is voluntary for ADS developers, it provides only a partial perspective into
ADS testing across the country. In addition, because NHTSA does not evaluate the information
provided by initiative participants, ADS developers largely ignore the response guidelines, and
their reports are generally devoid of technical and safety-relevant information. Such foundational
deficiencies require attention in NHTSA’s efforts to establish a safety framework for ADS.

Risk Mitigation Pertaining to Monitoring Driver Engagement

Section III of the ANPRM discusses the safety standard called “Safety of the Intended
Functionality” (SOTIF) as it relates to human-machine interaction and conceivable misuse of the
system, performance limitations of sensors or systems, and unanticipated changes in an automated
vehicle’s environment. As ADSs are developed and deployed, situations are likely to arise that
necessitate a vehicle-initiated handover or an operator-initiated takeover of vehicle control. In the
Tempe crash of a prototype ADS vehicle and in the four crashes involving partially automated
vehicles, the NTSB found that the drivers were distracted and not appropriately supervising
automation performance or monitoring the driving environment. In the Las Vegas crash, the NTSB
found that although the attendant was engaged, the design of the ADS vehicle did not enable an
operator-initiated takeover of vehicle control.

Driver/operator situational awareness and engagement are needed to ensure the safety of
ADS deployment, especially during on-road testing. SAE Level 2 partial automation systems, and
in some respects Level 3 ADSs, require the driver or operator to monitor the highway and remain
able to take control of the vehicle at any time or when signaled by the vehicle. The success of these
AVs depends on the driver completing a monitoring task that requires sustained attention; however,
humans generally perform poorly in the role of monitor. Also, if the automated control system
behaves consistently and reliably for prolonged periods, the user of that system can become
complacent about its operation and may not respond appropriately when a situation requires him
or her to act.

Because driver/operator attention is an integral component of lower level automation

systems, a driver monitoring system must be able to assess whether and to what degree the driver
is performing the role of automation supervisor. No minimum performance standards exist for the
appropriate timing of alerts, the type of alert (visual, auditory, or haptic [touch]), or the use of

14
See Safety Recommendation H-19-51, classifed “Closed—Acceptable Action.”
15
See NTSB response dated August 21, 2020, to notice of request for comments: Automated Vehicle Transparency
and Engagement for Safe Testing (AV TEST) Initiative, Docket No. DOT–NHTSA-2020–0070.
 7

redundant monitoring sensors to ensure driver/operator engagement. As a result of its investigation

of a crash in Mountain View, California, involving a vehicle operating with partial driving
automation, the NTSB recommended that NHTSA work with SAE to develop performance
standards for driver monitoring systems that will minimize driver disengagement, prevent
automation complacency, and account for foreseeable misuse of the automation. 16

NHTSA should include user monitoring in the development of an AV safety framework.

AVs must give alerts to capture the attention of a driver or operator and allow sufficient time for
the person to respond and assume the dynamic driving task for any level of automation that may
require human intervention. Driver/operator monitoring is critical during on-road testing of a
developmental ADS.

Risk Mitigation Pertaining to Operational Design Domain

The ANPRM describes one of NHTSA’s key research tracks, which focuses on identifying
methods, metrics, and tools to assess how well an ADS-equipped vehicle performs both normal
driving tasks and crash avoidance. Such assessments include system performance and behavior
relative to the system’s stated ODD and event detection and response capabilities, as well as
fail-safe capabilities if the system is confronted with conditions outside its ODD. 17 The NTSB
supports this research track and strongly recommends that the research extend to all levels of
automation, including partial driving automation systems. Lessons learned from lower levels of
automation can be applied to ADSs.

Today’s Level 2 partial driving automation systems can assess a vehicle’s location and the
roadway type or classification and determine whether the roadway is appropriate to the system’s
ODD. After a crash in Williston, Florida, that involved a driver operating outside the
manufacturer’s ODD in a vehicle with a partial driving automation system, the NTSB
recommended that NHTSA develop a method to verify that manufacturers of vehicles equipped
with Level 2 vehicle automation systems incorporate system safeguards that limit the use of
automated vehicle control systems to the conditions for which they were designed. 18

In response to the NTSB’s recommendation, NHTSA responded that “the agency has no
current plans to develop a specific method” to address the NTSB’s concern. Because of NHTSA’s
failure to act on this important safety recommendation, Tesla (the manufacturer of the Williston
crash vehicle) continued to permit AV operation outside the ODD. Contrary to SAE J3016
guidance, which considers the ODD for Level 2 systems to be limited, Tesla advised the NTSB
that it believes that “ODD limits are not applicable for Level 2 driver assist systems, such as
Autopilot, because the driver determines the acceptable operating environment.” In March 2019,
because of Tesla’s lack of appropriate safeguards and NHTSA’s inaction, another fatal crash
occurred in Delray Beach, Florida, under circumstances very similar to the Williston crash. The
NTSB found that a contributing causal factor in the Delray Beach crash was NHTSA’s failure “to

16
See Safety Recommendations H-20-3 and -4, currently classified “Open―Initial Response Received.”
17
ODD refers to the conditions in which the automation system is intended to operate. Examples of such
conditions include roadway type, geographic location, clear roadway markings, weather conditions, speed range,
lighting conditions, and other manufacturer-defined system performance criteria or constraints.
18
See Safety Recommendation H-17-38, currently classified “Open—Unacceptable Response.”
 8

develop a method of verifying manufacturers’ incorporation of acceptable system safeguards for

vehicles with Level 2 automation capabilities that limit the use of automated vehicle control
systems to the conditions for which they were designed.”

The NTSB remains concerned about NHTSA’s continued failure to recognize the
importance of ensuring that acceptable safeguards are in place so that vehicles do not operate
outside their ODDs and beyond the capabilities of their system designs. As manufacturers advance
the development of automated control systems, it is evident that there is a fluid progression of
capabilities and that the SAE levels of automation may not adequately reflect how control systems
are actually used. Because NHTSA has put in place no requirements, manufacturers can operate
and test vehicles virtually anywhere, even if the location exceeds the AV control system’s
limitations. For example, Tesla recently released a beta version of its Level 2 Autopilot system,
described as having full self-driving capability. By releasing the system, Tesla is testing on public
roads a highly automated AV technology but with limited oversight or reporting requirements.
Although Tesla includes a disclaimer that “currently enabled features require active driver
supervision and do not make the vehicle autonomous,” NHTSA’s hands-off approach to oversight
of AV testing poses a potential risk to motorists and other road users.

NHTSA refuses to take action for vehicles termed as having partial, or lower level,
automation, and continues to wait for higher levels of automation before requiring that AV systems
meet minimum national standards. As a result of its Mountain View crash investigation, the NTSB
concluded that NHTSA’s failure to ensure that vehicle manufacturers of SAE Level 2 driving
automation systems incorporate appropriate system safeguards to limit operation of these systems
to the ODD compromises safety. Policy direction needs to apply seamlessly as AV development
proceeds. NHTSA must take regulatory action now to minimize the risks associated with the ODD
of all levels of vehicle automation.

NHTSA Enforcement of AV Safety-Related Defects

NHTSA has informed the NTSB that it plans to ensure the safety of lower levels of driving
automation systems through its enforcement authority and a surveillance program aimed at
identifying safety-related trends in design or performance defects, and not through regulations.19
This approach is misguided because it relies on waiting for problems to occur rather than
addressing safety issues proactively. For an acceptable level of safety to be achieved, a robust
surveillance program must be in place, so that safety-related vehicle defects can be identified in a
timely manner.

NHTSA’s enforcement guidance states that when an automated safety technology causes
crashes or injuries, or poses other safety risks, the agency will evaluate such technology through
its investigative authority to determine whether the technology presents an unreasonable risk to

19
In response to Safety Recommendation H-17-38, NHTSA informed the NTSB that “the agency has no current
plans to develop a specific method to verify manufacturers of vehicles equipped with Level 2 systems incorporate
safeguards limiting the use of automated vehicle control systems to those conditions for which they were designed.
Instead, if NHTSA identifies through its research or otherwise, any incidents in which a system did not perform as
designed, it will exercise its enforcement authority as appropriate.”
 9

safety. 20 The guidance also states that manufacturers should take the necessary steps to ensure that
technology introduced on US roadways accounts for any foreseeable misuse, particularly in
circumstances that require driver interaction while a vehicle is in operation. 21

Included in the enforcement guidance is information directly relevant to the AV crashes

investigated by the NTSB. The NHTSA guidance states that “a semi-autonomous driving system
that allows a driver to relinquish control of the vehicle while it is in operation but fails to adequately
account for reasonably foreseeable situations where a distracted or inattentive driver must retake
control of the vehicle at any point may be an unreasonable risk to safety.”

In determining whether a vehicle design poses an “unreasonable risk” to safety, NHTSA is

charged with answering the question through a forward-looking risk analysis. According to
NHTSA’s enforcement guidance, the purpose of a forward-looking risk analysis “is not to protect
individuals from the risks associated with defective vehicles only after serious injuries have
already occurred; it is to prevent serious injuries stemming from established defects before they
occur.”

On June 28, 2016, NHTSA’s Office of Defects Investigation (ODI) opened a preliminary
investigation into the design and performance of the Tesla automation systems in use at the time
of the Williston crash. 22 An NTSB review of the ODI investigation report identified shortfalls in
the agency’s evaluation of Tesla’s Autopilot. 23 The NTSB concluded that the NHTSA ODI had
failed to thoroughly investigate the degree to which drivers misuse the Autopilot system, the
foreseeable consequences of its continued use by drivers beyond the system’s ODD, and the
effectiveness of the driver monitoring system in ensuring driver engagement.

As a result of the Mountain View crash investigation, which incorporated lessons learned
from the crashes in Williston, Delray Beach, and Culver City, the NTSB recommended that
NHTSA evaluate Tesla Autopilot-equipped vehicles to determine if the system’s operating
limitations, the foreseeability of driver misuse, and the ability to operate the vehicles outside the
intended ODD pose an unreasonable risk to safety; and that if safety defects are identified, the
agency should use its enforcement authority to ensure that Tesla takes corrective action. 24 To date,
NHTSA has shown no indication that it is prepared to respond effectively and in a timely manner
to potential AV safety-related defects. This deficiency requires immediate attention if NHTSA’s
enforcement authority is to be one of the primary mechanisms that the agency plans on using to
ensure ADS safety.

20
NHTSA Enforcement Guidance Bulletin 2016-02: “Safety-Related Defects and Automated Safety
Technologies,” 81 Federal Register 65705.
21
NHTSA has defined misuse as when an operator, having knowledge and understanding of the system’s
limitations and operational use instructions, deliberately chooses not to act according to the intent and design of the
automated component. When a driver, having full knowledge of the responsibility to supervise and monitor the
roadway, engages in a secondary task that may disrupt or eliminate the capability to effectively perform monitoring
duties, such disengagement can qualify as misuse.
22
NHTSA ODI investigation PE 16-007 (Automated vehicle control systems), closed January 19, 2017.
23
See our report on the crash in Mountain View, California (Highway Accident Report NTSB/HAR-20/01).
24
Safety Recommenation H-20-2, currently classified “Open—Initial Response Received.”
 10

Event Data Recorders for AVs

On December 13, 2012, NHTSA issued an NPRM that proposed a new FMVSS mandating
that an event data recorder (EDR) that meets 49 Code of Federal Regulations Part 563
requirements be installed on most light vehicles. On February 8, 2019, NHTSA withdrew the
NPRM because the agency determined that a mandate was not necessary: NHTSA’s internal
analysis showed that over 99 percent of light vehicles sold were already being equipped with EDRs
that met Part 563 requirements. NHTSA added that, given the near-universal installation of EDRs
in light vehicles, it no longer believed that the safety benefits of mandating EDRs justified
expending limited agency resources.

In withdrawing the final rule, NHTSA said that it would continue its efforts to modernize
and improve EDR regulations, including fulfilling the agency’s statutory mandate to promulgate
regulations establishing an appropriate recording duration for EDR data to “provide accident
investigators with vehicle-related information pertinent to crashes involving such motor
vehicles.” 25 Because the 49 Code of Federal Regulations Part 563 data recording requirements
codified more than a decade ago are limited (only 15 data elements require reporting), NHTSA
stated in withdrawing the final rule that it is actively investigating whether the agency should
consider revising the data elements covered by Part 563 to account for advanced safety features.

In recent AV crash investigations, NTSB investigators retrieved data from the EDR, but the
data did not address the status of AV activation, engagement, or object detection and classification.
As a result, the NTSB coordinated with the manufacturer and operator to use other proprietary data
to interpret the automated system’s functionality. However, this type of data is not available on
many vehicles operating with automated systems. Further, there are currently no commercially
available tools for independently retrieving and reviewing non-EDR vehicle data, and many
manufacturers maintain tight control and access to postcrash proprietary information associated
with their vehicles.

As more manufacturers deploy automation systems in their vehicles, it will be necessary

to develop detailed information about how the automated systems, and possibly drivers or vehicle
operators, perform and respond in a crash. Manufacturers, regulators, and crash investigators all
need specific data in the event of a system malfunction, near-crash, or crash. Recorded data can be
used to improve the automated systems and to understand situations that may not have been
considered in the original design. Further, data are needed to distinguish between automated
control and driver action.

After the Williston crash, the NTSB recommended that the DOT define the parameters
necessary to understand AV control systems. 26 Another recommendation was made to NHTSA to,
using the parameters defined by the DOT as necessary to understand AV control systems, define a
benchmark for new vehicles equipped with automated vehicle control systems so that they capture

25
See the Fixing America’s Surface Transportation (FAST) Act, Public Law 114-94 (December 4, 2015), section
24303.
26
See Safety Recommendation H-17-37, currently classifed “Open—Unacceptable Response.”
 11

data that reflect the vehicle’s control status and the frequency and duration of control actions
needed to adequately characterize driver and vehicle performance before and during a crash. 27

With the increasing number of AVs using different automated technologies being tested
and in some cases being sold to the public, standardized data elements, recording, and access to
safety event data are essential to the development of a framework for ADS safety. NHTSA needs
to advance its efforts to modernize and improve EDR regulations so that they focus on the
performance of advanced safety features.

Enhancements to New Car Assessment Program

Section IV of the ANPRM describes voluntary mechanisms that could be used to

implement a safety framework. Short of setting a safety standard, NHTSA discusses the potential
for adding an ADS competency evaluation to the NCAP. NHTSA envisions that an evaluation
could be used to measure the performance of an ADS in navigating a variable environment and a
complex set of interactions with other road users. Rather than evaluating the driving performance
of an ADS system through the NCAP, the NTSB believes that NHTSA should focus on the
development and application of testing procedures to assess the performance of forward collision
avoidance systems. 28 All ADS-equipped vehicles should be expected to avoid collisions while
adhering to a driving model that minimizes the risks of being involved in crash-imminent situations
and observes operational limitations. The information the NCAP provides would enable consumers
to compare the safety of new vehicles and make informed purchasing decisions, while providing
ADS developers with performance targets for collision avoidance systems. Moreover, the
information would encourage automakers to compete on the basis of safety.

For years, the NTSB has supported the concept of the NCAP being an incentive for
deploying collision avoidance technology. However, in 2015 we concluded that NHTSA’s existing
testing scenarios and protocols for the assessment of forward collision avoidance systems in
passenger vehicles do not adequately represent the range of velocity conditions seen in crashes. 29
As a result of its Mountain View crash investigation, the NTSB reiterated Safety Recommendation
H-15-4 and also recommended that NHTSA expand NCAP testing of forward collision avoidance
systems to address common obstacles found in the highway operating environment. 30

On November 21, 2019, NHTSA published a request for comments (RFC) on nine draft
test procedures to assess the performance of intersection safety assist systems in cross-traffic and
left-turn, across-path driving situations, as well as pedestrian automatic emergency braking
systems in daytime scenarios, both of which relate to forward collision avoidance. Nevertheless,
the NTSB remains concerned about NHTSA’s approach and continued delays in implementation.
As we stated in our response to the RFC, we remain very concerned by language used in the RFC
stating that NHTSA’s work is intended “for research purposes only” and not to support rulemaking

27
See Safety Recommendations H-17-39, currently classifed “Open—Unacceptable Response.”
28
See Safety Recommendation H-15-4, currently classified “Open—Unacceptable Response.”
29
See our special investigation report, The Use of Forward Collision Avoidance Systems to Prevent and Mitigate
Rear-End Crashes (NTSB/SIR-15/01).
30
See Safety Recommendation H-20-1, currently classified “Open—Initial Response Received.”
 12

or the NCAP. Since receiving the NTSB’s safety recommendations in 2015, NHTSA still has not
made any enhancements to the NCAP to address the performance of collision avoidance systems. 31

Evaluation by the NTSB has found that the European NCAP is much more robust than
NHTSA’s NCAP, includes many more testing scenarios and a wider range of speeds, and has begun
assessing the performance of partial driving automation systems. US consumers should be
provided with the same level of information about the safety of new vehicles as consumers in
Europe receive. Furthermore, manufacturers and ADS developers would benefit from a consistent
level of safety in the global environment. The NTSB supports the concept of enhancing NHTSA’s
NCAP and using it as a tool to improve ADS safety. However, the NTSB remains concerned about
NHTSA’s lack of progress on the performance of the building blocks for future automation
systems.

Summary

This response focuses on some of the foundational safety issues we believe must be
addressed before NHTSA can develop an effective framework for ADS safety. Even though we do
not comment specifically on other issues that complicate the development of a safe ADS (such as
cybersecurity standards, electronic safety standards, over-the-air update standards, and the FMVSS
revision process), we plan to continue using our crash investigations to make commonsense
recommendations for preventing future crashes and, we hope, improving consumer confidence in
AV safety. The NTSB appreciates the opportunity to comment and recognizes the challenges that
lie ahead for the DOT and NHTSA in developing a framework for ADS safety.

Sincerely,

Robert L. Sumwalt, III

Chairman

Attachment: NTSB Safety Recommendation List

31
See NTSB response dated January 15, 2020, to notice of request for comments: Advanced Driver Assistance
Systems Draft Research Test Procedures, Docket No. DOT–NHTSA-2019–0102.
 13

NTSB Safety Recommendation List

H-15-4: To the National Highway Traffic Safety Administration—Develop and

apply testing protocols to assess the performance of forward collision avoidance
systems in passenger vehicles at various velocities, including high speed and high
velocity-differential. (Status: Open—Unacceptable Response)

H-15-6: To the National Highway Traffic Safety Administration—Expand the New

Car Assessment Program 5-star rating system to include a scale that rates the
performance of forward collision avoidance systems. (Status: Open—Acceptable
Response)

H-15-7: To the National Highway Traffic Safety Administration—Once the rating

scale, described in Safety Recommendation H-15-6, is established, include the
ratings of forward collision avoidance systems on the vehicle Moroney labels.
(Status: Open—Acceptable Response)

H-15-8: To Passenger Vehicle, Truck-Tractor, Motorcoach, and Single-Unit Truck

Manufacturers—Install forward collision avoidance systems that include, at a
minimum, a forward collision warning component, as standard equipment on all
new vehicles. (Status: Open—Acceptable Response)

H-15-9: To Passenger Vehicle, Truck-Tractor, Motorcoach, and Single-Unit Truck

Manufacturers—Once the National Highway Traffic Safety Administration
publishes performance standards for autonomous emergency braking, install
systems meeting those standards on all new vehicles. (Status: Open—Acceptable
Response)

H-17-37: To the US Department of Transportation—Define the data parameters

needed to understand the automated vehicle control systems involved in a crash.
The parameters must reflect the vehicle’s control status and the frequency and
duration of control actions to adequately characterize driver and vehicle
performance before and during a crash. (Status: Open—Unacceptable Response)

H-17-38: To the National Highway Traffic Safety Administration—Develop a

method to verify that manufacturers of vehicles equipped with Level 2 vehicle
automation systems incorporate system safeguards that limit the use of automated
vehicle control systems in those conditions for which they were designed. (Status:
Open—Unacceptable Response)

H-17-39: To the National Highway Traffic Safety Administration—Use the data

parameters defined by the US Department of Transportation in response to Safety
Recommendation H-17-37 as a benchmark for new vehicles equipped with
automated vehicle control systems so that they capture data that reflect the vehicle’s
control status and the frequency and duration of control actions needed to
adequately characterize driver and vehicle performance before and during a crash;
the captured data should be readily available to, at a minimum, NTSB investigators
and NHTSA regulators. (Status: Open—Unacceptable Response)
 14

H-17-41: To the manufacturers of vehicles equipped with Level 2 vehicle

automation systems (Volkswagen Group of America, BMW of North America,
Nissan Group of North America, Mercedes-Benz USA, Tesla Inc., and Volvo
Group of North America)—Incorporate system safeguards that limit the use of
automated vehicle control systems to those conditions for which they were
designed. (Status: Open—Acceptable Response; Tesla Status: Open—
Unacceptable Response)

H-17-42: To the manufacturers of vehicles equipped with Level 2 vehicle

automation systems (Volkswagen Group of America, BMW of North America,
Nissan Group of North America, Mercedes-Benz USA, Tesla Inc., and Volvo
Group of North America)—Develop applications to more effectively sense the
driver’s level of engagement and alert the driver when engagement is lacking while
automated vehicle control systems are in use. (Status: Open—Acceptable
Response; Tesla Status: Open—Unacceptable Response)

H-19-47: To the National Highway Traffic Safety Administration—Require

entities who are testing or who intend to test a developmental automated driving
system on public roads to submit a safety self-assessment report to your agency.
(Status: Open—Unacceptable Response)

H-19-48: To the National Highway Traffic Safety Administration—Establish a

process for the ongoing evaluation of the safety self-assessment reports as required
in Safety Recommendation H-19-47 and determine whether the plans include
appropriate safeguards for testing a developmental automated driving system on
public roads, including adequate monitoring of vehicle operator engagement, if
applicable. (Status: Open—Unacceptable Response)

H-19-49: To the state of Arizona—Require developers to submit an application for

testing automated driving system (ADS)-equipped vehicles that, at a minimum,
details a plan to manage the risk associated with crashes and operator
inattentiveness and establishes countermeasures to prevent crashes or mitigate
crash severity within the ADS testing parameters. (Status: Open—Await Response)

H-19-50: To the state of Arizona—Establish a task group of experts to evaluate

applications for testing vehicles equipped with automated driving systems, as
described in Safety Recommendation H-19-49, before granting a testing permit.
(Status: Open—Await Response)

H-19-52: To the Uber Technologies, Inc., Advanced Technologies Group—

Complete the implementation of a safety management system for automated
driving system testing that, at a minimum, includes safety policy, safety risk
management, safety assurance, and safety promotion. (Status: Open—Acceptable
Response)
 15

H-20-1: To the National Highway Traffic Safety Administration—Expand New

Car Assessment Program testing of forward collision avoidance system
performance to include common obstacles, such as traffic safety hardware, cross-
traffic vehicle profiles, and other applicable vehicle shapes or objects found in the
highway operating environment. (Status: Open—Initial Response Received)

H-20-2: To the National Highway Traffic Safety Administration—Evaluate Tesla

Autopilot-equipped vehicles to determine if the system’s operating limitations, the
foreseeability of driver misuse, and the ability to operate the vehicles outside the
intended operational design domain pose an unreasonable risk to safety; if safety
defects are identified, use applicable enforcement authority to ensure that Tesla Inc.
takes corrective action. (Status: Open—Initial Response Received)

H-20-3: To the National Highway Traffic Safety Administration—For vehicles

equipped with Level 2 automation, work with SAE International to develop
performance standards for driver monitoring systems that will minimize driver
disengagement, prevent automation complacency, and account for foreseeable
misuse of the automation. (Status: Open—Initial Response Received)

H-20-4: To the National Highway Traffic Safety Administration—After

developing the performance standards for driver monitoring systems recommended
in Safety Recommendation H-20-3, require that all new passenger vehicles with
Level 2 automation be equipped with a driver monitoring system that meets these
standards. (Status: Open—Initial Response Received)