If you want to build a self-driving car and test it on public roads in California, the stateâ€™s Department of Motor Vehicles says that every year you have to submit a disengagement reportâ€”basically a list of every time the human driver had to take over for the car. This year, Bosch, Delphi, Google, Nissan, Mercedes-Benz, Tesla, and Volkswagen Group were required to submit disengagement reports, and the results are largely what youâ€™d expect from a novel and complicated technology.
Google, as the company that’s driven the most miles on public roads in California, said it experienced 341 significant disengagement events over 424,000 miles of drivingÂ (PDF). Similarly, Nissan reported that it drove 1,485 miles on public roads in California and it experienced 106 disengagements. Delphiâ€™s two autonomous vehicles drove 16,662 miles and the company reported 405 disengagements. Tesla, for its part, reported no disengagementsÂ (PDF) from fully-autonomous mode from the time it was issued a permit to test self-driving cars in California.
While itâ€™s tempting to use those numbers as a comparison point as to how good a companyâ€™s autonomous vehicles are, there are many variables that could obscure an otherwise accurate comparison. The numbers only reflect miles driven on California roads and disengagements that happen in that state. If a company primarily tests its public road driving in another state, those numbers wonâ€™t be reflected in these reports.
Also, as Google notes, the technology is still in development, so engineers donâ€™t necessarily want to keep disengagements from happening because every mistake is a chance to build more fail-proof software. “Our objective is not to minimize disengagements; rather, it is to gather, while operating safely, as much data as possible to enable us to improve our self-driving system,â€ Google writes.
Google also offers the most detail of all the automakers as to why the disengagements occurred. 272 disengagements happened when the car detected some sort of anomaly that it could not reconcileâ€”these include things like a broken wire, anomalies that come from accelerometer or GPS readings, or “anomalies in the monitoring of key functions like steering and braking.â€
69 of Googleâ€™s disengagements happened when the driver was forced to take control of the carâ€”in these cases, Google determined that without intervention from the driver, the car would have crashed with another car or an object. The Guardian points out that Google actually says that its drivers took over â€œmany thousands of times,â€ but â€œthe company is reporting only 69 incidents because Google thinks Californiaâ€™s regulations require it to only report disengagements where drivers were justified in taking over, and not those where the car would have coped on its own.”
The other automakers did not include as much analysis in their reports. Mercedes-Benz listsÂ (PDF) all of its disengagement events with a short comment on the cause for the disengagement: â€œtechnology evaluation managementâ€ and â€œdriver was uncomfortableâ€ seem to be the two most-used reasons for stopping the autonomous drive. Volkswagen Group similarly uses vague languageÂ (PDF) to describe its autonomous drive failures. Bosch reported thatÂ (PDF) it only ever disengaged autonomous mode for a â€œplanned test of technologies.â€
Nissan and Delphi both had more robust reports of what happened and why the disengagement was needed. â€œA route could not be generated due to a localization error,â€ Nissan wrote in one instanceÂ (PDF). â€œAutonomous control shut off and the driver resumed manual control of the AV [Autonomous Vehicle].â€
â€AV did not recognize stopped vehicle in front of it,â€ another entry noted. â€œThe driver overrode the system with manual brake input, causing the control to disengage.â€
Delphi even broke down the causesÂ (PDF) for disengagement in a table. Among the reasons that the test drivers disengaged the company’s two autonomous vehicles were to make way for â€œemergency vehicles,â€ as well as â€œpoor lane markings,â€ â€œprecautionary intervention to give extra space for a cyclist,â€ and â€œtraffic light detectionâ€ caused by â€œpoor sun conditions.â€
The Guardian noted that when the California DMV was drafting rules to allow testing of autonomous vehicles on its roads, Google lobbied hard against the requirement to report disengagement numbers. Separately, companies that test autonomous driving on public roads are required to report any accidents that occur to the DMV. In June of this year, Google reported that it had been in 13 minor accidents with its self-driving cars, none for which the self-driving car was at fault.