We’re engineering leads at Waymo and we're here to answer your questions on hardware and software development for self-driving technology - Ask Us Anything!

Questions most likely for Nathaniel.

As riders(Waymo One), my wife and I get strange routing decisions. Curious what ultimately causes the vehicles to intentionally avoid quicker routes? Do the cars prefer less traffic at the expense of time? Will they avoid construction areas even if the road is still in use and more efficient? And sometimes they'll go all the way around an area, ignoring a straight line to our destination. Do the turns required to get into a destination also change their pathing?

Thanks so much for riding with us! I ride in the cars all over Mountain View, and I sometimes get the scenic route, so I know where you're coming from!

When the car is choosing its preferred route, it considers a number of factors: the length of different routes, traffic on the roads, any construction or other slowdowns.. As you guessed, some maneuvers may lead to a longer route time so we’re able to take those into consideration too. - Nathaniel

Pedestrians will often make eye contact with drivers to make sure they have been noticed before crossing the road. How will you address the inability of pedestrians to know whether the driverless cars is aware of their presence?

Overall, we are extremely cautious around pedestrians, and we slow and yield, which helps indicate that we see them. We've found that actions speak pretty clearly -- which is the same situation that humans are in at night, for example, when eye contact doesn't work! - Nathaniel

You have been scaling driverless tests and driverless rides, so you must have some high level of confidence that these vehicles will not cause an accident. What kind of internal metrics do you use that give you this confidence? Do you have a way to calculate the probability an accident would occur for a given route? Do you have an estimated accident per mile ( or at fault accident per mile ) for those driverless vehicles you are dispatching in Chandler area? Or do you measure confidence in an entirely different way?

(credit /u/Mattsasa for question)

what key metrics or milestones will you wait to hit before removing the safety driver?

We have been giving some riders completely driverless rides in parts of Phoenix, which is pretty exciting.

This is a nuanced challenge, and you can fool yourself if you use any one method or statistic to get confidence of the safety of the system. So we use a combination of methods that lets us build a fuller picture… and we continue to improve and add to those methods. At a high level they includes approaches like:

• driving millions of miles on real public roads and reviewing disengages and other events
• simulating software over billions of miles, and against an extensive set of challenging scenarios, including a lot that we've never seen in real life, but could happen!
• real-world testing at our closed course. That’s because there are lots of rare situations that we want to test for but that don’t occur enough in the real-world to get meaningful data. At our private test track we can create situations and measure our technology performance against those scenarios.
• rigorous design principles (redundant actuation, fallback systems), safety-based engineering (hazard analysis, DFMEAs), and extensively validating that our system reacts as expected.
• We actually cover some of this in the Waymo safety report waymo.com/safety

- Nathaniel & Miz

Great topic for an AMA. Waymo is famous for using deep neural nets at the perception of your cars. Do you also use any kind of machine learning techniques on the behaviour level as well? If so, how do you guarantee the car will never do anything bad?

Thanks for the question. We use deep learning across our stack: perception, prediction, etc. In behavior, specifically, we use ML for predicting what other agents will do, or how they will react to us, or what is a natural baseline to drive down a narrow road. But your question is spot on: the key requirement when we incorporate ML is to make sure that it works within a framework where we can leverage the strengths of ML but combine it with our engineered solutions and rigorously evaluate the safety of the output. - Nathaniel

What is the current cost of all of self driving hardware that goes into one vehicle with the 4th Gen platform? What is the current cost of all of self driving hardware that goes into one vehicle with the 5th Gen platform?

Aside from the cost of the vehicle, how long until you can get the cost of all of the self driving hardware (sensors / compute) under $10,000 per vehicle?

We’ve significantly reduced the price of our hardware suite, which my team designs in-house. The latest generation of our hardware suite is more than 50% cheaper than our last generation. And our next gen lidar delivers more than an order of magnitude improvement in cost and functionality. - Satish

What exactly do you mean by 'driverless'?

Could you confirm that there are no humans that monitor the car, either locally or remotely, and are able to brake or steer the car in case of emergency?

Thanks for the question! There is a lot of confusion around this term -- it's used pretty broadly in the industry. When we use “driverless” we mean there are no other humans in the car (besides our riders, of course!).

Our entire fleet is connected to our operations center, which is staffed by a fleet response team who can monitor the vehicles and give the vehicles contextual information as needed but they can’t brake or steer the car. The vehicle itself handles braking and steering at all times. - Nathaniel

What about a natural disaster? What can be done in case of flooding or forest fires? Will your cars just stop?

WRT natural disasters: Of course, natural disasters can encompass a lot of different situations. In general, our cars detect water, smoke, dust, etc., and slow down or stop if that’s the right thing to do. Second, our ops team constantly monitors local conditions, and can request that all vehicles return to the depot, or pull over immediately, or reroute the cars, depending on the situation.

There isn't a remote emergency *brake*, but there is the ability to request the vehicles to pull over ASAP.

• I've heard that the Waymo driver code started off being based on the Stanford Car code. Is that true?

• Deep learning became popular several years after Google started the SDC project. How has this affected the approaches used? Was the project leveraging ML/DL significantly before, or have breakthroughs been incorporated over time?

• How do you deal with ingesting the sheer amount of data generated by an ever growing fleet? What sort of engineering challenges stem from this?

• What sort of redundancies have to exist for an L4 vehicle to run with 0 humans in it?

• How do you deploy vehicle software? Is it containerized, or are vehicle imaged?

• What sort of bring up / bring down problems have to be solved to turn a great AI driver into a functioning robotaxi service that can run unattended?

• Why was Firefly killed, especially after prototypes were already on the road?

Deep learning became popular several years after Google started the SDC project. How has this affected the approaches used? Was the project leveraging ML/DL significantly before, or have breakthroughs been incorporated over time?

We greatly benefited from advances in deep learning and have been investing in the area for years, starting from the earliest days of the field around 2011.

Our first applications of deep learning were in perception and allowed us to greatly improve the accuracy of our object detection/classification. Since then, we’ve expanded our use of deep learning to other parts of the stack beyond perception, and currently use it in prediction, planning, mapping, and simulation. We continue to push on the leading edge of ML research, and we've also got great collaborations with other ML groups (like Brain and Deep Mind). - Nathaniel

What sort of redundancies have to exist for an L4 vehicle to run with 0 humans in it?

We build a number of redundancies into our vehicles, including power, braking/steering, communication/connectivity, sensors/compute, and onboard software stack. - Satish

What are the top 3 tech challenges you’re working on right now?

From a hardware perspective, the top challenge we’re working on is ensuring our hardware sensor suite can perform optimally in all types of weather. We’re testing our vehicles in all types of weather to continuously improve our hardware stack, including severe rain in Florida, snow in the Upper Peninsula of Michigan, and haboobs (dust storms) in Arizona. We’re excited about the improved weather capabilities of the fifth generation of our hardware suite, which we’re putting the finishing touches on now. - Satish

How are you dependable on your maps? Are your cars able to drive safely without any map?

Our cars rely on maps, but we designed our system so we can navigate safely where the roads have changed.

Maps are useful to let our vehicle know where it is, as well as to help it anticipate what is coming up next. e.g. it’s useful for our vehicles to know there’s a stop sign coming around the corner before we visually detect it.

We build our own high definition 3D maps. While driving, our vehicles benefit from sharing new information with the rest of the fleet dynamically -- for example if there is a construction zone or some other new/temporary object or situation. Maps, combined with sensing, gives us the best understanding of the world. - Miz

How long do you think it will be, realistically, before a fully self-driving car is available on the market?

Think you’re asking about personally owned vehicles, is that right?

Right now, our top focus = ride-hailing and long-haul trucking, along with commercial b2b deliveries. For personal car ownership, we’re actively talking with our automaker partners to offer their customers personally owned vehicles with the Waymo Driver. Stay tuned for more on that...we've got a cool business approach in mind. - Satish

What would be the best form factor change to an automobile in terms of making it much easier to develop autonomous driving that you wish you could popularize, but can't because consumers would think it is too ugly?

Hmmm….

Any plans to use sensors that are not attached to the vehicle? Say if a you know a certain place has a visibility problem in a mapped area, you add a camera that all Waymo cars have access to so they can see around the corner or tall building? Or using quadcopters to route traffic around accidents and emergencies in real-time?

So far in our testing we have not come across a need for a sensor that is external to one of our vehicles. - Satish

How do you deal with really dark objects, stuff that cameras or Lidars can’t see at close distance, like shiny black cars?

Our lidars are capable of seeing really dark objects, even things like a tire on a freeway at night, at a range where we can react safely. For example, shiny black cars have a number of features that reflect plenty of light back to our lidars. Our radars are also very good at detecting vehicles, regardless of their color. Our system is designed to detect all sorts of road users, like cars, people, cyclists, motorcycles, etc. - Satish

Are Waymo vehicles aware of each other? Do they communicate? Are there plans to do this?

Yes, our Waymo vehicles share information in real time. This is particularly useful when one of our vehicles encounters something unexpected in its environment -- for example, a construction zone. - Nathaniel

Hi! All of you have impressive and unique histories which undoubtedly fit your current work. What would you say to others fascinated with the AV space and wanting to get involved but without an engineering pedigree? What in-demand roles do you see for non-engineering folks? What about in the ecosystem of AV - municipalities, suppliers, etc. Are there trends or counter intuitive market need you're seeing where someone could get involved?

Thank you so much for this question! We do have a super talented team and we’re growing across all areas including non-eng business units.

On top of that, our Eng team is expanding -- hardware, onboard software (including perception, prediction, and planning), simulation, and other areas. I hope you’ll keep an eye on our open roles at waymo.com/joinus - Satish

• Does your in-car compute hardware consist of TPU/Edge TPU?

• Do you currently perform any 'Waymo Engineering', 'Experimental' or more advanced future software tests that consists of driving with no Lidar-type HD mapping in a city?

  Thanks again.

This is one of the best parts of working here! We explore all kinds of ideas: new sensors, snow, "less map", exotic stereo camera configurations, synthetic aperture radar, golf carts. - Nathaniel

We have designed our computer architecture with flexibility so the exact mix of silicon we use changes. Today we use a combination of CPUs, GPUs, accelerators and IO processing engines. - Satish

Satish, what are the best camera options available on the market now for self-driving vehicles? IMO It's kind of ridiculous that any GMSL camera solutions on the market (e.g. Leopard) are $400 per camera + $1000 for a daughter board of some sort while the equivalent USB 3.0 cameras (same sensors!) cost <$75. Considering it's basically the same USB camera minus the USB hardware and just some serdes chips one would think that GMSL should be cheaper than USB for the same sensors, but it isn't.

I understand economies of scale, but with the number of self-driving cars out there there should be some scaling already.

The cameras that are part of our hardware sensor suite for our vehicles actually are designed in-house here at Waymo. I haven’t seen any off-the-shelf camera that would provide the same level of performance, low cost, and latency. - Satish

What are your abilities to swerve? Do you think your cars are able to avoid a potential head-on collision?

Our cars have discretion to swerve. Our basic approach is to think through the possibilities ahead of time and choose the behavior that is most safe in the most contexts. In some cases, that is swerving to avoid oncoming collisions (and we have extensive closed-course tests to validate our behavior). - Nathaniel

Thank you for your AMA. Do you consider adding an additional type of a sensor? If yes, what will it be?

Yes! In the fifth generation of our in-house designed hardware sensor suite, we will be introducing a new sensor modality. - Satish

Hi Nathaniel, I believe predicting the behavior of other road users is very challenging. Can you say anything about which kind of techniques do you use at waymo to make sure your predictions are correct? How do you make sure that waymo will not cause accidents if some of those predictions are wrong?

Thanks for the question, Mario! Two really key parts of the answer. First, we have a huge amount of real world data that we can measure our prediction performance. Second, we don't just predict one possibility! We predict all the potential actions of the other agents (together with the likelihood of each action). The planner then considers all these possibilities when coming up with a safe plan. - Nathaniel