Right. As I've pointed out previously, Tesla seems to be unable to detect sizable stationary obstacles that are partly blocking a lane, especially if they don't look like the rear end of a car. In addition to emergency vehicles, Teslas on autopilot have plowed into freeway barriers and a street sweeper. That's the usual situation for first responders, who usually try to block as little of the road as possible but often don't have enough shoulder space.
It's clear what Tesla really has - a good lane follower and cruise control that slows down for cars ahead. That's a level 2 system. That's useful, but, despite all the hype about "full self driving", it seems that's all they've got.
"Full self driving" just adds some lane-changing assistance and hints from the nav system.
I feel like part of the problem with the kind of autopilot crashes you describe here is how inexplicable they are to humans.
Whilst humans can be dangerous drivers, the incidents they cause generally have a narrative sequence of events that are comprehensible to us -- for instance, driver was distracted, or visibility was poor.
But when a supposedly 'all-seeing always watching' autopilot drives straight into a large stationary object in clear daylight, we have no understanding of how the situation occurred.
This I think has a couple of effects:
1) The apparent randomness makes the idea of these crashes a lot more scary -- psychologically we seem to have a greater aversion to danger we can't predict, and we can't tell ourselves the 'ah but that wouldn't happen to me' story.
2) Predictability of road incidents actually is a relevant piece of information. As a road user (including pedestrian), most of my actions are taken on the basis of what I am expecting to happen next, and my model for this is how humans drive (and walk). Automated drivers have different characteristics and failure modes, and that makes them an interaction problem for me.
In my opinion the underlying assumption autopilots are built with are wrong.
It is assumed that the road is free to drive on.
Only when the vehicle computer detects a known object on the road that it knows should not be there it is applying brakes or trying to steer around.
I would feel safer if the algorithm would assume the negative case as default and only give the „green light“ once it determined that the road is free to drive on.
In case of unknown (not yet supervised) road obstructions the worst needs to be assumed.
That’s where the ‚unexplainable‘ crashes are coming from. Something the size of an actual truck is obstructing the road. But couldn’t quite classify it because the truck has tipped over and is lying on the road sideways. Not yet learned by the algorithm. Can't be that bad, green light, no need to avoid or brake.
> Only when the vehicle computer detects a known object on the road that it knows should not be there it is applying brakes or trying to steer around.
The problem with Tesla's "No LIDAR ever, cameras are good enough" approach is that it fails to detect emergency vehicles: they filter out stationary items out of radar signal as noise[1],and Tesla's ML models probably can't reliably identify oblique vehicles and semi trailers as obstacles.
1. Makes sense in isolation: frequent radar returns from roadside and overhead signs would be a pain to deal with
The stated reason is "your eyes dont shoot lasers, so a camera is good enough". But the implied reason is cost for sure. With how fast the price of lidar drops, and its abilities increase (think solid state lidar), I wonder how long until first tesla with lidar rolls down the production line, or if Elon is too proud to ever allow that
> It is assumed that the road is free to drive on.
Trying to remember if the opposite of this is how human drivers are taught, or if this is implicit in how we move about the world. My initial gut reaction says yes and this is a great phrasing of something that was always bothering me about automated driving.
Perhaps we should model our autopilots after horses: refusal to move against anything unfamiliar, and biased towards going back home on familiar routes.
In my high school’s Drivers Ed class I distinctly remember the one-question pop quiz: “What is the most dangerous mile of road?”
The answer was “the mile in front of you”
Additionally there was some statistic about the frequency of accidents within a very short distance of the drivers residence, which seemed to underscore the importance of being aware of just how much your brain filters out the “familiar” in contrast to a newly stimulating environment.
I had always assumed the "close to home" numbers were just bad statistics, because I never saw them control for % of driving that was done "close to home".
If I google it, I get like three pages of law firms.
In my opinion the underlying assumption autopilots are built with are wrong. It is assumed that the road is free to drive on. Only when the vehicle computer detects a known object on the road that it knows should not be there it is applying brakes or trying to steer around. I would feel safer if the algorithm would assume the negative case as default and only give the „green light“ once it determined that the road is free to drive on.
I agree, but it will up the false alarm rate in a system without good depth perception for all objects. This is tough with cameras only. Reflective puddles are a problem; they're hard to range with vision only.
Anything that doesn't range well, which is most very uniform surfaces, becomes a reason to slow down. As you get closer, the sensor data gets better and you can usually decide it's safe to proceed.
Off-road autonomous vehicles have to work that way, but on-road ones can be more optimistic.
Waymo takes a hard line on this, and their vehicles drive rather conservatively as a result. They do have false-alarm problems and slowdowns around trouble spots.
Would you rather optimize for a faster overall fleet, or a fleet with stress free driving, no incidents, no need to intervene or be to be worried.
If the system gets faster over time, even better. But I cannot imagine huge adoption unless the system gets actually reliable.
I am pretty much in favor of the Waymo approach.
I agree. In the north east at least pothole avoidance is a critically important skill. Any "autopilot" without it would be fairly useless around me as I'd have to take over every 30 seconds to not end up with a flat tire. I have adaptive cruse control and that's about as far as I'll trust a computer to drive given the current tech.
My problem with those crashes is that they are entirely explicable: The car is blind to stationary objects in the road. (My best guess at the logic is they assume that "anything stationary cannot possibly be in the road, right?")
To me, that blindness is simply unacceptable. If there is anything in the road, whether identified or not, it should automatically be flagged as a hazard. That flag should only be removed if it is detected to be moving in a way such that it will be somewhere else when you get there.
I have Subaru EyeSight. It has no problem seeing stationary objects. What's Tesla's problem?
I’m not sure about newer models without radar, but the older ones explicitly discard stationary returns on their radar. As I understand it, without elevation data it can’t know if it’s a bridge you’ll pass under, a soda can in the road, or a stopped car - so just ignore it all.
Of course the vision system is supposed to compensate for this, and it performs poorly on objects it doesn’t see often, like emergency vehicles.
The vision system is supposed to be able to determine an accurate depth map based on a combination of stereo vision and depth-from-defocus. I've seen demos of the real-time depth map, and it looks high-resolution and accurate to about 5-10cm.
So, if they have the input data, why is it being ignored by autopilot?
why should it matter how often it sees something? Or even if it's something the car has never seen before? All it should care about is whether there is an obstacle, not what the obstacle is. Whether it's an emergency vehicle, a sofa, a boulder, a canoe, a table saw, or a dolphin, you don't want to hit it!
How often it’s seen in training data that is, which is pulled from data in the wild.
It’s simply not possible to do depth estimation like this without priors. That’s one of the serious limitations of such systems - you have to train on every class of object you don’t want to hit.
Then they are doing it wrong. There are all manner of things that can end up in the road that have never been (and will never be) classified. If their system must classify a thing to not hit the thing, then they will kill people. It's gross negligence to work so hard to not, at the very minimum, install two cameras for stereo vision.
Another aspect of unpredictability is that drivers are expected to be alert and vigilant while using ADAS features, but I get the impression that Tesla's implementation sometimes does things that are completely unexpected. Sometimes you might have to react immediately to something you didn't see coming, because you didn't expect the car to suddenly try to steer into a concrete pillar or something.
It's one thing to have to deal with inexplicable behavior from other cars, but to have to deal with inexplicable behavior from your own car seems quite a bit more unnerving.
I think the problem we're seeing here is that Tesla's autopilot system is on the cusp of a fully automated driving experience and that feels good enough to the driver. Yet it's not quite good enough, as we can see from the mistakes it has made.
Honestly, I see this as a necessary transition pain towards fully automated vehicles. No matter how you slice it there's going to be periods where fully automated driving systems aren't quite there yet but are good enough 97% of the time that human drivers let their guard down. It's going to take some sacrifices to get to fully autonomous driving.
The good news is that even with these accidents self-driving features are a bazillion times safer than human drivers. It sure seems like the occasional vehicle collision into stationary objects is going to throw a great big wrench into self-driving safety statistics but it isn't even a rounding error compared to the sheer number of accidents caused by human drivers.
> I feel like part of the problem with the kind of autopilot crashes you describe here is how inexplicable they are to humans.
I don't see why these are inexplicable to humans. It's certainly no more difficult to explain than, say, a (non-adaptive) cruise control in a car from 2000 doing the same thing.
> Whilst humans can be dangerous drivers, the incidents they cause generally have a narrative sequence of events that are comprehensible to us -- for instance, driver was distracted, or visibility was poor.
But that is arguably a sufficient explanation for these Tesla crashes as well. The driver being distracted or inattentive or unable to see clearly is a requirement for all of these Tesla crashes, as far as I know.
Perhaps 'unintuitive' is a better word to convey what I mean -- as in, there isn't an easily understandable (non-technical) narrative chain of events, there's just 'opaque box malfunctioned'. The cruise-control example you give feels a bit different, as CC doesn't claim to include automated collision avoidance, whereas something labelled 'autopilot' does.
It's perfectly explainable. You have a blind machine with an imperfect sensorium trying to describe an elephant. Correct identification is just getting lucky. The layers of ML improve the odds but can never achieve 100%. The whole scheme is playing dice with other people's safety.
"""
These events occur typically when a vehicle is partially in a lane and radar has to ignore a stationary object. This is pretty standard and inherent with TACC + radar.
The faster Tesla pushes the vision only stack to all cars after they’ve validated the data, the faster this topic becomes moot. Andrej Karpathy talks and shows examples of what that would do here. Minutes 23:00-28:00
https://youtu.be/a510m7s_SVI
Older examples from manuals of other TACC systems which use radar:
Volvo’s Pilot Assist regarding AEB/TACC.
According to Wired, Volvo’s Pilot Assist system is much the same. The vehicles’ manual explains that not only will the car fail to brake for a sudden stationary object, it may actually race toward it to regain its set speed:
“Pilot Assist will ignore the stationary vehicle and instead accelerate to the stored speed. The driver must then intervene and apply the brakes.”
Cadillac Super Cruise - Page 252
Stationary or Very Slow-Moving Objects
ACC may not detect and react to stopped or slow-moving vehicles ahead of you. For example, the system may not brake for a vehicle it has never detected moving. This can occur in stop-and-go traffic or when a vehicle suddenly appears due to a vehicle ahead changing lanes. Your vehicle may not stop and could cause a crash. Use caution when using ACC. Your complete attention is always required while driving and you should be ready to take action and apply the brakes.
BMW Driving Assistant Plus - Page 124
A warning may not be issued when approaching a stationary or very slow-moving obstacle. You must react yourself; otherwise, there is the danger of an accident occurring.
If a vehicle ahead of you unexpectedly moves into another lane from behind a stopped vehicle, you yourself must react, as the system does not react to stopped vehicles.
"""
I’ll believe it when I see it. From what I can tell, Tesla has made no progress at all in three years. I just drove my buddy’s 3, and it was still diving to the right when a lane merges and the line disappears. This drove me nuts when I test drove years ago. Other cars do lane keeping so much better than Tesla at this point.
The problem with radar on the ground is that most of what comes to a radar detector is reflections from a stationary world, with relative delays so small as to be undetectable. So the first step in processing is to filter out everything at the speed of that motionary world. All fixed objects therefore disappear, and you are left sorting out moving objects. Which means you now can't detect stationary objects at all.
Tesla has a different problem. They probably don't have depth perception. They therefore have to classify objects, and make educated guesses about where they are relative to the car. Unexpected kinds of objects, or objects in unexpected configurations, fail to be classified and therefore fail to be analyzed.
In principle, Tesla can succeed. After all we don't have binocular vision past 6 meters either. Tesla is improving.
FWIW: Tesla AP is primarily vision based now. Newer cars in the US aren't even being fitted out with the radar units anymore (mine doesn't have it, for instance). So while this may in some sense be an unavoidable edge case for radar, it really shouldn't be for Tesla Autopilot.
It's worth checking out for sure. Not worth the headline bandwidth and flamage budget being spent on it.
Subaru's eyesight absolutely will stop you when you think you're about to hit something, regardless of whether or not that something was moving previously.
It's actually really annoying if you live in a rural area without clearly defined lanes, and large, stationary objects (tractors and whatnot) close to the road.
As I think I previously posted about this. It will also see exhaust coming up on cold winter day as an obstacle and brake unexpectedly at light. It literally is worst and wish it could be disabled by default.
Additionally, the back up sensor is a tad over zealous also.
I think we need to add a new level 2 assisted driving skills section to driving tests. Level 2 can be safer but it really requires understanding how level 2 works and limitations. For example, when I use level 2 (mine is Honda but applies to other level 2 as well since they mostly share the same vendor), these are the rules I follow:
- Car switching in/out of my lane, I manually take over
- Tight curve in the freeway, manually take over
- Very frequently check the dashboard indicator that shows if the sensors "sees" the car front or not
- Anything unusual like construction, cones, car on shoulder, manually take over
- Anything that looks difficult like weird merging lanes, manually take over
- Any bad weather or condition like sun directly in front, manual drive
- Frequently adjusting max speed setting on ACC. It's safer to not be too much above the prevailing speeds. Otherwise, if ACC suddenly becomes blind, it can accelerate dangerously as it tries to reach max set speed.
- I don't trust lane keep much, it's mostly a backup for my own steering and making my arms less tired turning the wheel
The key thing is to recognize just how dumb this technology is. It's not smart, it's not AI. It's just a bit above the old cruise control. With that mindset it can be used safely.
If level 2 requires such handholding, what’s the point? Seems to me like it just leads to a false sense of security, giving drivers the feeling that they can trust the self-driving system a lot more than they safely can.
It's basically ultra advanced cruise control that can also handle stop and go traffic on a well marked road. That frees the driver up to dedicate more of the situational awareness budget to other things.
If you use it that way it's fine. If you expect it to be as smart as a student driver it's not fine.
I'd rather at least get the pleasure of driving, rather than basically becoming the supervisor for my car. Quartlery performance reviews, checking on KPI's.
Autopilot and the like are absolutely not on my list of features I'm looking for when buying a new car. Crusie-control? Handy. AP, waste of (my) time.
The point is that it makes driving on the highway easier, just like automatic transmissions, power steering, etc. I'm sure early critics of all of these technologies have said things like "but if the driver doesn't need to spend mental and physical energy to that this they will stop paying attention to the road and will be less safe."
I think that requires more numerate analysis than you're giving though. The data from the story is a sample size of 11 crashes over three years (I think). If that's really the size of the effect, then your "but [...]" clause seems very suspect.
There are almost two million of these cars on the roads now.
It seems extremely likely that the number of accidents prevented by AP dwarfs this effect, so arguing against it even by implication as you do here seems likely to be doing more harm than good.
That doesn't mean it's not worth investigating what seems like an identifiable edge case in the AP obstacle detection.
But that's a bug fix, not an argument about "Level 2 Autonomy" in general.
I think it's early to be adding stuff like that to government mandated driving tests when these cars are only theoretically available to the ever dwindling middle class and above. Unless my circumstances change there's no chance I'll be in one for at least 10-15 years.
I wonder if some sort of standard display showing what the vehicle "sees" and is predicted to do will be regulated. For example, if the display shows the vehicle doesn't understand a firetruck parked half way in the lane or the tight curve on the freeway, at least the driver can validate on the display and have some time to react.
Elon keeps warning the world about the impending hyperintelligent AI that will retool human economies, politics, and religions, yet year after year his cobbled-together AI fails at basic object detection.
AI is a bogus term. It was well-discredited in the 1980s and has gone through an ill-deserved rehabilitation. Deep neural nets are a step forward for some tasks like vision, but are still just scratching the surface of what really goes on in our brains when sense-making from visual data.
I'm split over this. On one hand The AI is so incompetent that it could never take over the world and enslave humanity. On the other, its so incompetent that it winds up killing humans anyway. lose-lose.
> That's the usual situation for first responders, who usually try to block as little of the road as possible but often don't have enough shoulder space.
Actually, from talking with friends who are first responders, many times they will park fire trucks, etc so that they are blocking enough of the road to protect the first responders and the victims. The last thing you want is to have another car come and crash into first responders or victims of the initial accident. That is why they will deliberately park the truck at an angle to protect the people.
Exactly this. We'll place the heaviest apparatus 'upstream' of the event and in a way that 'encourages' cars away from the incident.
Larger departments or those dealing with busier freeways have even started re-purposing older engines with water ballasts and attenuators as 'blocker' engines.
I don't understand how its even possible for these cars to be crashing. My car beeps like a missile is locked on when I am coming too close to an object I might hit. Just a simple sensor in the front. If my car can beep and give me enough time to slam on the brakes, why can't Tesla's do the same?
You're probably referring to parking sensors, which are typically ultrasonic sensors mounted to the bumpers. Unfortunately, ultrasonics have both a limited range of ~20ft for practical uses, and more damningly, a relatively wide field of view with no ability to distinguish where in the field of view an object is. While 20ft range is more than enough to give you time to slam on the brakes in your garage, it's basically useless for high speed autonomy, except for some very limited blindspot-awareness type tasks.
It's not a parking sensor and there isn't anything on the bumper as far as i can tell. It's toyotas precollision system that uses a simple radar and a cheap camera. It even brakes for you if you don't.
In other parts of this thread, people are suggesting that these crashes are the radar's fault and deploying their vision-only system will fix the problem.
I don’t see how you can blame a radar for your vision system not being able to classify a fire truck (not to mention the unforgivable act of ignoring objects it can’t classify)
As the sibling post says, that’s ridiculous at face value. All the radar is check for collisions because that’s how a radar works...is there a non-intuitive answer here???
Tesla does not offer a vehicle that does not require human drivers (not even momentarily). Tesla's autopilot and FSD systems are both SAE Level 2, which means the human is still in operation of the vehicle at all times. All of Tesla's driving assistance technologies require a human to monitor operation of the vehicle and intervene if necessary. The fact that they have given anyone an impression otherwise is problematic.
> The fact that they have given anyone an impression otherwise is problematic.
Good grief. This meme will not die. The car literally tells you to keep your hands on the wheel every time you engage autopilot, yells at you if you don't, will lock you out of the system as punishment if you don't comply, and if you really seem disabled will bring the car to a stop and turn the hazards on. It simply will not operate without an attentive driver, or at the very least one spending considerable energy at defeating the attention nags.
There are exactly zero Tesla drivers in the world who don't know these rules. Just stop with the nonsense. Please.
> There are exactly zero Tesla drivers in the world who don't know these rules. Just stop with the nonsense. Please.
Tesla's marketing also knows that there are exactly zero drivers in the world who follow those rules, but that doesn't stop them from overselling the capabilities of what they ship.
Stop it. Please. Again, there are no Tesla drivers who have been misled about the capabilities of the system. The people who have been mislead are folks like you who read arguments on the internet and don't drive these cars. Go try one and see how the system works. It doesn't permit the kind of confusion that everyone constantly assumes. It just doesn't.
There are many instances of people defeating the lockout system. Social media is full of these types of demonstrations. Plenty have the attitude that it is okay to do this. Some have died while showing it off.
"Defeating the lockout system" wasn't the discussion at hand. The contention upthread is that Tesla drivers did not know that they needed drive the car.
And there are Tesla materials that say, word for word, "The driver is only there for legal purposes. The car is driving itself."
It only yells at you now because Tesla had to be forced to make it do so. Previously it'd let you go for a quarter of an hour before checking in on you.
Because the effectiveness of those warnings are diminished by mixed messaging and peer pressure from non-owner passengers.
There are plenty of people who have been convinced that those safety features/warnings are “just there for lawyers” and have attached items to the wheel to defeat the safety lockouts in order to show off their “self driving car” to their friends.
No one is saying this should be the case, just that the feature is not what the company advertises (the ability for the car to fully drive itself) and that said feature is further away from completion than many might lead you to believe. as someone who drives a tesla with autopilot, I agree with this. Autopilot is the best lane assistance system ive used, but thats all it is.
Less than 100 first responders are hit annually in the USA. The fact that just Tesla has managed to hit 11 since 2018 makes it pretty clear that human drivers are not "10x worse" than Tesla's tech, but quite the opposite.
I was curious and went looking for corroboration / refutation of the 10x you stated. Tesla and Musk both have cited 10x safer (but I didn't see any independent confirmation). But, there was a critical analysis by Forbes that came to a different conclusion. Per Forbes, "[with autopilot] it looks like a Tesla is slightly less safe. But not a lot less safe."
the point isn't to ignore it but to make sure everyone using it is 100% aware they need to be paying attention as this is a edge case that is common that the system cant handle so drivers are still responsible to take over
It's clear what Tesla really has - a good lane follower and cruise control that slows down for cars ahead. That's a level 2 system. That's useful, but, despite all the hype about "full self driving", it seems that's all they've got.
"Full self driving" just adds some lane-changing assistance and hints from the nav system.