If you haven’t heard, Tesla CEO Elon Musk is not a LiDAR fan. Most companies working on autonomous vehicles – including Ford, GM Cruise, Uber and Waymo – think LiDAR is an essential part of the sensor suite. But not Tesla. Its vehicles don’t have LiDAR and rely on radar, GPS, maps and other cameras and sensors.
“LiDAR is a fool’s errand,” Musk said at Tesla’s recent Autonomy Day. “Anyone relying on LiDAR is doomed. Doomed! [They are] expensive sensors that are unnecessary. It’s like having a whole bunch of expensive appendices. Like, one appendix is bad, well now you have a whole bunch of them, it’s ridiculous, you’ll see.”
“LiDAR is lame,” Musk added. “They’re gonna dump LiDAR, mark my words. That’s my prediction.”
While not as anti-LiDAR as Musk, it appears researchers at Cornell University agree with his LiDAR-less approach. Using two inexpensive cameras on either side of a vehicle’s windshield, Cornell researchers have discovered they can detect objects with nearly LiDAR’s accuracy and at a fraction of the cost.
The researchers found that analyzing the captured images from a bird’s-eye view, rather than the more traditional frontal view, more than tripled their accuracy, making stereo camera a viable and low-cost alternative to LiDAR.
Tesla’s Sr. Director of AI Andrej Karpathy outlined a nearly identical strategy during Autonomy Day.
“The common belief is that you couldn’t make self-driving cars without LiDARs,” said Kilian Weinberger, associate professor of computer science at Cornell and senior author of the paper Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving. “We’ve shown, at least in principle, that it’s possible.”
LiDAR uses lasers to create 3D point maps of their surroundings, measuring objects’ distance via the speed of light. Stereo cameras rely on two perspectives to establish depth. But critics say their accuracy in object detection is too low. However, the Cornell researchers are saying the data they captured from stereo cameras was nearly as precise as LiDAR. The gap in accuracy emerged when the stereo cameras’ data was being analyzed, they say.
“When you have camera images, it’s so, so, so tempting to look at the frontal view, because that’s what the camera sees,” Weinberger says. “But there also lies the problem, because if you see objects from the front then the way they’re processed actually deforms them, and you blur objects into the background and deform their shapes.”
Cornell researchers compare AVOD with LiDAR, pseudo-LiDAR, and frontal-view (stereo). Ground- truth boxes are in red, predicted boxes in green; the observer in the pseudo-LiDAR plots (bottom row) is on the very left side looking to the right. The frontal-view approach (right) even miscalculates the depths of nearby objects and misses far-away objects entirely.
For most self-driving cars, the data captured by cameras or sensors is analyzed using convolutional neural networks (CNNs). The Cornell researchers say CNNs are very good at identifying objects in standard color photographs, but they can distort the 3D information if it’s represented from the front. Again, when Cornell researchers switched the representation from a frontal perspective to a bird’s-eye view, the accuracy more than tripled.
“There is a tendency in current practice to feed the data as-is to complex machine learning algorithms under the assumption that these algorithms can always extract the relevant information,” said co-author Bharath Hariharan, assistant professor of computer science. “Our results suggest that this is not necessarily true, and that we should give some thought to how the data is represented.”
“The self-driving car industry has been reluctant to move away from LiDAR, even with the high costs, given its excellent range accuracy – which is essential for safety around the car,” said Mark Campbell, the John A. Mellowes ’60 Professor and S.C. Thomas Sze Director of the Sibley School of Mechanical and Aerospace Engineering and a co-author of the paper. “The dramatic improvement of range detection and accuracy, with the bird’s-eye representation of camera data, has the potential to revolutionize the industry.”
Lidar requires moving parts, does not scale, is expensive and fragile. No one that is familiar with alternatives is going to like Lidar.
All accurate points, and I would also add that LIDAR is incapable of operating in the rain or snow, as the light is scattered and absorbed by water molecules. RADAR does not have this issue as it operates at more energetic wavelengths.
Some of the more advanced LIDARs are not Velodyne bulky rotating jokes, they fully, or almost fully solid state, much smaller, with costs going down drastically to 000’s of USD per unit over the next ~3 years. Look at the Ouster or Innoviz newest models and see where this is going.
And lidar doesn’t see attitude, position, and therefore has less potential in anticipating in which direction an object is gonna move next
No,LIDAR Can be used for state Localization. Also attitude, position is not what cameras give, we use it with GNSS and IMU to get location
The bird’s-eye view shows better accuracy due to the nature of stereo-photogrammetric calculations and mathematical modelling. I did this study back in 2005 as a PhD.
Look here:
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.362.9698&rep=rep1&type=pdf
Check Figure 6-15 page 118
The Tesla technology does not have the ability to adjust to changing environments such as a set of traffic cones temporarily used to redirect traffic
@Anne, I believe this article may interest you. Tesla cars do actively detect and adjust to traffic cones.
https://www.engadget.com/2019/11/03/tesla-autopilot-traffic-cones/
Yes they definitely can see traffic cones, garbage cans etc due to its learning capabilities. More cars on the road the more new things it can recognize. Thats the beauty of the vision approach.
Dear Anne, pls check out
https://youtu.be/gWKK9A88O04
https://youtu.be/ztym9rOEtZ0
Tesla keeps fooling people with its so-called Auto-Pilot, which is nothing better than other companies’ adaptive cruise control, e.g. Super Cruise of GM. While most of the Automotive companies are very cautious about self-driving cars, Tesla claims they already have the Autonomous car even without LIDAR!
I’m an AI expert (https://www.linkedin.com/in/glenn-jocher), and I can tell you that Tesla is far ahead of any other company by virtue of the massive advantage in training data collection capabilities. 1M cars on the road observing and learning puts them light years ahead of their rivals.
If you were aware, my 10 year old Mercedes has the same learning data capabalities and in addition to multiple front cameras and multiple front radars and a lidar, not to mention also infrared projectors and cameras.
And radars in the rear.
They have collected much more data over the last 16 years or so as they had cameras in their cars.
All of that is used in their current vehicles which are capable of learning as well. Not to mention they backed out saying this is not feasible
I agree with Glenn Jocher. I have had the pleasure of being a passenger in one of the new Tesla vehicles. It is far more advanced than your basic cruise control. Once set to a certain speed the car will maintain that speed plus adjust to the condition of the road and the traffic around you veer left,veer right,slow down,speed up. Its basically Auto-pilot with minor disadvantages. It doesn’t self park yet. That package is soon to come, I’d imagine once all that data is researched and put to use. Passanger vehicles w/ Combustion engines should be banned.
Your first 4 sentences describe what a 2005 Mercedes does, and every economy car on the road today does as well.
And all of them park, and some could have parked automatically since the 2005.
Not sure how the method of propulsion is related to this. But do not worry, it will take more than a century from today, year 2021 to have any sensible systems
I still think within 6 months someone will come out with a Lidar chip cheap enough to be installed in a vehicle for under 45k. Be it Veoneer, APTIV, Wayne whoever…They have to hit the $2000-$2500 price point. Also supposedly someone is working on smart road sensors that will give LiDAR and extra advantage. Seems like a lot of Tesla fan boys here..
You actually need 3 “cheap” lidar sensors and 1 expensive lidar sensor. Have you seen the latest Waymo Jaguar Ipace? So its going to be stuck with 100k+ cars for the next decade, unless they go down the hybrid approach with more cameras and less Lidar as a crutch
If Musk changed his mind and said LiDAR is the future then every single person that rails against LiDAR would change their tune too. The argument is not about science. It’s tribalism. Waymo uses LiDAR but it also uses cameras, radar, and microphones. And Cornell isn’t even claiming cameras are better – just that the are almost as good in most conditions. The “addition” of LiDAR is undoubtedly better. The question then is cost. Waymo is getting into the LiDAR selling business and has brought the cost down from $75k to $7,500. As the cost comes down and the device becomes more streamlined it might see widespread adoption. I don’t know the future. And neither does Musk. But right now, the technology is at least marginally better than his best effort and when we are talking about a kid running into the street to grab a ball I’d rather vehicles have the marginally better technology. Regulators may agree. And again, costs are coming down so his arguments about the cost are becoming less valid and less relevant. And yes, Telsa has a major advantage in real world conditions. But only with cameras and they have no data on LiDAR so they are not light-years ahead of the competition. Not Waymo. And the fact that some people on here think that Tesla’s autopilot is more than just branding and that other high end apaptive cruise control isn’t the same… is scary. Don’t treat it like autopilot! I like Tesla. I like electric cars. But I don’t like how science is turning into politics. Let LiDAR technology develop and if it works great. No one should be hoping it fails. Other than maybe people that have a financial interest in Tesla and don’t value human life. Also, I don’t like that Tesla isn’t working harder to try to open its charging network to non-Tesla. Even charging a premium would be fine but don’t intentionally stifle EV adoption. Tesla has done the impossible and that’s great but it doesn’t mean Musk doesn’t make mistakes. Look no further than his plan to automate his factories. It failed. But to his credit, he changed positions and admitted his mistake. If LiDAR comes down to sub-$2,000 I’d be surprised if he doesn’t adopt it. And while I’m on an epic rant, Tesla’s valuation makes no sense to me. I have always thought Tesla would be very successful so it is not that I’m betting against them. It’s just that if 10 years from now we have self-driving cars it won’t make sense to own cars at the same level as today. There will be a decline. And even if Tesla is a leader in autonomous vehicles it doesn’t seem possible that others won’t be competing in that space considering how evey car manufacturer has invested in adaptive crusie control tech dev (unlike EV… which they lagged significantly). And it’s hard to imagine people will only use Tesla self-driving. Premium will have its place (limos) but people will tend to go with the cheapest functional self driving service – that I’m sure about. So the next 5-10 years should be great but at best… after that is unknown. At worst, expect car ownership to drop and self driving taxi companies to be a dime a dozen. Too hard to predict which means the value is too high. It’s like investing in Garmin in the early 2000s because you think navigation systems are the future but ignoring that rise of smartphones with maps.
Well stated! Couldn’t agree more!
Elon has been saying all along that he is willing to open up the charging network to other manufacturers. So far no one has been interested in taking him up on it though I think it would be to their advantage. They would immediately have access to a wide network for less cost than building their own smaller one. EJB
He has said that. But it hasn’t happened because he’s seeking payments from other car companies and only willing to let each automaker on separately. If Musk wanted it to happen it would have happened. He’s recently suggested he’s reach some low key deals but it must be very low key because no one knows what he’s talking about. The network is very important right now but it may not always be important so I wish he would do more now. If solid state batteries can perform like recent prototypes at scale or range increases with standard battery tech and we get to 1000+ mile range… networks won’t matter.
And to remind people what a 800 PE ratio means – it means if profits don’t rise it would take 800 years to breakeven on investment. Tesla income will rise but it has more ground to cover than anyone can imagine. Even Musk has said its overvalued. In words and actions. In a low interest rate environment he’s issuing shares. People buyback shares when they are undervalued and issue when overvalued. Musk has been issuing all year.
first of self driving cars will not make ownership decline. we own cars for the convenience not for the driving itself, that the car, my car, in the state i left sits in my garage and is ready to serve me whenever I need it. not full of vomit and cigar smoke after some party bunch had a ride on it. this is just a greenie delusion. second of just because lidar is marginally better than stereo cams as technology it does not mean they are better in a sdcar, they are missing immense amount of data that cameras DO have. lidars only have position information and while its helpful in a dumb system, its totally unnecessary in a clever system… let me ask you something: does your head have lidar? no? so you shouldn’t drive, right? all you have is 2 cameras and 2 mics, and this is all you need to drive a car
LiDAR is only half of the solution (or more precisely, the problem.) Because it requires geofence HD Map, which is like hardcoded “AI”. It won’t be capable of self-driving in areas that it never drives there before (i.e. it cannot “live” w/o HD map.)
Seems you do not understand how a lidar works. And no one uses a lidar alone. 10 year old mercedes has a few radars and a few cameras in the front and the rear too. Today it is usually limited to a camera and a radar, and a lidar.