Although autonomous vehicle testing is occurring around the world, New York City isn’t generally considered a hotbed for the technology. The biggest city in the U.S. and second biggest in North America could still play a key role in the development of self-driving cars, thanks in part to tests in the Brooklyn Navy Yard. CARMERA Inc., which creates “living” maps for autonomous vehicles, is a participant in those trials and is working with researchers and government agencies toward the goal of safe and reliable deployments.
The CARMERA Autonomous Map provides high-definition (HD) maps and navigation data to autonomous vehicles (AVs) in real time. The company uses data collected by sensors in partner vehicles. It says its “proprietary data pipelines conform to any customer-specific implementation of localization, verification, path planning, and simulation, allowing AVs to know where they are, confirm what they’re seeing, and know where to go next.”
Last year, CARMERA raised Series B funding of $20 million from GV, formerly Google Ventures. The company also won the Best in Show and Startup of the Year awards at the 2018 North American International Auto Show.
The Robot Report spoke with Ro Gupta, CEO of CARMERA, about the importance of mapping to AVs and what the startup has learned in New York.
A key part of the AV stack
“If you think about the three or four core parts of any autonomous vehicle or robotics stack, they include localization, perception, planning, and control or actuation,” explained Gupta. “With the exception of Tesla, everyone in the industry is using maps to get to redundancy for the first three of these questions: ‘Where am I?’ ‘What’s around me?’ ‘What’s next?’ and ‘How do I do it?'”
“Years ago, maps were used for only one or two of these questions, but now they’re useful for all of them, and we want them as close to the truth and the real world as possible,” he said. “To get to the level of 9s for the accuracy we want, a so-called living map is essential, with aspects looking backward and forward in time.”
“We want knowledge of what the street looks like, down to the centimeter — as well as lane markings, new signs, or lights — and the mapping service also says what’s around the corner or six blocks ahead to take into account,” Gupta said. “Also, there’s the concept of wisdom. By keeping a map up to date, you can pick up on historical information that’s useful for risk management. You can also have a longitudinal view of pedestrian activity or double-parking density.”
“This is not the same as monitoring in real time, but if you know that a certain part of New York or San Francisco is busy at a certain time of day or month, you can take it into account for planning,” he added. “A living map that’s always regenerating is useful for both short-range motion planning and longer-range path planning.”
CARMERA Site Intelligence — colorized, cm-accurate 3D point cloud from New York City’s West Village. Source: CARMERA
Cartography with CARMERA
“To produce the quality data needed to support Level 4 autonomy in urban areas, we need to maintain and update maps in minutes,” said Gupta. “This can be hard to scale, so we had to make lots of architecture decisions.”
“CARMERA uses lidar for the initial observations, but it’s too speculative for maintenance,” he said. “We use cameras for detections and updates. We are crowdsourcing data, not only from our own customers’ vehicles, but also delivery fleet partners.”
CARMERA is working with Toyota Research Institute-Advanced Development Inc. (TRI-AD) on its Automated Mapping Platform. The open software platform combines data gathered from vehicles of participating companies to generate HD maps. CARMERA has also joined the Baidu Apollo open-source program.
“We’re I/O-agnostic and have ingested data from subcontractors, Voyage robotic taxicabs, and Toyota cameras. There are no standards yet,” Gupta said. “With our Data-as-a-Service business model, we’re combining ODDs, or operational design domains — the network, geofenced areas in the city, combined with application data, frequency, and change management. This is similar to other automotive industry services, such as SiriusXM entertainment.”
CARMERA shares data
“We have a data-sharing framework with the city of New York, starting with pedestrian analytic data,” said Gupta. “Thanks to fleet partnerships with sensors, we have a good sense by time, date, and location, which is helpful for the departments of transportation or sanitation. CARMERA also shares construction event data, but that’s not aligned with the permanent database.”
Privacy is less of a concern for data from mobile sensors than it is for static sensors, which can watch one location and observe who is coming and going, Gupta said. “We had to adapt machine vision to blur license plates, but with mobile sensors, there’s less of a risk,” he said.
Is 5G something that autonomous vehicle developers and smart city advocates should be watching? “We have to assume the worst and hope for the best, which applies to everything we do,” Gupta said. “We have to apply our technologies for customers who have to get out now.”
“We assume the worst in that there are a lot of edge cases to incorporate, and we have to prepare for zero connectivity. We have to have in-vehicle modules and caching so the vehicle still has access to a true map for safety,” he said. “On the OK side, we’re on the fat part of the bell curve for 4G LTE cellular, and increased connectivity and ubiquitous reliability from 5G would be great.”
CARMERA promises to democratize autonomous vehicle data. Source: CARMERA
Moving from Level 2 to 4 autonomy
“Most people think of Level 4 as true driverless vehicles, but the fleet mobility service model is more likely to make sense first technologically,” acknowledged Gupta, who cited Optimus Ride, Voyage, and Waymo as examples. Six self-driving shuttles from Optimus Ride are participating in the Brooklyn Navy Yard tests.
“So much of the problem now is on the data and perception software side rather than on the hardware side. There’s still a lot of [research] activity in individual vehicles, and the parallel paths will bleed into each other,” he predicted. “There will be Level 2 streams and lakes, and you’ll start to see true Level 4 service models in cities. It will look like old cellphone coverage maps.”
“The Brooklyn Navy Yard is a lake for mobility as a service,” Gupta said. “It’s relevant, not only for shuttle deployments with Optimus Ride, but also for the policy side with other stakeholders.”
CARMERA engages with stakeholders
CARMERA has been in Brooklyn, N.Y., since it was founded in 2015. Since then, it has opened offices in Seattle and San Francisco, largely for recruiting purposes, said Gupta. “We’re looking for machine learning, computer vision, and mapping engineering backgrounds, as well as computer science generalists and software engineers for the full stack,” he said.
“All of these cities have lots of technical talent, but not as many opportunities to work on self-driving problems,” he said. “The cost of living and quality of life are better compared with the Bay Area. Washington [state] doesn’t have any state taxes, and we qualified for a program in New York in which employees don’t have to pay state or city taxes.”
“We’re still the only pure-play AV company in New York City, and we got pulled in by a lot of constituents — the city, state, borough, Brooklyn Navy Yard Development Corp., and research institutions,” he recalled. “We started talking collaboratively and asked about New York’s role, since it wasn’t on the bleeding edge.”
“On the one hand, it’s the biggest opportunity in the U.S., but it’s also complex for environmental, density, and political reasons,” Gupta said. “One of the things that we’ve learned is that you can’t just move fast and break things or try to go around stakeholders. It’s not like the traditional Silicon Valley attitude of asking for forgiveness later [rather than permission first].”
“CARMERA was a tenant of the New Lab at the Brooklyn Navy Yard, so it was a natural place to continue our own research and development,” he said. “It was very supportive of bringing new technologies like aerial, terrestrial, and marine robotics there. CARMEA took advantage of the hybrid operating structure — it’s tied to government entities but operating on private property. It’s better for safety and permit reasons to have a closed-loop environment.”
“This sparked a whole process with various government representatives and institutions to identify the Brooklyn Navy Yard as a place where we could prove out the technologies,” said Gupta. “It also offers educational opportunities for local universities and robotics job opportunities.”
“In addition, these modes of autonomy make sense across the broader portfolio of the city,” he said. “Even with mass transit, there are transit deserts that could be served by autonomous fleets.”
The Robot Report has launched the Healthcare Robotics Engineering Forum, which will be on Dec. 9-10 in Santa Clara, Calif. The conference and expo focuses on improving the design, development and manufacture of next-generation healthcare robots. Learn more about the Healthcare Robotics Engineering Forum.
Answering customer hails
“The frequency and density of mapping is very customer- and use-case-specific,” Gupta said. “We get requests for up to hourly updates, as well as every few months. A robotaxi will have a higher threshold than an automotive OEM.”
“CARMERA’s data exhausts are building the maps we find useful for other stakeholders,” he explained. “It’s not raw data, but the analytics on top of that. As long as we meet service-level agreements for a given customer, for historical purposes, all we need are the right samples to be statistically significant and predictive.”
“We’re a horizontal supplier in many ways, since almost all core AV brain questions mention the map,” Gupta said. “We see a broad cross-section of what everyone is doing. Optimus Ride or May Mobility’s shuttles are one aspect, and the other is our public partners, Toyota, Voyage, and Baidu Apollo.”
“I’ve taken rides from Level 2/3 AVs that can operate at 70 mph on limited-access highways, while some robotaxis are Level 4 in actual city environments,” he said. “Voyage is in The Villages in Florida, an area that’s actually bigger than Manhattan, Boston, or central San Francisco but is still a geofenced, closed-loop ODD. Point-to-point services over one to three miles for senior citizens is a big opportunity.”
“I got my start designing autonomous transit systems in the ’90s, when the concept was personal rapid transit on a monorail serving airports or office complexes,” Gupta said. “It was cost-prohibitive to lay down the infrastructure for such people movers, but a simplified version of what we do is build digital rails for our customers for redundancy and safety.”
“Our next milestone is around democratizing the technology, and we’re doing a lot of work with research institutions and the developer community,” he said. “We’re the only third-party, living map, Level 4-focused company in the Baidu Apollo ecosystem. It’s all about production paths for our customers.”
Certainly the mapping function provided by the Carmera organization would be very useful and have great value in enabling computer driven cars to operate. But these maps require a great deal of effort, and they will be expensive, both to produce and to maintain. And while mapping some portion of New York city is useful in that area, the cost of mapping the whole area where cars travel would be huge. That cost alone would be the end of the computer driven car dream. That is the unfortunate reality, that the price is far too great and there is no way to pay it.