Autonomous Vehicles Need Thermal Cameras

Share this post

We live in a colorful world where road signs are in colors for easy detection and understanding. The risk level increases dramatically at night and unfortunately that is when most of the fatal accidents occur.

A pedestrian crossing a dark suburban street. Visible light camera vs. FLIR® thermal camera captured by Foresight’s test vehicle

The future of autonomous vehicles requires driving in harsh weather and lighting conditions. This is practically impossible without the use of thermal cameras. The QuadSight 2.0™ vision system is the most advanced technology combing both visible-light and thermal cameras, resulting in incomparable capabilities which will be the backbone of future autonomous driving. There is no doubt that cameras will be used in future self-driving cars. We live in a colorful world and all our road signs are in colors for easy detection and understanding. But what happens at night? In urban areas there is usually still some light coming from streetlamps or other light sources, but even then, our vision capabilities decrease and awareness to hazards on the road is compromised in low light. In the suburbs, or while driving on a dark road, the risk level increases dramatically and unfortunately that is when most of the fatal accidents occur. This was the case on March 18th, 2018, just before 10 PM. An Uber employee was behind the wheel of a Volvo, cruising just under the speed limit down a four-lane road in Tempe, Arizona. She was not watching the road. Instead, she gazed at her cell phone, which was streaming a reality TV show. The driver wasn’t being completely negligent, after all, the car had eyes of its own: a lidar imager, seven optical cameras, and ten radars. It also had a powerful computer brain, programmed to prevent the car from hitting other vehicles, crashing into fixed objects, and running over pedestrians. On that last point, Uber’s Volvo failed spectacularly. It struck a pedestrian named Elaine Herzberg at a speed of 56 km/h. Herzberg was taken to the local hospital where she died of her injuries. The National Transportation Safety Board’s investigation of the crash reported that while the car’s sensors had indeed picked up on something up ahead, they couldn’t provide sufficient and coherent information to the computing system for it to conclusively figure out what that thing was leading the car to essentially “not make” a decision, ending with a fatal crash. Since the Tempe tragedy, many experts in the remote sensing community have pointed out that if Uber’s sensor array had included a thermal infrared camera, the car would have likely identified Hertzberg in time to hit the brakes.

Footage taken from the Uber camera just before the it hit Elaine Herzberg

Thermal cameras are not new to the automotive industry. In fact, they have been helping human drivers identify pedestrians, animals, and cyclists for nearly two decades. In 2000, Cadillac introduced its Night Vision system on the Deville product line.

Using a grill-mounted, 3-inch passive sensor, Cadillac’s system (developed by Raytheon) picked up thermal imagery from the oncoming road, then transmitted it to a display in the car’s center dashboard. Cooler objects were darker, while warm objects, like humans or animals, showed bright white.

Cadillac introducing its night vision assistant system in 2000

About five years later, BMW also introduced thermal infrared. and in 2008, they added an upgrade: the onscreen display would automatically detect, and emphasize, humans or animals in the line of traffic. Now this tool wasn’t just enhancing situational awareness, it was a step towards the type of situational awareness AI needs to develop for automated driving.
FLIR Systems designed BMW’s thermal infrared cameras. And while the system was initially meant to help drivers see better at night, FLIR realized that they can help bring autonomous driving to the next level. Implementing thermal cameras in a vehicle is no easy task. Auto manufacturers are tough customers; they want everything cheap, reliable, and durable. Companies worldwide have been working extensively on the durability and manufacturing process in order to bring down prices of thermal cameras. Developing a highly-reliable detection system that meets autonomous requirements, requires extensive knowledge and experience of image processing and AI development, and is an ongoing learning process that only few companies can manage.
Foresight has been developing advanced vision solutions for the automotive industry and has unique capabilities and knowledge in thermal imaging and stereoscopic imaging. The Company recently joined the Thermal by FLIR® program, enabling Foresight to more quickly develop safer, more reliable, and durable autonomous vehicle technology.

Get in touch

Subscribe to our newsletter