The development of autonomous vehicles represents a dramatic change in transportation systems. These autonomous cars are based on a cutting-edge set of technologies that enable them to drive safely and effectively without the need for human intervention.
Computer vision is a key component of self-driving cars. It empowers the vehicles to perceive and comprehend their surroundings, including roads, traffic, pedestrians, and other objects. To obtain this data, a vehicle makes use of cameras and sensors. It then makes quick decisions and drives safely in various road conditions based on what it observes.
In this article, we’ll elaborate on how computer vision enhances these cars. We will describe the object detection models, data processing with a LiDAR device, analyzing scenes, and planning the route.
Development Timeline of Autonomous Vehicles
A growing number of automobiles with technology that allow to operate the vehicles under human supervision have been manufactured and released onto the market. Advanced driver assistance systems (ADAS) and automated driving systems (ADS) are both new forms of driving automation.
Here we present the development timeline of the autonomous vehicles.
- 1971 – Daniel Wisner designed an electronic cruise control system
- 1990 – William Chundrlik developed the adaptive cruise control (ACC) system
- 2008 – Volvo invented the Automatic Emergency Braking (AEB) system.
- 2013 – Introducing computer vision methods for vehicle detection, tracking, and behavior understanding
- 2014 – Tesla launched its first commercial autonomous vehicle Tesla model S
- 2015 – Algorithms for vision-based vehicle detection and tracking (collision avoidance)
- 2017 – 27 publicly available data sets for autonomous driving
- 2019 – 3D object detection (and pedestrian detection) methods for autonomous vehicles
- 2020 – LiDAR technologies and perception algorithms for autonomous driving
- 2021 – Deep learning methods for pedestrian, motorcycle, and vehicle detection
Key CV methods in Autonomous Vehicles
To navigate safely, autonomous vehicles employ a combination of sensors, cameras, and intelligent algorithms. To accomplish this, they require two key components: machine learning and computer vision.
The eyes of the automobile are computer vision models. They record images and videos of everything surrounding the vehicle using cameras and sensors. Road lines, traffic signals, people, and other vehicles are all examples of this. The vehicle then interprets these images and videos using specialized methods.
Machine learning methods represent the brain of the car. They analyze the information from the sensors and cameras. After that, they utilize specialized algorithms to identify trends, predict outcomes, and absorb fresh data. Here we’ll present the main CV techniques that allow autonomous driving.
Object Detection
Training self-driving cars to recognize objects on the road and around them is a major component of making them function. To differentiate between objects like other cars, pedestrians, road signs, and obstacles, the vehicles use cameras and sensors. The vehicle recognizes these items in real-time with speed and accuracy using sophisticated computer vision techniques.
Vehicles can recognize the appearance of the cyclist, pedestrian, or car in front of them thanks to class-specific object detection. The control system triggers visual and auditory alerts to advise the driver to take preventative action when it estimates the likelihood of a frontal collision with the identified pedestrian, bicycle, or vehicle.
Li et al. (2016) introduced a unified framework to detect both cyclists and pedestrians from images. Their framework generates several object candidates by using a detection suggestion method. They utilized a Faster R-CNN-based model to classify these object candidates. The detection performance is then further enhanced by a post-processing step.
Garcia et al. (2017) developed a sensor fusion approach for detecting vehicles in urban environments. The proposed approach integrates data from a 2D LiDAR and a monocular camera using both the unscented Kalman filter (UKF) and joint probabilistic data association. On single-lane roadways, it produces encouraging vehicle detection results.
Chen et al. (2020) developed a lightweight vehicle detector with a 1/10 model size that is three times faster than YOLOv3. EfficientLiteDet is a lightweight real-time approach for pedestrian and vehicle detection by Murthy et al. in 2022. To accomplish multi-scale object detection, EfficientLiteDet utilizes Tiny-YOLOv4 by adding a prediction head.
Object Tracking
When the vehicle detects something, it must keep an eye on it, particularly if it is moving. Understanding where objects, such as other vehicles and people, could move next is vital for path planning and preventing collisions. The vehicle predicts these things’ next location by tracking their movements over time. It is achieved by computer vision algorithms.
Deep SORT (Simple Online and Realtime Tracking with a Deep Association Metric), incorporates deep learning capabilities to increase tracking precision. It incorporates appearance data to preserve an object’s identity throughout time, even when it is obscured or briefly leaves the frame.
Monitoring the movement of items surrounding self-driving automobiles is crucial. To plan the movement of a steering wheel and prevent collisions, Deep SORT assists the vehicle in predicting the movements of these objects.
Deep SORT enables the self-driving cars to trace the paths of objects that are spotted by YOLO. This is particularly useful in traffic jams when vehicles, bikes, and people move in different ways.
Semantic Segmentation
For autonomous cars to comprehend and interpret their surroundings, semantic segmentation is essential. Semantic segmentation gives a thorough grasp of the objects in a picture, such as roads, cars, signs, traffic signals, and pedestrians, by classifying each pixel.
For autonomous driving systems to make wise decisions regarding their motions and interactions with their environment, this knowledge is crucial.
Semantic segmentation is now more accurate and efficient thanks to deep learning techniques that utilize neural network models. Semantic segmentation performance has improved as a result of more precise and effective pixel-level categorization made possible by convolutional neural networks (CNNs) and autoencoders.
Additionally, autoencoders acquire the ability to rebuild input images while preserving important details for semantic segmentation. Using deep learning techniques, autonomous cars can perform semantic segmentation at remarkable speeds without sacrificing accuracy.
Semantic segmentation real-time data analysis requires scene comprehension and visual signal processing. To categorize pixels into distinct groups, visual signal processing techniques extract valuable information from the input data, such as image attributes and characteristics. Scene understanding denotes the ability of the vehicle to understand its surroundings using segmented images.
Sensors and Datasets
Cameras
The most widely used image sensors for detecting the visible light spectrum reflected from objects are cameras. Cameras are comparatively less expensive than LiDAR and Radar. Camera images offer simple two-dimensional information that is useful for lane or object detection.
Cameras have a measurement range of several millimeters to one hundred meters. However, light and weather circumstances like fog, haze, mist, and smog have a major impact on camera performance, limiting its use to clear skies and daylight hours. Furthermore, since a single high-resolution camera normally produces 20–60 MB of data per second, cameras also struggle with enormous data volumes.
LiDAR
LiDAR is an active ranging sensor that measures the round-trip time of laser light pulses to determine an object’s distance. It can measure up to 200 meters because of its low divergence laser beams, which reduce power degradation over distance.
LiDAR can create precise and high-resolution maps because of its high-accuracy distance measuring capability. However, LiDAR is not appropriate for recognizing small targets due to its sparse observations.
Additionally, weather conditions can affect its measurement accuracy and range. Lastly, LiDAR’s extensive application in autonomous vehicles is restricted by its expensive cost. Additionally, LiDAR generates between 10 and 70 MB of data per second, which makes it difficult for onboard computer platforms to process this data in real-time.
Radar and Ultrasonic sensors
Radar detects objects by using radio or electromagnetic radiation. It can determine the distance to an object, the object’s angle, and relative speed. Radar systems typically run at 24 GHz or 77 GHz frequencies.
A 24 GHz radar can measure up to 70 meters, and a 77 GHz radar can measure up to 200 meters. Radar is better suited for measurements in environments with dust, smoke, rain, poor lighting, or uneven surfaces than LiDAR. The data size generated by each radar ranges from 10 to 100 KB.
Ultrasonic sensors use ultrasonic waves to measure an object’s distance. They receive the ultrasonic wave reflected from the target after the sensor head emits it. The time between emission and reception is measured to calculate the distance.
The advantages of ultrasonic sensors include their ease of use, excellent accuracy, and capacity to detect even minute changes in location. They are extensively utilized in car anti-collision and self-parking systems. Moreover, their measuring distance is restricted to fewer than 20 meters.
Data sets
The ability of full self driving vehicles to sense their surroundings is essential to their safe operation. Generally speaking, autonomous cars use a variety of sensors in addition to advanced computer vision algorithms to gather the data they need from their surroundings.
Benchmark data sets are necessary since these algorithms typically rely on deep learning methods, particularly convolutional neural networks (CNNs). Researchers from academia and industry have gathered a variety of data sets for assessing various aspects of autonomous driving systems.
The data sets utilized for perception tasks in autonomous vehicles that were gathered between 2013 and 2023 are compiled in the table below. The table displays the types of sensors, the existence of unfavorable circumstances (such as time or weather), the quantity of the data set, and the location of data collection.
Furthermore, it presents the types of annotation formats and possible applications. Therefore, the table provides guidelines for engineers to select the best data set for their particular application.
What’s Next for Autonomous Vehicles?
Autonomous vehicles will become significantly more intelligent as artificial intelligence (AI) advances. Although the development of autonomous technology has brought many exciting breakthroughs, there are still important obstacles that must be carefully considered:
- Safety features: Ensuring the safety of these vehicles is a significant task. In addition, developing safe mechanisms for automobiles is essential, e.g. traffic light obeying, blind spot detection, lane departure warning, etc. Also, to fulfill the requirements of the highway traffic safety administration.
- Reliability: These vehicles must always function properly, regardless of their location or the weather conditions. This kind of dependability is essential for gaining human drivers’ acceptance.
- Public trust: To get trust – autonomous vehicles require more than just demonstrating their reliability and safety. Educating the public about the advantages and limitations of these vehicles and being transparent about their operation, including security and privacy.
- Smart city integration: It will result in safer roads, less traffic congestion, and more efficient traffic flow. It all comes down to linking automobiles to the infrastructure of smart cities.
Frequently Asked Questions
Q1: What systems for assisted driving were predecessors of autonomous vehicles?
Answer: Advanced driver assistance systems (ADAS) and automated driving systems (ADS) are forms of driving automation that are predecessors to autonomous vehicles.
Q2: Which computer vision methods are crucial for autonomous driving?
Answer: Methods like object detection, object tracking, and semantic segmentation are crucial for autonomous driving systems.
Q3: What devices enable the sensing of the environment in autonomous vehicles?
Answer: Cameras, LiDAR, radars, and ultrasonic sensors – all these enable remote sensing of the surrounding traffic and objects.
Q4: Which factors affect the broader acceptance of autonomous vehicles?
Answer: The factors that affect broader acceptance of autonomous vehicles include their safety, reliability, public trust (including privacy), and smart city integration.
