End-to-end autonomous driving method based on multi-source sensor and navigation map

doi:10.3969/j.issn.1674-8484.2022.04.015

Abstract

Abstract:

This paper proposed a multiple-input single-output (end-to-end) autonomous driving decision control model based on multi-source sensors and navigation map to make up for the shortcomings of existing end-to-end autonomous driving methods in active obstacle avoidance and passing through intersection by using the PilotNet model based on Deep Neural Network (DNN). The sensor data in model’s input end consisted of three parts: a monocular front view camera, a 2-D top view obtained by 360° multi-layer Light-Laser Detection and Ranging (LiDAR), and a local navigation map based on accurate positioning. The model’s output end generated the steering wheel angle, which was a vehicle control command. Some multi-condition simulations and real vehicle tests were conducted. The results show that the root mean square error (RMSE) of steering wheel angle decreases by 37% compared with that by using the PilotNet model. Therefore, the model has the ability of active obstacle avoidance and intersection passing.

Key words: autonomous vehicles, end-to-end autonomous driving, neural networks, camera, navigation maps, light-laser detection and ranging (LiDAR)

CLC Number:

U467.1+3

ZHU Bo, ZHANG Jiwei, TAN Dongkui, HU Xudong. End-to-end autonomous driving method based on multi-source sensor and navigation map[J]. Journal of Automotive Safety and Energy, 2022, 13(4): 738-749.

Figures/Tables 26

References 21

[1]	陈颖. 自动驾驶技术研究现状及发展趋势[J]. 能源技术与管理, 2021, 46(3): 34-37.
	CHEN Ying. Research status and development trend of automatic driving technology[J]. *Energy Tech Manag*, 2021, 46(3): 34-37. (in Chinese)
[2]	Broggi A, Cerri P, Debattisti S, et al. PROUD Public road urban driverless-car test[J]. *IEEE Trans Intell Transp Syst*, 2015, 16(6): 3508-3519.
[3]	Bojarski M, Yeres P, Choromanska A, et al. Explaining how a deep neural network trained with end-to-end learning steers a car[EB/OL]. (2017-04-25), https://arxiv.org/abs/1704.07911.
[4]	WANG Tinghan, LUO Yugong, LIU Jinxin, et al. End-to-end self-driving approach independent of irrelevant roadside objects with auto-encoder[J]. *IEEE Trans Intell Transp Syst*, 2020, 23(1): 641-650.
[5]	Pomerleau D A. Alvinn: An autonomous land vehicle in a neural network[C]// Adva Neural Info Proc Syst. San Francisco: Morgan Kaufmann Publishers Inc, 1989: 305-313.
[6]	LeCun Y, Muller U, Ben J, et al. Off-road obstacle avoidance through end-to-end learning[C]// Int’l Conf Neural Info Proc Syst, Vancouver, 2005: 1-8.
[7]	Bojarski M, Testa D, Dworakowski D, et al. End to end learning for self-driving cars[EB/OL]. (2016-04-25), http://arxiv.org/abs/1604.07316.
[8]	XU Huazhe, GAO Yang, YU Fisher, et al. End-to-end learning of driving models from large-scale video datasets[C]// Proc IEEE Conf Computer Vision Pattern Recog, IEEE, Honolulu, 2017: 2174-2182.
[9]	胡学敏, 童秀迟, 郭琳, 等. 基于深度视觉注意神经网络的端到端自动驾驶模型[J]. 计算机应用, 2020, 40(7): 1926-1931. doi: 10.11772/j.issn.1001-9081.2019112054
	HU Xuemin, TONG Xiuchi, GUO Lin, et al. End-to-end autonomous driving model based on deep visual attention neural network[J]. *Computer Appl*, 2020, 40(7): 1926-1931. (in Chinese)
[10]	Codevilla F, Mller M, Lpez A, et al. End-to-end driving via conditional imitation learning[C]// 2018 IEEE Int’l Conf Robo Automation (ICRA). Brisbane, 2018: 4693-4700.
[11]	Sallab A, Abdou M, Perot E, et al. Deep reinforcement learning framework for autonomous driving[J]. *Electronic Imaging*, 2017(19): 70-76.
[12]	CHEN Siyuan, WANG Meiling, SONG Wenjie, et al. Stabilization approaches for reinforcement learning-based end-to-end autonomous driving[J]. *IEEE Trans Vehi Tech*, 2020, 69(5): 4740-4750.
[13]	SONG Wenjie, LIU Shixian, LI Yujun, et al. Smooth actor-critic algorithm for end-to-end autonomous driving[C]// 2020 Ame Control Conf, ACC, Colorado: 2020: 3242-3248.
[14]	Abdou M, Kamal H, El-Tantawy S, et al. End-to-end deep conditional imitation learning for autonomous driving[C]// 2019 31st Int’l Conf Micro- Electronics (ICM), IEEE, Beijing, 2020: 346-350.
[15]	卢宏涛, 张秦川. 深度卷积神经网络在计算机视觉中的应用研究综述[J]. 数据采集与处理, 2016, 31(1): 1-17.
	LU Hongtao, ZHANG Qinchuan. A review of the application of deep convolutional neural networks in computer vision[J]. *Data Acqu Proce*, 2016, 31(1): 1-17. (in Chinese)
[16]	Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[EB/OL]. (2015-04-10), https://arxiv.org/abs/1409.1556.
[17]	Toth L. Phone recognition with deep sparse rectifier neural networks[C]// IEEE Acoustics Speech Signal Proc (ICASSP), IEEE, California: 2013: 6985-6989.
[18]	CHEN Zhilu, HUANG Xinming. End-to-end learning for lane keeping of self-driving cars[C]// 2017 IEEE Intell Vehi Symp (IV). IEEE, Los Angeles, 2017: 1856-1860.
[19]	Jhung J, Bae I, Moon J, et al. End-to-end steering controller with CNN-based closed-loop feedback for autonomous vehicles[C]// 2018 IEEE Intell Vehi Symp (IV). IEEE, Changshu, 2018: 617-622.
[20]	Mihalea A, Samoilescu R, Nica A, et al. End-to-end models for self-driving cars on UPB campus roads[C]// 2019 IEEE 15th Int’l Conf Intell Computer Commu Proc (ICCP). IEEE, Tokyo, 2019: 35-40.
[21]	YANG Zhengyuan, ZHANG Yixuan, YU Jerry, et al. End-to-end multi-modal multi-task vehicle control for self-driving cars with visual perceptions[C]// 2018 24th Int’l Conf Pattern Recog (ICPR). IEEE, Beijing, 2018: 2289-2294.

整车装备质量，m	1.532 t
前轴到车辆质心的距离, l_f	1.21 m
后轴到车辆质心的距离, l_r	1.5 m
质心高度, h_cog	0.5 m
车辆的宽度, w_v	2.07 m
车辆的长度 l	4.63 m

整车装备质量，m	1.532 t
前轴到车辆质心的距离, l_f	1.21 m
后轴到车辆质心的距离, l_r	1.5 m
质心高度, h_cog	0.5 m
车辆的宽度, w_v	2.07 m
车辆的长度 l	4.63 m

网络模型	RMSE / (°)
PilotNet	5.67
Model-A	2.81
Model-B	3.23
Model-C	3.57

网络模型	RMSE / (°)
PilotNet	5.67
Model-A	2.81
Model-B	3.23
Model-C	3.57

模型	输入数据	t / ms	车道保持功能	主动避障行驶	通过交叉路口
PilotNet	摄像头	3.9	√	×	×
Model-A	摄像头	5.6	√	×	×
Model-B	摄像头 + LiDAR	9.8	√	√	×
Model-C	摄像头 + LiDAR + 导航地图	12.7	√	√	√