End-to-end decision-making model for multi-task autonomous driving

doi:10.3969/j.issn.1674-8484.2025.04.011

Abstract

Abstract:

To address the challenges of spatiotemporal feature processing and inter-task dependencies in autonomous driving decision-making, this paper proposed an end-to-end driving decision model based on a 3D window self-attention mechanism. By applying window self-attention to compute the spatiotemporal features of the input sequence, and combining multi-task learning with loss weight allocation, the model effectively extracts features from driving videos and predicts vehicle speed and steering angle. The results demonstrate that the proposed model achieves prediction accuracies of 86.32% for steering angle and 85.36% for vehicle speed, outperforming models such as FMNet, Swin-Transformer, and MobileT-DSM. Moreover, it requires only 57.48 GFLOPs of computational cost, exhibiting superior spatiotemporal feature extraction as well as a better trade-off between performance and efficiency.

Key words: autonomous driving, decision-making and control, deep learning, multi-task, attention mechanism

CLC Number:

U461.6

OUYANG Delin, QIU Yifan, WANG Yingchen, YANG Liang, MIN Haigen, WANG Wenjun, LI Guofa. End-to-end decision-making model for multi-task autonomous driving[J]. Journal of Automotive Safety and Energy, 2025, 16(4): 610-619.

Figures/Tables 11

References 21

[1]	Federal Highway Administration. How Do Weather Events Affect Roads? [EB/OL]. [2025-07-22] https://ops.fhwa.dot.gov/weather/roadimpact.htm.
[2]	Highway Traffic Safety Administration. Early estimate of motor vehicle traffic fatalities in 2024[R]. Washington DC. U.S. Department of Transportation, 2025, DOT HS 813 710.
[3]	Handiso A, Mekebo G G, Galdassa A. Trends and determinants of road traffic accident human death in Kembata Tembaro zone, SNNPR, Ethiopia[J]. Sci J Appl Math Statist, 2022, 10(5): 85-89.
[4]	Frédéric V. A rule-based support system for dissonance discovery and control applied to car driving[J]. Expert Syst Appl, 2016, 65(12): 361-371.
[5]	LI Liangzhi, OTA Kaoru, DONG Mianxiong. Humanlike driving: Empirical decision-making system for autonomous vehicles[J]. IEEE Trans Vehi Tech, 2018, 67(8): 6814-6823.
[6]	CHEN Li, WU Penghao, CHITTA Kashyap, et al. End-to-end autonomous driving: Challenges and frontiers[J]. IEEE Trans Patt Anal Mach Intel, 2024, 46(12): 10164-10183.
[7]	LI Guofa, YANG Yifan, QU Xingda, et al. A deep learning-based image enhancement approach for autonomous driving at night[J]. Know-Based Syst, 2021, 213: 106617.
[8]	Luca A, Paola B, Silvio B, et al. An end-to-end curriculum learning approach for autonomous driving scenarios[J]. IEEE Trans Intel Transport Syst, 2022, 23(10): 19817-19826.
[9]	YI Xianyong, Hakim G, Yehia M. End-to-end neural network for autonomous steering using lidar point cloud data [C]// 2022 IEEE 65th Int’l Midwest Symp Circ Syst (MWSCAS). Fukuoka, Japan, 2022: 1-4.
[10]	TENG Siyu, CHEN Long, AI Yunfeng, et al. Hierarchical interpretable imitation learning for end-to-end autonomous driving[J]. IEEE Trans Intel Vehi, 2022, 8(1): 673-683.
[11]	Song T-Taek-Jin, Jeong Jongoh, Kim Jong-Hwan. End-to-end real-time obstacle detection network for safe self-driving via multi-task learning[J]. IEEE Trans Intel Transport Syst, 2022, 23(9): 16318-16329.
[12]	Youssef M, Abdallah W, Abdel-Halim M, et al. Time fusion net for end-to-end self-driving cars[C]// 2023 5th Novel Intel Lead Emerg Sci Conf (NILES). Giza, Egypt, 2023: 42-47.
[13]	ZHOU Wujie, DONG Shaohua, LEI Jingsheng, et al. MTANet: Multitask-aware network with hierarchical multimodal fusion for RGB-T urban scene understanding[J]. IEEE Trans Intel Vehi, 2022, 8(1): 48-58.
[14]	LI Guofa, LIN Yongjie, QU Xingda. An infrared and visible image fusion method based on multi-scale transformation and norm optimization[J]. Info Fusion, 2021, 71: 109-129.
[15]	Andrew G H, ZHU Menglong, CHEN Bo, et al. Mobilenets: Efficient convolutional neural networks for mobile vision applications[J]. arXiv preprint arXiv: 170404861, 2017.
[16]	Maryam Z, Parham M K, Abbas K, et al. A survey of imitation learning: Algorithms, recent developments, and challenges[J]. IEEE Trans Cyber, 2024, 54(12): 7173-7186.
[17]	WU Weishang, DENG Xiaoheng, JIANG Ping, et al. Crossfuser: Multi-modal feature fusion for end-to-end autonomous driving under unseen weather conditions[J]. IEEE Trans Intel Transport Syst, 2023, 24(12): 14378-14392.
[18]	François C. Xception: Deep learning with depth wise separable convolutions[C]// Proc IEEE Conf Compu Visi Patt Recog. Honolulu, HI, USA, 2017: 1251-1258.
[19]	XIANG Qian, WANG Xiaodan, LAI Jie, et al. Quadruplet depth-wise separable fusion convolution neural network for ballistic target recognition with limited samples[J]. Expert Syst Appl, 2024, 235: No 121182.
[20]	Mark S, Andrew H, ZHU Menglong, et al. Mobilenetv2: Inverted residuals and linear bottlenecks[C]// Proc IEEE Conf Compu Visi Patt Recog. Salt Lake City, UT, USA, 2018: 4510-4520.
[21]	Alex K, Yarin G, Roberto C. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics[C]// Proc IEEE Conf Compu Visi Patt Recog. Salt Lake City, UT, USA, 2018: 7482-7491.

Operator	t	c	n	s
Conv2d 3×3	-	16	1	2
Bottlenecks	6	8	1	2
Bottlenecks	6	8	2	2
Bottlenecks	6	16	3	2
Bottlenecks	6	24	4	2
Bottlenecks	6	31	3	1
Bottlenecks	6	56	3	1
Conv2d 1×1		1280	1	1
Global Average Pooling	-	-	1	-
Linear	-	512	1	1

Operator	t	c	n	s
Conv2d 3×3	-	16	1	2
Bottlenecks	6	8	1	2
Bottlenecks	6	8	2	2
Bottlenecks	6	16	3	2
Bottlenecks	6	24	4	2
Bottlenecks	6	31	3	1
Bottlenecks	6	56	3	1
Conv2d 1×1		1280	1	1
Global Average Pooling	-	-	1	-
Linear	-	512	1	1

模型	车辆转向角				车辆行驶速度
模型	MAE / rad	RMSE / rad	AUC	Acc / %	MAE / (km·h^-1)	RMSE / (km·h^-1)	AUC	Acc / %
FMNet	0.129	0.214	0.893	83.78	-	-	-	-
Swin-Transformer	0.245	0.292	0.862	78.91	2.31	2.69	0.845	80.17
MobileT-DSM	0.163	0.212	0.878	81.69	1.91	2.47	0.857	83.17
3D ST-DSM	0.119	0.148	0.897	86.32	1.77	2.38	0.868	85.36

模型	车辆转向角				车辆行驶速度
模型	MAE / rad	RMSE / rad	AUC	Acc / %	MAE / (km·h^-1)	RMSE / (km·h^-1)	AUC	Acc / %
FMNet	0.129	0.214	0.893	83.78	-	-	-	-
Swin-Transformer	0.245	0.292	0.862	78.91	2.31	2.69	0.845	80.17
MobileT-DSM	0.163	0.212	0.878	81.69	1.91	2.47	0.857	83.17
3D ST-DSM	0.119	0.148	0.897	86.32	1.77	2.38	0.868	85.36

模型	参数量	计算量(GFLOPs)
FMNet	99 393 775	424.91
Swin-Transformer	27 521 661	4.49
MobileT-DSM	45 029 621	16.66
3D ST-DSM	27 485 088	57.48