Recognition of front vehicle behavior based on visual information from vehicle perspective

doi:10.3969/j.issn.1674-8484.2023.06.007

Abstract

Abstract:

Unlike traditional vehicle behavior recognition methods, which are mostly based on historical track information from a bird's eye view, this paper proposed a new approach for autonomous vehicle behavior recognition based on visual information. A vehicle behavior labeling method based on vehicle-mounted video information was proposed, and a behavior recognition data set was constructed from vehicle-mounted perspective. A vehicle behavior recognition algorithm based on SlowFast network was designed; A Focal Loss function was designed; And a Non-local operation module was introduced. The results show that compared with the original SlowFast model, the overall accuracy of the new model is improved by 20.56%, and the behavior recognition of multiple vehicles in front of the video is realized.

Key words: autonomous vehicles, on-board perception, front vehicle behavior recognition, SlowFast networks, vehicle perspective

CLC Number:

U471

LIU Yanwei, HUANG Zhiming, GAO Bolin, ZHONG Wei, CHEN Jiaxing, LIU Jiaxi. Recognition of front vehicle behavior based on visual information from vehicle perspective[J]. Journal of Automotive Safety and Energy, 2023, 14(6): 707-714.

Figures/Tables 10

References 17

[1]	王相海, 丛志环, 方玲玲, 等. 基于 HMM 的车辆行驶状态实时判别方法研究[J]. 自动化学报, 2013, 39(12): 2131-2142.
	WANG Xianghai, CONG Zhihuai, FANG lingling, et al. Determination of real-time vehicle driving status using HMM[J]. Acta Automatica Sinica, 2013, 39(12): 2131-2142. (in Chinese) doi: 10.3724/SP.J.1004.2013.02131 URL
[2]	XU Donghao, HE Xu, ZHAO Huijing, et al. Ego-centric traffic behavior understanding through multi-level vehicle trajectory analysis[C]//2017 IEEE Int’l Conf Robo Automa (ICRA), 2017: 211-218.
[3]	LIU Zhi, WU Xuegang, NI Jie, et al. Driving intention recognition based on HMM and SVM cascade algorithm[J]. Autom Engineering, 2018, 40(7): 858-864.
[4]	黄慧玲, 杨明, 王春香, 等. 基于前方车辆行为识别的碰撞预警系统[J]. 华中科技大学学报(自然科学版), 2015, 43(S1): 117-121.
	HUANG Huiling, YANG Ming, WANG Chunxiang, et al. Collision warning system based on forward vehicle behavior recognition[J]. J Huazhong Univ Sci Tech (Nat Sci Edit), 2015, 43(S1) : 117-121. (in Chinese)
[5]	党彤. 基于贝叶斯网络的车辆变道行为分析[D]. 西安: 西安理工大学, 2018.
	DANG Tong. Analysis of vehicle lane change behavior based on Bayesian network[D]. Xi'an: Xi'an University of Technology, 2018. (in Chinese)
[6]	卫星, 乐越, 韩江洪, 等. 基于长短期记忆的车辆行为动态识别网络[J]. 计算机应用, 2019, 39(7): 1894-1898. doi: 10.11772/j.issn.1001-9081.2018122448
	WEI Xing, LE Yue, HAN Jianghong, et al. Vehicle behavior dynamic recognition network based on long short-term memory[J]. J Comp Appl, 2019, 39(7): 1894-1898. (in Chinese)
[7]	季学武, 费聪, 何祥坤, 等. 基于LSTM网络的驾驶意图识别及车辆轨迹预测[J]. 中国公路学报, 2019, 32(6): 34-42. doi: 10.19721/j.cnki.1001-7372.2019.06.003
	JI Xuewu, FEI Cong, HE Xiangkun, et al. Driving intention recognition and vehicle trajectory prediction based on LSTM network[J]. Chin J Highw Transport, 2019, 32(6): 34-42. (in Chinese) doi: 10.19721/j.cnki.1001-7372.2019.06.003
[8]	蔡英凤, 邰康盛, 王海, 等. 无人驾驶汽车周边车辆行为识别算法研究[J]. 汽车工程, 2020, 42(11): 1464-1472. doi: 10.19562/j.chinasae.qcgc.2020.11.003
	Cai Yingfeng, Tai Kangsheng, Wang Hai et al. Research on behavior recognition algorithm of surrounding vehicles for driverless ca[J]. Autom Engineering, 2020, 42(11): 1464-1472. (in Chinese)
[9]	MOU Luntian, XIE Haitao, MAO Shasha, et al. Vision-based vehicle behaviour analysis: A structured learning approach via convolutional neural networks[J]. IET Intel Transport Syst, 2020, 14(7): 792-801.
[10]	Krajewski R, Bock J, Kloeker L, et al. The highd dataset: A drone dataset of naturalistic vehicle trajectories on german highways for validation of highly automated driving systems[C]// 2018 21st Int’l Conf Intel Transp Syst (ITSC), 2018: 2118-2125.
[11]	Geiger A, Lenz P, Urtasun R. Are we ready for autonomous driving? the KITTI vision benchmark suite[C]// 2012 IEEE Conf Comp Visi Patt Recog, 2012: 3354-3361.
[12]	YU Fisher, CHEN Haofeng, WANG Xin, et al. BDD100K: A diverse driving dataset for heterogeneous multitask learning[C]//Proc IEEE/CVF Conf Comp Visi Patt Recog, 2020: 2636-2645.
[13]	Feichtenhofer C, FAN Haoqi, Malik J, et al. Slowfast networks for video recognition[C]// Proc IEEE/CVF Int’l Conf Comp Visi, 2019: 6202-6211.
[14]	Carreira J, Zisserman A. Quo vadis, action recognition? A new model and the kinetics dataset[C]// Proc IEEE Conf Comp Visi Patt Recog, 2017: 6299-6308.
[15]	HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]// Pro IEEE Conf Comp Visi Patt Recog, 2016: 770-778.
[16]	LIN Tsung-Yi, Goyal P, Girshick R, et al. Focal loss for dense object detection[C]// Proc IEEE Int’l Conf Comp Visi, 2017: 2980-2988.
[17]	WANG Xiaolong, Girshick R, Gupta A, et al. Non-local neural networks[C]// Proc IEEE Conf Comp Visi Patt Recog, 2018: 7794-7803.

帧序号	X1	Y1	X2	Y2	行为标签
31	0.397	0.460	0.466	0.555	2
41	0.462	0.480	0.503	0.536	2
51	0.478	0.477	0.512	0.527	2
61	0.556	0.479	0.646	0.592	2
71	0.534	0.479	0.606	0.570	2
81	0.515	0.477	0.574	0.557	1
91	0.504	0.479	0.554	0.546	1
101	0.520	0.479	0.583	0.561	1
111	0.515	0.477	0.574	0.557	1

帧序号	X1	Y1	X2	Y2	行为标签
31	0.397	0.460	0.466	0.555	2
41	0.462	0.480	0.503	0.536	2
51	0.478	0.477	0.512	0.527	2
61	0.556	0.479	0.646	0.592	2
71	0.534	0.479	0.606	0.570	2
81	0.515	0.477	0.574	0.557	1
91	0.504	0.479	0.554	0.546	1
101	0.520	0.479	0.583	0.561	1
111	0.515	0.477	0.574	0.557	1

阶段	慢速路径	快速路径	输出大小：T×S²
视频Clips	-	-	64×224²
Data layer	Stride 16, 1²	Stride 16, 1²	Slow: 4×224² Fast: 32×224²
Conv1	1×7², 64 Stride 1, 2²	5×7², 8 Stride 2, 1²	Slow: 4×112² Fast: 32×112²
Pool1	1×3², max Stride 1, 2²	1×3², max Stride 1, 2²	Slow: 4×56² Fast: 32×56²
Res2			Slow: 4×56² Fast: 32×56²
Res3			Slow: 4×28² Fast: 32×28²
Res4			Slow: 4×14² Fast: 32×14²
Res5			Slow: 4×7² Fast: 32×7²
	平均池化，全连接层		Classes

阶段	慢速路径	快速路径	输出大小：T×S²
视频Clips	-	-	64×224²
Data layer	Stride 16, 1²	Stride 16, 1²	Slow: 4×224² Fast: 32×224²
Conv1	1×7², 64 Stride 1, 2²	5×7², 8 Stride 2, 1²	Slow: 4×112² Fast: 32×112²
Pool1	1×3², max Stride 1, 2²	1×3², max Stride 1, 2²	Slow: 4×56² Fast: 32×56²
Res2			Slow: 4×56² Fast: 32×56²
Res3			Slow: 4×28² Fast: 32×28²
Res4			Slow: 4×14² Fast: 32×14²
Res5			Slow: 4×7² Fast: 32×7²
	平均池化，全连接层		Classes

方案	模型	精确率 / %					召回率 / %					总体准确率/ %
		变道		转弯		横行穿越	变道		转弯		横行穿越
		左	右	左	右	横行穿越	左	右	左	右	横行穿越
1	SlowFast	66.20	43.10	34.59	36.05	47.83	47.96	59.53	23.23	24.41	81.76	47.56
2	SlowFast + Focal Loss	72.57	64.58	44.55	52.48	53.16	71.09	57.67	47.47	41.73	68.24	59.16
3	SlowFast + Non-local	73.70	70.87	43.84	63.64	63.45	72.45	67.91	61.11	33.07	62.16	62.52
4	SlowFast + Focal Loss + Non-local	78.74	77.44	47.19	70.73	70.80	80.61	70.23	63.64	45.67	65.54	68.12