Deep reinforcement learning-based lane-changing trajectory planning method of intelligent and connected vehicles

doi:10.3969/j.issn.1674-8484.2022.04.012

Abstract

Abstract:

A deep reinforcement learning-based lane-changing trajectory planning method of intelligent and connected vehicles (ICVs) was proposed to improve the lane-changing safety and efficiency of ICVs and reduce fuel consumption. A hierarchical ICV lane-changing trajectory planning architecture was designed based on the functional requirements of ICVs in complex traffic scenarios. Considering vehicle safety and lane-changing efficiency, a lane-changing behavior decision model was constructed based on complete information pure strategy game. A joint optimization function representing fuel consumption and passenger comfort was also constructed with decoupling the longitudinal and lateral motion states of vehicles. And based on twin delayed deep deterministic policy gradient (TD3) algorithm, a longitudinal and lateral lane-changing trajectory planning method of ICVs was proposed to achieve the longitudinal and lateral optimized lane-changing trajectory. The effectiveness of the algorithm was verified by using 3 typical lane-changing simulation scenarios. The results show that compared with the deep deterministic policy (DDPG) algorithm, the training efficiency of the proposed method in the experiment of left lane-changing and right lane-changing is increased by about 10.5% on average, the average fuel consumption is reduced by 65% and 44%, respectively, and the single step trajectory planning time is within 10 ms, which can obtain a safe, energy-saving and comfortable lane-changing trajectory in real time.

Key words: intelligent and connected vehicles (ICVs), deep reinforcement learning, lane-changing, trajectory planning

CLC Number:

TP273

FENG Yao, JING Shoucai, HUI Fei, ZHAO Xiangmo, LIU Jianbei. Deep reinforcement learning-based lane-changing trajectory planning method of intelligent and connected vehicles[J]. Journal of Automotive Safety and Energy, 2022, 13(4): 705-717.

Figures/Tables 16

References 31

[1]	李克强, 戴一凡, 李升波, 等. 智能网联汽车(ICV)技术的发展现状及趋势[J]. 汽车安全与节能学报, 2017, 8(1): 1-14.
	LI Keqiang, DAI Yifan, LI Shengbo, et al. State-of-the-art and technical trends of intelligent and connected vehicles[J]. *J Autom Safe Energ*, 2017, 8(1): 1-14. (in Chinese)
[2]	李立, 徐志刚, 赵祥模, 等. 智能网联汽车运动规划方法研究综述[J]. 中国公路学报, 2019, 32(6): 20-33. doi: 10.19721/j.cnki.1001-7372.2019.06.002
	LI Li, XU Zhigang, ZHAO Xiangmo, et al. Review of motion planning methods of intelligent connected vehicles[J]. *Chin J High Transport*, 2019, 32(6): 20-33. (in Chinese)
[3]	ZHAO Ding, Lam H, PENG Huei, et al. Accelerated evaluation of automated vehicles safety in lane-change scenarios based on importance sampling techniques[J]. *IEEE Trans Intel Transport Syst*, 2017, 18(3): 595-607.
[4]	卢兆麟, 李升波, Schroeder F, 等. 结合自然语言处理与改进层次分析法的乘用车驾驶舒适性评价[J]. 清华大学学报(自然科学版), 2016, 56(2): 137-143.
	LU Zhaolin, LI Shengbo, Schroeder F, et al. Driving comfort evaluation of passenger vehicles with natural language processing and improved AHP[J]. J Tsinghua Univ (Sci Tech), 2016, 56(2): 137-143. (in Chinese)
[5]	MA Liang, XUE Jianru, Kawabata K, et al. Efficient sampling-based motion planning for on-road autonomous driving[J]. *IEEE Trans Intel Transport Syst*, 2015, 16(4): 1961-1976.
[6]	Shaikh E A, Dhale A. AGV path planning and obstacle avoidance using Dijkstra’s algorithm[J]. *Int’l J Appl Innov Engi Manag*, 2013, 2(6): 77-83.
[7]	ZHANG Jing, WU Jun, SHEN Xiao, et al. Autonomous land vehicle path planning algorithm based on improved heuristic function of A-star[J]. *Int’l J Advan Robo Syst*, 2021, 18(5): 1-10.
[8]	张卫波, 肖继亮. 改进RRT算法在复杂环境下智能车路径规划中的应用[J]. 中国公路学报, 2021, 34(3): 225-234. doi: 10.19721/j.cnki.1001-7372.2021.03.017
	ZHANG Weibo, XIAO Jiliang. Application of improved RRT algorithm in intelligent vehicle path planning under complicated environment[J]. *Chin J High Transport*, 2021, 34(3): 225-234. (in Chinese)
[9]	YOU Feng, ZHANG Ronghui, LIE Guo, et al. Trajectory planning and tracking control for autonomous lane change maneuver based on the cooperative vehicle infrastructure system[J]. Expert Syst Appl: An Int’l J, 2015, 42(14): 5932-5946.
[10]	Broggi A, Medici P, Zani P, et al. Autonomous vehicles control in the vislab intercontinental autonomous challenge[J]. *Annu Rev Contr*, 2012, 36(1): 161-171.
[11]	Chu K, Lee M, Sunwoo M. Local path planning for off-Road autonomous driving with avoidance of static obstacles[J]. *IEEE Trans Intel Transport Syst*, 2012, 13(4): 1599-1616.
[12]	陈成, 何玉庆, 卜春光, 等. 基于四阶贝塞尔曲线的无人车可行轨迹规划[J]. 自动化学报, 2015, 41(3): 486-496.
	CHEN Cheng, HE Yuqing, BU Chunguang, et al. Feasible trajectory generation for autonomous vehicles based on quartic bezier curve[J]. *Acta Autom Sinica*, 2015, 41(3): 486-496. (in Chinese)
[13]	徐杨, 陆丽萍, 褚端峰, 等. 无人车辆轨迹规划与跟踪控制的统一建模方法[J]. 自动化学报, 2019, 45(4): 799-807.
	XU Yang, LU Liping, CHU Duanfeng, et al. Unified modeling of trajectory planning and tracking for unmanned vehicle[J]. *Acta Autom Sinica*, 2019, 45(4): 799-807. (in Chinese)
[14]	江浩斌, 施凯津, 华一丁, 等. 基于HP自适应伪谱法的智能汽车紧急变道轨迹规划与优化[J]. 中国公路学报, 2019, 32(6): 71-78. doi: 10.19721/j.cnki.1001-7372.2019.06.007
	JIANG Haobin, SHI Kaijin, HUA Yiding, et al. Lane-changing trajectory planning and optimization for intelligent vehicle through HP-adaptive pseudospectral method[J]. *Chin J High Transport*, 2019, 32(6): 71-78. (in Chinese)
[15]	Sallab A, Abdou M, Perot E, et al. Deep reinforcement learning framework for autonomous driving[J]. *Electr Imag*, 2017(19): 70-76.
[16]	FU Yuchuan, LI Changle, YU Richard, et al. A Decision-making strategy for vehicle autonomous braking in emergency via deep reinforcement learning[J]. *IEEE Trans Vehi Tech*, 2020, 69(6): 5876-5888.
[17]	Saxena D M, Bae S, Nakhaei A, et al. Driving in dense traffic with model-free reinforcement learning[J]. *IEEE Int’l Conf Robot Auto*, 2020(99): 5385-5392.
[18]	Hoel C J, Wolff K, Laine L. Automated speed and lane change decision making using deep reinforcement learning[J]. *IEEE Int’l Conf Intel Transport Syst*, 2018(99): 2148-2155.
[19]	PENG Baiyu, SUN Qi, LI Shengbo E, et al. End-to-end autonomous driving through dueling double deep Q-network[J]. *Autom Innocation*, 2021, 4(3): 328-337.
[20]	CHEN Jianyu, LI Shengbo E, Tomizuka M. Interpretable end-to-end urban autonomous driving with latent deep reinforcement learning[J]. *IEEE Trans Intel Transport Syst*, 2021(99): 1-11.
[21]	LI Guofa, YANG Yifan, LI Shen, et al. Decision making of autonomous vehicles in lane change scenarios: Deep reinforcement learning approaches with risk awareness[J]. Transport Res Part C: Emerg Tech, 2021, 134: 1-18.
[22]	Lillicrap T P, Hunt J J, Pritzel A, et al. Continuous control with deep reinforcement learning[J/OL]. (2015-09-09), http//arxiv.org/abs/1509.02971.
[23]	Fujimoto S, Hoof H V, Meger D. Addressing function approximation error in actor-critic methods[J/OL]. (2018-02-26), http//arxiv.org/abs/1802.09477. http//arxiv.org/abs/1802.09477
[24]	采国顺, 刘昊吉, 冯吉伟, 等. 智能汽车的运动规划与控制研究综述[J]. 汽车安全与节能学报, 2021, 12(3): 279-297.
	CAI Guoshun, LIU Haoji, FENG Jiwei, et al. Review on the research of motion planning and control for intelligent vehicles[J]. J Autom Safe Energ, 2021, 12(3): 279-297. (in Chinese)
[25]	CAO Zhong, XU Shaobing, PENG Huei, et al. Confidence-aware reinforcement learning for self-driving cars[J]. *IEEE Trans Intel Transport Syst*, 2021(99): 1-12.
[26]	龚建伟, 姜岩, 徐威. 无人驾驶车辆模型预测控制[M]. 北京: 北京理工大学出版社, 2014: 5-20.
	GONG Jianwei, JIANG Yan, XU Wei, Model Predictive Control for Self-Driving Vehicles[M]. Beijing: Institute of Technology Press, Beijing, 2014: 5-20. (in Chinese)
[27]	Talebpour A, Mahmassani H S, Hamdar S H. Modeling lane-changing behavior in a connected environment: A game theory approach[J]. *Transport Res Part C*, 2015, 59(7): 216-232.
[28]	Shakarian P, Roos P, Johnson A. A review of evolutionary graph theory with applications to game theory[J]. *Bio Syst, 2012, 107*(2): 66-80.
[29]	Volodymyr M, Koray K, David S, et al. Human-level control through deep reinforcement learning[J]. *Nature, 2019, 518*(7540): 529-533.
[30]	JING Shoucai, HUI Fei, ZHAO Xiangmo, et al. Cooperative game approach to optimal merging sequence and on-ramp merging control of connected and automated vehicles[J]. *IEEE Trans Intel Transport Syst*, 2019 (99): 1-11.
[31]	Ahn K, Rakha H, Trani A, et al. Estimating vehicle fuel consumption and emissions based on instantaneous speed and acceleration levels[J]. *J Transport Engi, 2002, 128*(2): 182-190.

目标车道车辆	换道车辆M
目标车道车辆	m₁（换道）	m₂（不换道）
d_i₁(允许)	[R_M (d_i₁, m₁), R_D (d_i₁, m₁)]	[R_M (d_i₁, m₂), R_D (d_i₁, m₂)]
d_i₂(拒绝)	[R_M (d_i₂, m₁), R_D (d_i₂, m₁)]	[R_M (d_i₂, m₂), R_D (d_i₂, m₂)]

目标车道车辆	换道车辆M
目标车道车辆	m₁（换道）	m₂（不换道）
d_i₁(允许)	[R_M (d_i₁, m₁), R_D (d_i₁, m₁)]	[R_M (d_i₁, m₂), R_D (d_i₁, m₂)]
d_i₂(拒绝)	[R_M (d_i₂, m₁), R_D (d_i₂, m₁)]	[R_M (d_i₂, m₂), R_D (d_i₂, m₂)]

网络	层	维度	激活函数
Actor网络	L₁	(s, 30)	Relu
	L₂	(30, 30)	Relu
	L₃	(30, a)	Tanh
Critic1网络	L₁	(s+a, 30)	Relu
	L₂	(30, 30)	Relu
	L₃	(30, 1)	无
Critic2网络	L₁	(s+a, 30)	Relu
	L₂	(30, 30)	Relu
	L₃	(30, 1)	无

网络	层	维度	激活函数
Actor网络	L₁	(s, 30)	Relu
	L₂	(30, 30)	Relu
	L₃	(30, a)	Tanh
Critic1网络	L₁	(s+a, 30)	Relu
	L₂	(30, 30)	Relu
	L₃	(30, 1)	无
Critic2网络	L₁	(s+a, 30)	Relu
	L₂	(30, 30)	Relu
	L₃	(30, 1)	无

参数名称	参数值
训练回合数	500
每回合最大步数	200
批量大小	64
经验池容量	104
折扣系数	0.99
Actor网络学习率	3×10^-4
Critic网络学习率	3 ×10^-4
软更新速率	1×10^-3
动作探索噪声	0.1
策略噪声	0.5
延迟更新频率	2
优化器类型	Adam