Highway lane change decision control model based on deep reinforcement learning

doi:10.3969/j.issn.1674-8484.2022.04.016

Abstract

Abstract:

A lane change tracking control model was proposed based on the deep reinforcement learning algorithm and simulation experiments were carried out to solve the problem of safe lane change for automatic driving vehicles on highways. A model of the vehicle lane change path was built by using a quintuple polynomial approach with the tracking error functions. A three-degree-of-freedom vehicle dynamics model was fused with the deep reinforcement learning framework to build the lane change path tracking control model, which was updated by a deep deterministic policy gradient (DDPG) algorithm. The steering angle was learned for optimal lane change path tracking to control the vehicle complete the lane change process. The results show that at a speed of 100 km/h, the maximum value of the lateral position error absolute value is close to 0 with the maximum value of the angular deviation absolute value of 10 mrad controlled by using the proposed method; The lateral position error and angular error by using the proposed trajectory tracking method are smaller than that by using the traditional model prediction control method. Therefore, this model can achieve the lane change process autonomously in a high-speed environment, which is meaningful for ensuring traffic safety and alleviating traffic.

Key words: automatic driving vehicles, lane change model, path tracking, deep reinforcement learning, quintic polynomials, deep deterministic policy gradient (DDPG) algorithm

CLC Number:

U471.15

LI Wenli, QIU Fanke, LIAO Daming, REN Yongpeng, YI Fan. Highway lane change decision control model based on deep reinforcement learning[J]. Journal of Automotive Safety and Energy, 2022, 13(4): 750-759.

Figures/Tables 15

References 16

[1]	金立生, van Arem B, 杨双宾, 等. 高速公路汽车辅助驾驶安全换道模型[J]. 吉林大学学报(工学版), 2009, 39(3): 582-586.
	JIN Lisheng, van Arem B, YANG Shuangbin, et al. Model of auxiliary safety lane change of expressway vehicle[J]. J Jilin Univ (Eng Sci), 2009, 39(3): 582-586. (in Chinese)
[2]	李玮, 王晶, 段建民, 等. 基于多项式的智能车辆换道轨迹规划[J]. 计算机工程与应用, 2012, 48(3): 242-245.
	LI Wei, WANG Jing, DUAN Jianmin, et al. Trajectory planning of intelligent vehicle lane change based on polynomial[J]. *Computer Engi Appl*, 2012, 48(3): 242-245. (in Chinese)
[3]	WANG Chang, ZHENG Chuqing. Lane change trajectory planning and simulation for intelligent vehicle[C]. *Adva Mat Res, 2013, 671* (3): 2843-2846.
[4]	杨刚. 基于车车通信的多车协同自动换道控制策略研究[D]. 北京: 清华大学, 2016.
	YANG Gang. Research on multi-vehicle collaborative automatic lane change control strategy based on vehicle-vehicle communication[D]. Beijing: Tsinghua University. 2016. (in Chinese)
[5]	闫尧, 李春书, 唐风敏, 等. 基于五次多项式模型的自主车辆换道轨迹规划[J]. 机械设计, 2019, 36(8): 42-47.
	YAN Yao, LI Chunshu, TANG Fengmin, et al. Trajectory planning of autonomous vehicle lane change based on pentatonic polynomial model[J]. *Mechanical Design*, 2019, 36(8): 42-47. (in Chinese)
[6]	陈虹, 郭露露, 宫洵, 等. 智能时代的汽车控制[J]. 自动化学报, 2020, 46(7): 1313-1332.
	CHEN Hong, GUO Lulu, GONG Xun, et al. Automotive control in intelligent era[J]. *Acta Automatica Sinica*, 2020, 46(7): 1313-1332. (in Chinese)
[7]	WANG Pin, CHAN Chingyao. A reinforcement learning based approach for automated lane change maneuver[C]// IEEE Intell Vehi Symp (IV), 2018, 21(10): 1379-1384.
[8]	贺伊琳, 宋若旸, 马建, 等. 基于强化学习DDPG的智能车辆轨迹跟踪控制[J]. 中国公路学报, 2021, 34(11): 335-348. doi: 10.19721/j.cnki.1001-7372.2021.11.026
	HE Yilin, SONG Ruoyang, MA Jian, et al. Intelligent vehicle trajectory tracking control based on reinforcement learning DDPG[J]. *Chin J Highw Transp*, 2021, 34(11): 335-348. (in Chinese)
[9]	裴晓飞, 莫烁杰, 陈祯福, 等. 基于TD3算法的人机混架交通环境自动驾驶汽车换道研究[J]. 中国公路学报, 2021, 34(11): 246-254. doi: 10.19721/j.cnki.1001-7372.2021.11.020
	PEI Xiaofei, MO Shuojie, CHEN Zhenfu, et al. Research on lane change of autonomous vehicle based on TD3 algorithm in human-machine hybrid traffic environment[J]. *Chin J Highw Transp*, 2021, 34(11): 246-254. (in Chinese)
[10]	刘全, 翟建伟, 章宗长, 等. 深度强化学习综述[J]. 计算机学报, 2018, 41(1): 1-27.
	LIU Quan, ZHAI Jianwei, ZHANG Zongchang, et al. Review of deep reinforcement learning[J]. *J Computer Sci*, 2018, 41(1): 1-27. (in Chinese)
[11]	朱冰, 蒋渊德, 赵德, 等. 基于深度强化学习的车辆跟驰控制[J]. 中国公路学报, 2019, 32(6): 54-60.
	ZHU Bing, JIANG Yuande, ZHAO De, et al. Vehicle follow-up control based on deep reinforcement learning[J]. *Chin J Highw Transp*, 2019, 32(6): 54-60. (in Chinese)
[12]	张志勇, 龙凯, 杜荣华, 等. 自动驾驶汽车高速超车轨迹跟踪协调控制[J]. 汽车工程, 2021, 42(7): 995-1004.
	ZHANG Zhiyong, LONG Kai, DU Ronghua, et al. Coordinated tracking and control of high-speed overtaking trajectory of autonomous vehicles[J]. *Automotive Eng*, 2021, 42(7): 995-1004. (in Chinese)
[13]	ZHANG Rui, ZHANG Zhichao, GUAN Zhiwei, et al. Autonomous lane changing control for intelligent vehicles[J]. *Cluster Computing*, 2019, 22(4): 8657-8667.
[14]	聂枝根, 王万琼, 赵伟强, 等. 基于轨迹预瞄的智能汽车变道动态轨迹规划与跟踪控制[J]. 交通运输工程学报, 2020, 20(2): 147-160.
	NIE Zhigen, WANG Wanqiong, ZHAO Weiqiang, et al. Dynamic trajectory planning and tracking control of intelligent automobile lane change based on trajectory pre-aiming[J]. *J Transp Engi*, 2020, 20(2): 147-160. (in Chinese)
[15]	李文礼, 张友松, 韩迪, 等. 基于深度强化学习的车辆自主避撞决策控制模型[J]. 汽车安全与节能学报, 2021, 12(2): 201-209.
	LI Wenli, ZHANG Yousong, HAN Di, et al. Control model of vehicle autonomous collision avoidance decision based on deep reinforcement learning[J]. *J Auto Safe Energy*, 2021, 12(2): 201-209. (in Chinese)
[16]	段建民, 田晓生, 夏天, 等. 基于模型预测控制的智能汽车目标路径跟踪方法研究[J]. 汽车技术, 2017(8): 6-11.
	DUAN Jianmin, TIAN Xiaosheng, XIA Tian, et al. Research on target path tracking method of intelligent vehicle based on model predictive control[J]. *Auto Tech*, 2017(8): 6-11. (in Chinese)

质心离前轴的距离, l_f / m	1.40
质心离后轴的距离, l_f / m	1.60
整车质量, m / t	1.60
绕z轴转动惯量, I_z / (t·m)	2.875
前轮胎侧偏刚度, C_f / (t·rad^-1)	19.0
后轮胎侧偏刚度, C_r / (MN·rad^-1)	33.0
前轮转角范围/ rad	[0,0.26]
初始位置(x,y) / m	(0,0)
自车初始速度, v₀ / (m·s^-1)	15 ~ 30

质心离前轴的距离, l_f / m	1.40
质心离后轴的距离, l_f / m	1.60
整车质量, m / t	1.60
绕z轴转动惯量, I_z / (t·m)	2.875
前轮胎侧偏刚度, C_f / (t·rad^-1)	19.0
后轮胎侧偏刚度, C_r / (MN·rad^-1)	33.0
前轮转角范围/ rad	[0,0.26]
初始位置(x,y) / m	(0,0)
自车初始速度, v₀ / (m·s^-1)	15 ~ 30

参数	描述	数值
折扣因子	用于计算长期折扣奖励	0.99
Actor学习率	策略梯度更新步长	5×10^-4
Critic学习率	Q函数网络梯度下降更新步长	1×10^-3
批量大小	一次批量梯度下降中样本数量	64
经验池大小	用于样本储存	1×105
软更新系数, τ	Polyak移动平均衰减系数	1×10^-4
最大训练回合数	每次最大训练的回合	380
噪声方差	OU噪声，用于进行探索	0.1
噪声衰减率	动作噪声线性衰减程度	1×10^-5

参数	描述	数值
折扣因子	用于计算长期折扣奖励	0.99
Actor学习率	策略梯度更新步长	5×10^-4
Critic学习率	Q函数网络梯度下降更新步长	1×10^-3
批量大小	一次批量梯度下降中样本数量	64
经验池大小	用于样本储存	1×105
软更新系数, τ	Polyak移动平均衰减系数	1×10^-4
最大训练回合数	每次最大训练的回合	380
噪声方差	OU噪声，用于进行探索	0.1
噪声衰减率	动作噪声线性衰减程度	1×10^-5