Lateral velocity estimation for four-wheel-independent-drive electric vehicles based on deep reinforcement learning

doi:10.3969/j.issn.1674-8484.2022.02.011

Abstract

Abstract:

A lateral-velocity estimation method was proposed for an electric vehicle with four-wheel independent-drive to estimate the vehicle motion states precisely. An architecture was designed for the lateral velocity estimation method based on the deep reinforcement learning (DRL) paradigm; A DRL agent was designed with deep deterministic policy gradient (DDPG) algorithm; The actor network and the critic network of the DDPG algorithm were constructed with the recurrent neural network (RNN). The algorithm was realized and trained in Matlab/Simulink with the designed award function and training scenarios; The algorithm effectiveness was verified by the simulation of practical driving maneuvers such as double-lane changing. The results show that after 630 episodes of training and learning, the proposed method improves the estimation accuracy by 40%, compared with that of the extended Kalman filter (EKF) method. Therefore, the proposed method can be used to estimate vehicle lateral velocity in general driving scenarios.

Key words: vehicle dynamics control, four-wheel-independent-drive electric vehicle, vehicle lateral velocity estimation, deep reinforcement learning (DRL), deep deterministic policy gradient (DDPG)

CLC Number:

U461

ZHENG Yangjun, HE Shuai, SHUAI Zhibin, LI Jianqiu, GAI Jiangtao, LI Yong, ZHANG Ying, LI Guohui. Lateral velocity estimation for four-wheel-independent-drive electric vehicles based on deep reinforcement learning[J]. Journal of Automotive Safety and Energy, 2022, 13(2): 309-316.

Figures/Tables 13

References 17

[1]	余卓平, 高晓杰. 车辆行驶过程中的状态估计问题综述[J]. 机械工程学报, 2009, 45(5): 20-33.
	YU Zhuoping, GAO Xiaojie. Review of vehicle state estimation problem under driving situation[J]. J Mech Engi, 2009, 45(5): 20-33. (in Chinese)
[2]	陈慧, 高博麟, 徐帆. 车辆质心侧偏角估计综述[J]. 机械工程学报, 2013, 49(24): 76-94.
	CHEN Hui, GAO Bolin, XU Fan. Review on vehicle sideslip angle estimation[J]. J Mech Engi, 2013, 49(24): 76-94. (in Chinese)
[3]	郭洪艳, 陈虹, 赵海艳, 等. 汽车行驶状态参数估计研究进展与展望[J]. 控制理论与应用, 2013, 30(6): 661-672.
	GUO Hongyan, CHEN Hong, ZHAO Haiyan, et al. State and parameter estimation for running vehicle: recent developments and perspective[J]. Contr Theo Appl, 2013, 30(6): 661-672. (in Chinese)
[4]	金贤建. 分布式驱动电动汽车状态参数估计与侧向稳定性鲁棒控制研究[D]. 南京: 东南大学, 2017.
	JIN Xianjian. Parameter estimation and robust control of lateral stability for distributed drive electric vehicles[D]. Nanjing: Southeast University, 2017. (in Chinese)
[5]	樊东升. 基于联邦卡尔曼滤波的分布式驱动电动汽车状态与参数估计[D]. 锦州: 辽宁工业大学, 2020.
	FAN Dongsheng. State and parameters estimation of distributed drive electric vehicle based on federal Kalman filter[D]. Jinzhou: Liaoning University of Technology, 2020. (in Chinese)
[6]	肖峰. 轮穀电机驱动电动汽车状态估计及直接横摆力矩控制研究[D]. 长春: 吉林大学, 2016.
	XIAO Feng. Study on state estimation and direct yaw moment control for in-wheel motor electric vehicles[D]. Changchun: Jilin University, 2016. (in Chinese)
[7]	施树明, Lupker H, Bremmer P, 等. 基于模糊逻辑的车辆侧偏角估计方法[J]. 汽车工程, 2005, 27(4): 426-430.
	SHI Shuming, Lupker H, Bremmer P, et al. Estimation of vehicle side slip angle based on fuzzy logic[J]. Automotive Engineering, 2005, 27(4): 426-430. (in Chinese)
[8]	张凤娇, 汪䶮, 赵万忠. 基于深度学习的极限工况下车辆的状态估计[J]. 重庆理工大学学报(自然科学), 2018, 32(10): 64-70.
	ZHANG Fengjiao, WANG Yan, ZHAO Wanzhong. Vehicle state estimation of vehicle under limiting conditions based on deep learning[J]. J Chongqing Univ Technology (Nat Sci), 2018, 32(10): 64-70. (in Chinese)
[9]	汪䶮, 林懿伦, 戴星原, 等. 车辆关键状态的平行估计[J]. 指挥与控制学报, 2017, 3(3): 186-194.
	WANG Yan, LIN Yilun, DAI Xingyuan, et al. Parallel estimation of vehicle state[J]. Journal of Command and Control, 2017, 3(3): 186-194. (in Chinese)
[10]	Ribeiro A M, Moutinho A, Fioravanti A R, et al. Estimation of tire-road friction for road vehicles: a time delay neural network approach[J]. J Brazilian Soci Mech Sci Eng, 2020, 42(1): 4.
[11]	HAN Xuefeng, HE Hongwen, WU Jingda, et al. Energy management based on reinforcement learning with double deep Q-learning for a hybrid electric tracked vehicle[J]. Appl Energy, 2019, 254: 113708. doi: 10.1016/j.apenergy.2019.113708 URL
[12]	李文礼, 张友松, 韩迪, 等. 基于深度强化学习的车辆自主避撞决策控制模型[J]. 汽车安全与节能学报, 2021, 12(2): 201-209.
	LI Wenli, ZHANG Yousong, HAN Di, et al. Vehicle autonomous collision avoidance decision control model based on deep reinforcement learning[J]. J Auto Safe Energy, 2021, 12(2): 201-209. (in Chinese)
[13]	高洪森, 游国栋, 王雪, 等. 基于深度强化学习卡尔曼滤波锂离子电池SOC估计[J]. 天津科技大学学报, 2020, 35(4): 65-69.
	GAO Hongsen, YOU Guodong, WANG Xue, et al. Estimating the state of charge of Li-ion batteries based on reinforcement learning of Kalman filters[J]. J Tianjin Univ Sci Tech, 2020, 35(4): 65-69. (in Chinese)
[14]	Kim T, Kim H J. Path tracking control and identification of tire parameters using on-line model-based reinforcement learning[C]// 2016 16th Int’l Conf Contr Automa Syst (ICCAS 2016), Gyeongju, Korea, 2016: 215-219.
[15]	WANG Pengyue, LI Yan, Shekhar S, et al. Uncertainty estimation with distributional reinforcement learning for applications in intelligent transportation systems: a case study[C]// 2019 IEEE Intell Transp Syst Conf (ITSC), Auckland, New Zealand, 2019: 3822-3827.
[16]	樊东升, 李刚. 分布式驱动电动汽车行驶状态估计综述[J]. 汽车实用技术, 2019(16): 10-12.
	FAN Dongsheng, LI Gang. Review on driving state estimation of distributed drive electric vehicles[J]. Auto Appl Tech, 2019(16): 10-12. (in Chinese)
[17]	Lillicrap T P, Hunt J J, Pritzel A, et al. Continuous control with deep reinforcement learning[C]// 2016 4th Int’l Conf Learning Repres (ICLR 2016), San Juan, Puerto Rico, 2016: 1-14.

整车质量	1 181 kg
质心至前轴距离	1.491 m
质心至后轴距离	1.529 m
轮距	1.713 m
车轮有效滚动半径	0.309 m

整车质量	1 181 kg
质心至前轴距离	1.491 m
质心至后轴距离	1.529 m
轮距	1.713 m
车轮有效滚动半径	0.309 m

输入信号	信号特征
输入信号	δ / (°)	f / Hz
正弦波1	120	1.0
正弦波2	150	1.0
正弦波3	200	1.0
正弦波4	120	0.5
正弦波5	150	0.5
正弦波6	200	0.5

输入信号	信号特征
输入信号	δ / (°)	f / Hz
正弦波1	120	1.0
正弦波2	150	1.0
正弦波3	200	1.0
正弦波4	120	0.5
正弦波5	150	0.5
正弦波6	200	0.5

Critic网络参数学习率	10^-3
Actor网络参数学习率	10^-4
采样时间步长	20 ms
对未来奖励的折扣因子	0.99
最小批的数量（mini-batch size）	250
经验池长度	10⁶
网络参数更新的平顺因子	10^-3