Markov 链与Q-Learning算法的超轻度混动汽车模型预测控制

doi:10.3969/j.issn.1674-8484.2021.04.015

摘要/Abstract

摘要：

为了同时兼顾能量管理策略的全局最优性与运算实时性,本文提出了基于Markov 链与Q-Learning算法的超轻度混合动力汽车模型预测控制能量管理策略。采用多步Markov模型预测加速度变化过程,计算得出混合动力汽车未来需求功率;以等效燃油消耗最小与动力电池荷电状态（SOC）局部平衡为目标函数,建立能量管理策略优化模型;采用Q-Learning算法对预测时域内的优化问题进行求解,得到最优转矩分配序列。基于MATLAB/Simulink平台,对于ECE_EUDC+UDDS循环工况进行仿真分析。结果表明：采用Q-Learning求解的控制策略比基于动态规划（DP）求解的控制策略,在保证燃油经济性基本保持一致的前提下,仿真时间缩短了4 s,明显地提高了运行效率,实时性更好。

关键词: 超轻度混合动力汽车, 模型预测控制, Markov链（Markov chain）, Q-Learning算法, 多步Markov模型, 能量管理

Abstract:

A model predictive control energy management strategy for super-light hybrid electric vehicles (HEV) was proposed to take into account the global optimality of the energy management strategy and the real-time operation at the same time based on Markov chain and Q-Learning algorithm. The multi-step Markov model was used to predict the acceleration change process to calculate the future required power of HEV. An energy management strategy optimization model was established by taking the minimum equivalent fuel consumption and the local balance of the state of charge (SOC) of power battery as the objective function. The Q-learning algorithm was used to solve the optimization problem in the prediction time domain to obtain the optimal torque distribution sequence. The simulation analysis was carried out under the ECE_EUDC+UDDS cycle conditions on the base of MATLAB / Simulink platform. The results show that the control strategy solved by the Q-Learning solution reduces the simulation time by 4 s under the same fuel economy condition, comparing with the control strategy based on the dynamic programming (DP) solution. The proposed control strategy can significantly improve the operating efficiency and has better real-time performance.

Key words: super-light hybrid electric vehicles, model predictive control, Markov chain, Q-learning algorithm, multi-step Markov model, energy management

中图分类号:

U469.72

尹燕莉, 马永娟, 周亚伟, 王瑞鑫, 詹森, 马什鹏, 黄学江, 张鑫新. Markov 链与Q-Learning算法的超轻度混动汽车模型预测控制[J]. 汽车安全与节能学报, 2021, 12(4): 557-569.

YIN Yanli, MA Yongjuan, ZHOU Yawei, WANG Ruixin, ZHAN Sen, MA Shenpeng, HUANG Xuejiang, ZHANG Xinxin. Model predictive control of super-mild hybrid electric vehicle based on Markov chain and Q-Learning[J]. Journal of Automotive Safety and Energy, 2021, 12(4): 557-569.

图/表 14

图1 超轻度混合动力汽车结构简图 111L0：离合器;L1、L2、L3：湿式离合器;L4：单向离合器;B：制动器

部件	参数名称	数值
发动机	发动机类型额定功率峰值转矩	JL472Q1 (49 ±2.45) kW (82 ± 4.1) Nm
电动机	额定功率额定转速	5 kW 1 000 r/min
电池	额定容量额定电压	6.5 A·h 80 V
变速器	最大速比最小速比主减速器传动比	0.498 4.04 4.99
其他参数	整车质量迎风面积轮胎半径空气阻力系数滚动阻力系数	1 190 kg 1.7 m² 0.274 m 0.32 0.014

表1 整车主要参数

图2 发动机转矩模型

图3 电动机转矩模型

图4 模型预测控制原理图

图5 v = 35 km/h时加速度的多步转移概率

预测时域p中n_p	均方根误差Re
3	1.858 6
5	3.135 5
7	4.488 1
9	5.892 1

表2 多步Markov不同预测时域的均方根误差

图6 v = 35 km/h时多步Markov模型车速预测效果图

图7 滚动优化示意图

图8 Markov链+Q-Learning控制策略优化流程

图9 优化后的发动机转矩

图10 优化后的电动机转矩

图11 Markov链+Q-Learning、Markov链+DP控制策略的仿真结果

图12 Markov链 + Q-Learning、Markov链 + DP控制策略的发动机和电机工作点

参考文献 35

[1]	Kim H, Kum D. Comprehensive design methodology of input and output-split hybrid electric vehicles: In search of optimal configuration[J], *IEEE/ASME Trans Mechatronic*, 2016, 21:2912-2923. doi: 10.1109/TMECH.2016.2579646 URL
[2]	Kang J. Enabling localized peer-to-peer electricity trading among plug-in hybrid electric vehicles using consortium blockchains[J]. *IEEE Trans Ind Info*, 2017, 13:245-346.
[3]	苏岭, 曾育平, 秦大同. 插电式混合动力汽车能量管理策略研究现状和发展趋势[J]. 重庆大学学报, 2017, 40(2):10-15.
	SU Ling, ZENG Yuping, QING Datong. Research status and development trends in plug-in hybrid automobile energy management strategies[J]. J Chongqing Univ, 2017, 40(2):10-15. (in Chinese)
[4]	周能辉, 赵春明, 辛明华, 等. 插电式混合动力轿车整车控制策略的研究[J]. 汽车工程, 2013, 35(2):99-104.
	ZHOU Nenghui, ZHAO Chunming, XIN Minghua, et al. Research on vehicle control strategy of plug - in hybrid electric vehicle[J]. Autom Engi, 2013, 35(2):99-104. (in Chinese)
[5]	ZHAO Guangyao, CHEN Zeyu, et al. Energy management strategy for series hybrid electric vehicle[J]. *J Northeastern Univ*, 2013, 34:583-587.
[6]	LI S G, Sharkh S M, Walsh F C, et al. Energy and battery management of a plug-In series hybrid electric vehicle using fuzzy logic[J]. *IEEE Trans Vehi Tech*, 2011, 60(8):3571-3585.
[7]	PENG Jiankun, HE Hongwen, XIONG Rui. Rule based energy management strategy for a series-parallel plug-in hybrid electric bus optimized by dynamic programming[J]. *Appl Energ, 2017, 185*:224-256.
[8]	Jihun Han, Dongsuk Kum, Youngjin Park. Synjournal of predictive equivalent consumption minimization strategy for hybrid electric vehicles based on closed form solution of optimal equivalence factor[J]. *IEEE Trans Vehi Tech*, 2017, 66:5604-5616.
[9]	QIN Datong, ZHAN Seng, SU Ling. Management strategy of hybrid electrical vehicle based on driving style recognition[J]. *J Mech Engi*, 2016, 52:162-169.
[10]	LIN Xinyou, FENG Qigao, MO Liping, et al. Optimal adaptation equivalent factor of energy management strategy for plug-in CVT HEV[J]. *Proc IMECHE Part D; J Autom Engi, 2019, 233*:877-889.
[11]	王慧璇. 基于动态规划的 Plug-in混合动力汽车能量管理策略优化研究[D]. 济南:山东大学, 2012.
	WANG Huixuan. Research on plug-in hybrid electric vehicles energy management strategy based on dynamic planning[D]. Jinnan: Shan Dong Univ, 2012. (in Chinese)
[12]	ZHANG Shuo, XIONG Rui, SUN Fengchun, et al. Model predictive control for power management in a plug-in hybrid electric vehicle with hybrid energy storage system[J]. *Appl Energ, 2017, 185*:1654-1662. doi: 10.1016/j.apenergy.2015.12.035 URL
[13]	GUO Jinquan. A novel MPC-based adaptive energy management strategy in plug-in hybrid electric vehicles[J]. *Energy. 2019, 175*:378-392. doi: 10.1016/j.energy.2019.03.083
[14]	LI Liang, YOU Sixiong, YANG Chao, et al. Driving-behavior-aware stochastic model predictive control for plug-in hybrid electric buses[J]. *Appl Energ, 2016, 162*:868-879. doi: 10.1016/j.apenergy.2015.10.152 URL
[15]	曾祥瑞, 黄开胜, 孟凡博. 具有实时运算潜力的并联混合动力汽车模型预测控制[J]. 汽车安全与节能学报, 2012, 3(2):65-172.
	ZENG Xiangrui, HUANG Kaisheng, MENG Fanbo. Model predictive control for parallel hybrid electric vehicles with potential real-time capability[J]. J Autom Safe Energ, 2012, 3(2):165-172. (in Chinese)
[16]	赵韩, 吴迪. 基于随机模型预测控制的并联式混合动力汽车控制策略研究[J]. 汽车工程, 2014, 36(11):1289-1294.
	ZHAO Han, WU Di. A research on the control strategy of a parallel HEV based on stochastic model predictive control[J]. Autom Engi, 2014, 36(11):1289-1294. (in Chinese)
[17]	钱立军, 荆红娟, 邱利宏. 基于随机模型预测控制的四驱混合动力汽车能量管理[J]. 中国机械工程, 2018, 29(11):1342-1348.
	QIAN Lijun, JING Hongjuan, QIU Lihong. Energy management of a 4WD HEV based on SMPC[J]. Chin Mech Eng, 2018, 29(11):1342-1348. (in Chinese)
[18]	SUN Chao, Moura S J, HU Xiaosong, et al. Dynamic traffic feedback data enabled energy management in plug-in hybrid electric vehicles[J]. *IEEE Trans Contr Syst Tech*, 2015, 23(3):1075-1086. doi: 10.1109/TCST.2014.2361294 URL
[19]	孙超. 混合动力汽车预测能量管理研究[D]. 北京: 北京理工大学, 2016.
	SUAN Chao. Predictive energy management of hybrid electric vehicle powertrains[D]. Beijing:Beijing Institute of Technology, 2016. (in Chinese)
[20]	Johannesson L, Murgovski N, Jonasson E, et al. Predictive energy management of hybrid long-haul trucks[J]. *Contr Engi Pract*, 2015, 41:83-97.
[21]	钱立军, 邱利宏, 司远, 等. 智能网联环境下的混合动力汽车分层能量管理[J]. 汽车工程, 2017, 39(6):621-629.
	QIAN Lijun, QIU Lihong, SI Yuan, et al. Hierarchical energy management of hybrid electric vehicles under the environment of intelligent and connected vehicles[J]. Autom Engi, 2017, 39(6):621-629. (in Chinese)
[22]	邱利宏, 钱立军, 杜志远, 等. 车联网环境下车辆最优车速闭环快速模型预测控制[J]. 中国机械工程, 2017, 28(10):1245-1252.
	QIU Lihong, QIAN Lijun, DU Zhiyuan, et al. A closed-loop FMPC of optimal velocities for connected vehicles[J]. Chin Mech Engi, 2017, 28(10):1245-1252. (in Chinese)
[23]	ZENG Xiangrui, WANG Junming. A parallel hybrid electric vehicle energy management strategy using stochastic model predictive control with road grade preview[J]. *IEEE Trans Contr Syst Tech*, 2015, 23(6):2416-2423. doi: 10.1109/TCST.2015.2409235 URL
[24]	孙延秋. 基于模型预测控制的HEV能量管理策略的研究[D]. 大连: 大连理工大学, 2017.
	SUN Yanqiu. Energy management strategy of HEV based on model predictive control[D]. Dalian:Dalian University of Technology, 2017. (in Chinese)
[25]	Cairano S D, Bernardini D, Bemporad A, et al. Stochastic MPC with learning for driver-predictive vehicle control and its application to HEV energy management[J]. *IEEE Trans Contr Syst Tech*, 2014, 22(3):1018-1031. doi: 10.1109/TCST.2013.2272179 URL
[26]	秦大同, 秦岭. 基于显示随机模型预测控制的功率分流式混合动力汽车能量管理策略[J]. 华南理工大学学报 (自然科学版), 2019, 47(7):112-120.
	QING Datong, QING Ling . Power shunt hybrid electric vehicle energy management strategy based on display random model prediction control[J]. J Huanan Univ Tech (Nat Sci Edit), 2019, 47(7):112-120. (in Chinese)
[27]	YU Kaijiang, YANG Haizhu, TAN Xingguo, et al. Model predictive control for hybrid electric vehicle platooning using slope information[J]. *IEEE Trans Intel Transport Syst*, 2016, 17(7):1894-1909. doi: 10.1109/TITS.2015.2513766 URL
[28]	ZHANG Jiangyan, SHEN Tielong. Real-time fuel economy optimization with nonlinear MPC for PHEVs[J]. *IEEE Trans Contr Syst Techn*, 2016, 24(6):2167-2175. doi: 10.1109/TCST.2016.2517130 URL
[29]	GUO Lulu, GAO Bingzhao, GAO Ying, et al. Optimal energy management for HEVs in eco-driving applications using bi-level MPC[J]. *IEEE Trans Intel Transport Syst*, 2017, 18(8):2153-2162. doi: 10.1109/TITS.2016.2634019 URL
[30]	XIE Shaobo, HU Xiaosong, XIN Zongke, et al. Pontryagin’s minimum principle based model predictive control of energy management for a plug-in hybrid electric bus[J]. *Appl Energ, 2019, 236*:893-905. doi: 10.1016/j.apenergy.2018.12.032
[31]	WU Jinda, HE Hongwen,, PENG Jiankun, et al. Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus[J]. *Appl Energ*, 2018, 22:799-811.
[32]	QI Xuewei, LUO Yadan. Deep reinforcement learning enabled self-learning control for energy efficient driving[J]. *Transport Res Part C: Emerg Tech*, 2019, 99:67-81. doi: 10.1016/j.trc.2018.12.018 URL
[33]	LIU Teng, WANG Bo, YANG Chenglang. Online Markov chain-based energy management for a hybrid tracked vehicle with speedy Q Learning[J]. *Energy, 2018, 160*:544-555. doi: 10.1016/j.energy.2018.07.022 URL
[34]	YUAN Jingni, YANG Lin, CHEN Qu. Intelligent energy management strategy based on hierarchical approximate global optimization for plug-in fuel cell hybrid electric vehicles[J]. *Int’l J Hydro Energ*, 2018, 43:8063-8078.
[35]	陈虹. 模型预测控制[M]. 北京: 科学出版社, 2013: 1-407.
	CHENG Hong. Model Prediction Control[M]. Beijing: Science Press, 2013: 1-407. (in Chinese)