基于强化学习的控制系统仿真（相关搜索：强化学习）

Question

目前正在做基于DDPG算法的双有源全桥dcdc变换器的调制策略，目前程序可以运行，但是action不会根据reward的变化而变化

```c
% 打开模型
open_system('SPSDAB')
%设置观测信息
obsInfo = rlNumericSpec([3 1],...
    'LowerLimit',[-inf -inf 0]',...
    'UpperLimit',[inf inf inf]');
obsInfo.Name = 'observations';
obsInfo.Description = 'integrated error, error, and measured height';
numObservations = obsInfo.Dimension(1);

% 设置动作信息
actInfo = rlNumericSpec([1 1]);
actInfo.Name = 'Transfer ratio';
numActions = actInfo.Dimension(1);

% 创建环境对象
env = rlSimulinkEnv('SPSDAB','SPSDAB/RL Agent',...
    obsInfo,actInfo);
env.ResetFcn = @(in)localResetFcn(in);

Ts = 1.0;
Tf = 200;
rng(0)

% 定义评论网络结构
statePath = [
    featureInputLayer(numObservations,'Normalization','none','Name','State')
    fullyConnectedLayer(64,'Name','CriticStateFC1')
    reluLayer('Name','CriticRelu1')
    fullyConnectedLayer(32,'Name','CriticStateFC2')];
actionPath = [
    featureInputLayer(numActions,'Normalization','none','Name','Action')
    fullyConnectedLayer(32,'Name','CriticActionFC1')];
commonPath = [
    additionLayer(2,'Name','add')
    reluLayer('Name','CriticCommonRelu')
    fullyConnectedLayer(1,'Name','CriticOutput')];

criticNetwork = layerGraph();
criticNetwork = addLayers(criticNetwork,statePath);
criticNetwork = addLayers(criticNetwork,actionPath);
criticNetwork = addLayers(criticNetwork,commonPath);
criticNetwork = connectLayers(criticNetwork,'CriticStateFC2','add/in1');
criticNetwork = connectLayers(criticNetwork,'CriticActionFC1','add/in2');

% 定义评论网络选项
criticOpts = rlRepresentationOptions('LearnRate', 1e-03, 'GradientThreshold', 1);
critic = rlQValueRepresentation(criticNetwork, obsInfo, actInfo, 'Observation', {'State'}, 'Action', {'Action'}, criticOpts);

% 定义演员网络结构
actorNetwork = [
    featureInputLayer(numObservations, 'Normalization', 'none', 'Name', 'State')
    fullyConnectedLayer(64, 'Name', 'actorFC1')
    reluLayer('Name', 'actorRelu1')
    fullyConnectedLayer(32, 'Name', 'actorFC2')
    reluLayer('Name', 'actorRelu2')
    fullyConnectedLayer(numActions, 'Name', 'Action')
    ];

% 演员网络选项
actorOptions = rlRepresentationOptions('LearnRate', 1e-04, 'GradientThreshold', 1);
actor = rlDeterministicActorRepresentation(actorNetwork,obsInfo,actInfo,'Observation',{'State'},'Action',{'Action'},actorOptions);

% 定义代理对象选项
agentOpts = rlDDPGAgentOptions(...
    'SampleTime',Ts,...
    'TargetSmoothFactor',1e-3,...
    'DiscountFactor',1.0, ...
    'MiniBatchSize',128, ...
    'ExperienceBufferLength',1e6); 
agentOpts.NoiseOptions.Variance = 0.3;
agentOpts.NoiseOptions.VarianceDecayRate = 1e-5;

agent = rlDDPGAgent(actor, critic, agentOpts);

maxepisodes = 5000;
maxsteps = ceil(Tf/Ts);

% 训练选项
trainOpts = rlTrainingOptions(...
    'MaxEpisodes', maxepisodes, ...
    'MaxStepsPerEpisode', maxsteps, ...
    'ScoreAveragingWindowLength', 20, ...
    'Verbose', false, ...
    'Plots', 'training-progress',...
    'StopTrainingCriteria', 'AverageReward',...
    'StopTrainingValue', 800);

doTraining = false;

if doTraining
    % 训练代理
    trainingStats = train(agent, env, trainOpts);
else
    % 加载预训练代理
    load('SPSDABDDPG.mat', 'agent')
end

```

Answer 1

引自免费微信小程序：皆我百晓生

在基于DDPG（Deep Deterministic Policy Gradient）算法的双有源全桥DC-DC变换器调制策略仿真中，action未能随reward变化而优化的问题可能涉及多个方面。以下是针对该问题的潜在解决方案和调试步骤：

1. 检查网络结构与参数：确保DDPG算法中的Actor网络和Critic网络结构正确无误，并且学习率、批大小、神经网络层数和节点数等超参数设置合理。例如：

actor_net = layerGraph([featureInputLayer(num_features), fullyConnectedLayer(hidden_units), ...
    reluLayer, fullyConnectedLayer(output_units), tanhLayer]);
critic_net = layerGraph([featureInputLayer(num_features), fullyConnectedLayer(hidden_units), ...
    reluLayer, fullyConnectedLayer(hidden_units), reluLayer, ...
    fullyConnectedLayer(1)]);

此处num_features是状态空间维度，hidden_units是隐藏层节点数，output_units是动作空间维度。

1. 奖励函数设计：确保奖励函数能够有效引导agent学习到期望的调制策略。应根据变换器的实际工作性能指标来设计，如输出电压稳定度、效率、开关频率等。当状态或动作不理想时，应给予负奖励，反之则给予正奖励。

2. 探索与利用平衡：DDPG使用了 Ornstein-Uhlenbeck 过程作为探索噪声，需适当调整其参数，如噪声强度（theta），以保证在探索和利用之间取得良好平衡。

3. 经验回放缓存与批量更新：确认是否正确实现了经验回放缓存（Replay Buffer），并定期从缓冲区中随机抽取样本进行批量更新：

replay_buffer = circularArrayQueue(max_size);
...
mini_batch = replay_buffer.getBatch(batch_size);
state_mb = mini_batch(:, 1:num_features);
next_state_mb = mini_batch(:, num_features+1:end-num_actions);
action_mb = mini_batch(:, end-num_actions+1:end-1);
reward_mb = mini_batch(:, end); % 或者更复杂的奖励计算
...

1. 策略更新与目标网络同步：在训练过程中，确保Actor网络和Critic网络的目标网络定期与主网络进行软更新：

target_actor_net = cloneNetwork(actor_net);
target_critic_net = cloneNetwork(critic_net);
...
updateTargetNetwork(target_actor_net, actor_net, tau);
updateTargetNetwork(target_critic_net, critic_net, tau);

其中tau为目标网络更新率。

1. 训练步数与收敛性：可能需要增加训练迭代次数以让模型充分学习和收敛。同时，监测loss曲线和reward曲线有助于判断训练过程是否正常。

请根据实际情况对照上述建议进行排查和修改。由于Matlab环境下的代码实现依赖于具体的库和接口，以上代码仅为示例，具体实现可能需要参考Matlab Reinforcement Learning Toolbox或其他相关库。