Lreg is always oscillatory and can‘t converge

[DrewdropLife]

I try to use the KL loss in Mask R-CNN, though the loss was small at first(Lreg = 0.0256), it didn't change much after training 50000 times(Lreg = 0.0223). The loss is usually 0.0002 when KL is not used after training 50000 times.
The learning rate is 0.002, and I also try to use 0.0002. The learning rate decreases when training after 20000 and 40000 ,decay_gamma = 0.1, max_steps = 50000 .
Here is my KL code.

reg_target = self.bbox_transform_inv_xyxy(proposals[0].bbox, coor_target.bbox)
variance = variance[sampled_pos_inds_subset]
kl_loss = torch.exp(-variance)*smooth_l1_loss(box_regression[sampled_pos_inds_subset[:, None], map_inds],
                                              reg_target[sampled_pos_inds_subset], 1/9, KL=True) + variance/2
box_loss = kl_loss.mean(0).sum()*0.1

Thank you for your help!