Fast RCNN loss doesn't change when trained with my own dateset. Not fine tuned on pretrained model

[xiaoxiangyeyuwangye]

Hi, everyone
I am training Faster RCNN with my own dataset. It is trained from origin, not fine tuned on pretrained model.
In my experiment, when i did not initial the weight from a pretrained model, the rpn network was trained well and converged, but the fast rcnn network did not converge. The loss of it even did not change.
Does anyone have any idea about this.