It’s not really letting go of the past that is the problem. It is coming to terms with letting go of a possible future that will never be. That is the struggle. The mind wants to keep its fantasies. Even when they are wrong, unhealthy, dangerous, or even cruel. To let go of the past you must let go of the future and live in the present.