Need a clarification on what you said. Does that heaviness comes from God so that i won't go back there or is it from evil spirit ?
You've got to identify what that heaviness is.
If you did any kind of fornication- fornication is particularly difficult to shake because it can take the genuine love for that person, and use it to remind you of sin for the rest of your life. But how you respond to that reminder is what makes the difference.