The time too leave was after he stopped talking to you during such an emotionally draining time in your life. Tell him to fuck off. You dont need his shit.