If the element that your controller is instantiated on is removed from the DOM then all event listeners on that instance will be destroyed. It doesn't sound like that is happening though because you said a generic $ selector hooked into the events.
I'd suggest instantiating your controller on an ancestral element that is not removed from the DOM - then you can retain your scoped event listeners.