It seems that you are in a seem-complicated situation now. As I am not in the situation, I only have some thoughts:
1. It seems to me that it could be the politics between your big boss and small boss and you are the victim and tool used by the small boss?
2. Since you are not in the same department with the big boss any more, you are fine - the gossip will go away with time moving on. Just put your head down and focus on your work - a few months later or even sooner, people will start to talk something else. After all, most likely you were not even the target as a lower level employee - you don't even have much value any more with your changing department. But, if you keep worrying so as to take some more actions, you can be in a real trouble at work.
Cheer up and enjoy more important things in your work and life - everything will be fine.
