you are only responsible for your supervisor!
Tell everybody else that you do not have the right to perform any work assigned by anybody except your supervisor!
if somebody else asked you to do some kind work, just say: Please ask my supervisor first, if my supervisor ask me to do it, then I can take the responsibility for the assignment.
After a while by doing this way, nobody will come to bother you anymore!
That is my expereince, which will work for your situation.