From your description, he is abusing you. And he is a two face.
It should be your immediate supervisor's responsibility to check/verify your leave balance.