Don't let him get started. Be smart, be firm, say no. He will respect you if you respect yourself.
Unless you want to take advantage of this as gold digger, then be it, be a pushover.
I agree with one say 蒼蠅不盯無縫的蛋!. I am sure there is a reason why he chose you as a target of harassment. If some physical relationship happens between you and him, you are partially responsible, I am sure.