Asking you out for work lunch is not considered harassment

If you do not want to go, just say no. Although refusing to go to lunch with your supervisor sounds like a career limiting move. Did he do anything else, other than asking you out for lunch, that might be perceived as sexual harassment? Regardless, you should probably line up other job opportunities.