Is he interested in only hanging out with you sometimes or have some sexiual attempts? I was in a similar situation but the manager was just like hanging out with me - certainly, much more than what I wanted so it was a big pressure to me. I left the company.]
Good luck!