one possible explanation is that he is considering moving you up to replace your boss. every new senior manager will need to build his/her own trustworthy team. He may think that you are a good candidate for his inner circle. Your boss obviously belongs to the old generation. it is time to change.
If he is thinking about having an affair, you can pretend that you don't get his hints and subtleties. It is a tricky business to handle a relationship like this. you most likely will have to move on eventually in this case. Moving up would be a chip in his hand. if you don't agree to the affair, then you will not get it.
You are the only one who knows how to interpret the situation. give him the benefit of doubts and see how it plays out. not sure about the level he is in, but if he is in a very senior position, I would bet on the first scenario because he has put himself into a very awkward position if the second is true. A senior person could be unskillful in recruiting advocates, but should not be that stupid.