not very appropriate. but I think you should go

if he can have some impact on yoru future,especially if he is a person in authority and accustomed to people obey him. as far as you stay put, nothing can happen. But of cause you need to sweet talk with him to make him comfortable but still protect yourself. good thing is he is not your own boss. so impact is not that big whichever way you choose.

If you and your boss are in confidence, then you can let your own boss know about this, in case sth happened in the future, at least you have someone on your side.