Frankly, this was not a good way to handle such situation since it is difficult to build up your reputation but easy to ruin it.
One of ways of handling such situation can be leaving your desk and logout out your computer while you were talking to him.
One of ways of handling such situation can be leaving your desk and logout out your computer while you were talking to him.