I think you should make sure what intention he has. If indeed he is upset about your leaving, try to soothe him if possible. But let him know that your are not afraid of him.