I don't think it is point

It seems that kind of REF let you describe that why you don't apply PERM. You cannot use only that your position is not fit PERM to answer this question. Your citiation and papers are important but not enough.

Usually, I think you should describe what unique you are. You can say that your position is easy to find canadiate (such as other PhD), however, you have special skill and you will have contributed more. So perm not fit you because a lot of American PhD can apply same position.