Tell him the true ones and tell him that

your focus is the efficiency of the whole team, if I'm the only one who can do it efficiently, or if you need me to do it because no one else wants to do it, I'll do it with my best. Or explain more on why you don't like it, e.g. I've done this so long. But if he sticks on it too much, give up this position because it's simply not your best interest.