If you had given a full-throated endorsement for everyone to come back and someone gets infected, you'd feel responsible for it. By offering a pro-and-con assessment, you got yourself off the hook.
Nothing is wrong with that because making the decision was not your job.
The manager wanted that burden shared by more people. It's human nature not want to be blamed. But leadership carries consequences.
Sure, you could have chosen to lighten his burden.