Well, I have to say the manager is correct legally.
When applying an H1B, "work location" is a must fill in entry. And USCIS randomly select h1B to check if the person is at the "work location".
USCIS sends a person to the company and talk with the H1B and his/her coworker to figure out if there is any fraud.
If your "shidi" was not in the office when USCIS staff comes to your office, it will be a serious problem for him and the sponor company.
The chance is not low to be selected. I was.