H1B working location ...

Well, I have to say the manager is correct legally.

When applying an H1B, "work location" is a must fill in entry. And USCIS randomly select h1B to check if the person is at the "work location". 

USCIS sends a person to the company and talk with the H1B and his/her coworker to figure out if there is any fraud.

If your "shidi" was not in the office when USCIS staff comes to your office, it will be a serious problem for him and the sponor company.

The chance is not low to be selected. I was.