Of course the landlord can

You guys don't hold anything to occupy the property, so in theory the landlord can get you guys out at any time.

However, this is not really about law or governance, it's about how to handle tough situations with caution and strategy and good tactics -- and perhaps how to prevent this kind of thing from happening at all.

It is obvious to me: it's simply not wise to get this thing further complicated, and you get out is a better solution -- once you left, the landlord can just tell the man leave, and if he refuses, the landlord can just get police pulling him over.

Why your leaving is better solution? You simply want to make sure that man has no idea where you go and live, just for security purposes -- you're just in a vulnerable position, and it is in your interest to fix it ASAP. How about the landlord? I'm sure you guys don't know his residence address -- he may have an address for mails so that he can receive checks from you, but a careful NYC landlord wouldn't make his residence tractable by any potentially dangerous occupiers of his property.