When you bought the 2nd house, on the contract, did you say "investment property" or "primary home"? If there was a loan, did you get as primamry home or rental property loan?
In tax terms, at any given point of time, you can only declare one as "primary".
So in strict terms, it is hard to pretend one over another. Everything is timestampped and logged.