Then I think you should talk to your PD

Telling the PD that you worry about your baby. If possible, let the PD write a letter or something like that to the apt. manager to allow you to switch. If PD does not want to write the letter, talk to your manager again and tell him that you worry the negative effect the whole thing to your baby. The risk is too high to wait for going to court. If your manager still will not back off, talk to State Child Service department and see if they have any ideas.

Remember, the most important thing for you is to protect your baby; not to get even with your neighbor (or apartment or whatever).