Your green card will be fine, and also he has no right to remove your green card at all. Going to court or not is up to you. Even if you two divorced, your green card is still Ok. Ex-Immigration Officer.