Depend on state law if you do not say anything and just let them to put their fence on your land, after a period of time, that part of the land will become your neighbor's property by law.
But different states have different laws, you should find a lawyer as soon as possible.