Before purchasing a house the buyer would run a title search to find out not only the title was marketable, but also what are the limitations of the title is (such as HOA, easement, etc)
Party wall agreement is one of the limitations on the title. It could be recorded in the county clerk's office (or whereever your local land record office is), so that it "runs with the land", that is, the agreements binds the underlying land instead of the transient owners. This is because all future buyers would see this agreement when they do a title search. In such case, new owners don't need to sign another party wall agreement to have such agreement to bind them.
But if you (and your neighbor) didn't record the agreement with the clerk's office, new buyers would not know about the agreement and therefore the agreemet doesn't bind them.
I still don't understand why party wall agreement is so important, unless you are actualy doing certain home improvement project involving the wall. If you have real need to worry about the agreement, state your particular situation. Otherwise I'd ask you not to worry about it.
