When I came back US with my AP, INS officers took my AP, passport for such a long time that I almost missed my plane. I guess they will verify with your record. But if you have to go back, then you have to go. Just be prepared when they question you and bring all your relavant documents. The attorney is a good backup too. Good luck!