If it's your technology, you have nothing to worry about. You are an inventor, period. The patent agent (no, he is not an attorney) would be crazy to remove you if he knew perfectly that you are the orginial inventor. And the patent could be easily challenged/invalidated based on faulty inventorship. But, it all comes down to the record. So, keep a good record. Tell the patent attorney (not the agent) that you are an inventor. At the same time, you can always stick it to their face -- find a third party who are interested in the technology, and tell them, if they don't back off, you will assign your rights to the third party (every inventor has the equal right no matter how big/small the contribution is). At least, threaten to do so.
Hope this helps.