probably yes

A client can fire the lawyer at any time. Whether you can get back your retainer probably depends on whether the retainer is the "flat fee" for the service or is it the fee to reserve his service. Since the fee is pretty large, I guess it is the flat fee for his service, in which case you probably can get it back. You can tell him that he violates his "duty to communicate" with the client and threaten to tell the bar association (they can discipline the lawyers). Good luck.