$4000 for a lawyer letter is too much, but I don't know how you can negotiated that down without another local lawyer.
You should have paid the late fee too. That way you are sqaure with the HOA and you will not have dispute with HOA and HOA will not be in the middle of all the future fights.
All that is left is your dispute with the HOA's attorney. I suggest you hire a lawyer to negotiate.