However, why they charged twice? You can argue about at lease one of the two charges.
On the other hand, if you have loan agent's "written" statement saying that he/she would pay for it if the loan was not approved, then the loan officer should pay.
Your agent's words mean nothing in the picture.
If there is no written commitment, the only case that the loan office would pay is that he still wants to hook you for the (next) business.