Remember that the contract you signed explicitly stipulates that the sellers be allowed a chance to fix the problems arising from the home inspection, "to the reasonable satisfaction of the buyers".
My sense is that your agent is right (notwithstanding he/she stands to lose the commission if the deal falls through). You will not be getting your deposit back without allowing the sellers to redress the mold and water damage issues.
Even if you shall lose $1,500 to $2,000 (perhaps?) without other contractual consequences, it may still be worthwhile to back out of this deal for the peace of mind down the road. Almost everywhere in US it is a buyers market.