don't worry too much..

the seller's agent's broker must responsible for the earnst money, if they refuse to refund your money, get ready to file suit, they will owe you your deposit, plus interest and attorney fee...

most brokers are not that stupid..

the question is do you still want to buy the house?