Some updates:
Yesterday I sent an email to the agent which said: "We want to buy the house as long as the process is handled properly and honestly. We believe we have this right according to the regulations of the Board of the Registration of Real Estate Brokers and Salespersons". This morning we received a bank check from him which covers the home inspection and appraisal fee. Apparently he used overnight service when he got the email.
Looks like the agent did have some wrong doings in the process and he is afraid of a complaint to the board. This refund should be from his own pocket (I don't believe the home inspector and appraiser would refund him).
The lesson to us (and some other people) is that you cannot trust a real estate agent even though he/she may seem to be very honest, very diligent.