You have to understand unless you do loans with Fannie or Freddie directly, otherwise whoever you get loan thru are considered as "broker" however they call themselves(whether wholesale broker or retail banks).
Retail agents work for commission only and they are "middle man" as well. The bank who you pay mortgage to does not own your loan, most of the time, they are "servicer".
If you are so concern and the other agent guarantee you that they won't go after you for the fees for refi last than 6months, why don't you ask him/her provide a signed letter with company letterhead? In case, previous broker gets fined and go after you, you can call the retail agent...
