After the permit is issued, which is your case, the appeal time is 15 days. This appeal goes to appeal board.
I guess you will go to a hearing. My advice will be to state the facts and not the feelings. Facts include encroachment or not complying with building code. It's hard to get the permit revoked though since now you are against both the city inspection board and the neighbor.
Your best chance is work out some compromise, such as putting a fence screen or shifting the fence from your side a couple feet. Normal sunset lot is 25 feet wide so that's possible.