Man, just bite the bullet and get it over with

To me, your suspicion is reasonable. But to prevail in the court, you need much more than than. If I were you, I would just bite the bullet and get it over with. After all, you are not without fault to begin with. If I understand it correctly, the agent is not technically your agent. Rather he/she is seller's agent. How could you trust someone whose only interest was to sell the house to collect the commission? You should have found a termite company yourself to get an impartial report.