at best, you can prove that you own the "easement" on that land, which means, he can't make you to tear down the driveway.
Your dream of somehow having a driveway entitles you to build a garage on that spot is pure fantasy.
This is how legal dispute starts, greedy clients, not greedy lawyers.