according to you, mowing lawn is exclusive use

at best, you can prove that you own the "easement" on that land, which means, he can't make you to tear down the driveway.

Your dream of somehow having a driveway entitles you to build a garage on that spot is pure fantasy.

This is how legal dispute starts, greedy clients, not greedy lawyers.