For lower ball joints, most people don't know they are bad, until one day people go to do alignment. Before doing alignment, they will ask owners to change ball joints first. For your case, if you don't feel bad, just keep driving, it won't cause major issues, loose is loose, so what?
If I were you, I'd keep it as-is, until couple years later you change to another car. If you can DIY, and you think it can last more than 5 years, sure do these, for parts only about $100 or less.