I don't do much ladder throwing at the fire house but I certainly do more house work than you do...
Doing physical work such as fixing up houses does give you decent amount of exercises, but is not sufficient to prevent age related deterioration. You still need regular workout on cardio and muscle to be able to maintain at desirable levels.
I think the key is the intensity. 1 day of low intensity phsical work does not have the same effect of a 1/2 hour cardio, and certainly even less effective at maintaining muscle strength.
But at least doing physical work regularly gives you regular exercises on complicated brain-muscle coordinations for various non-repeating tasks, that today's typical office worker lacks. For me, the house fixing works keep me not too far behind so that when I started regimented workout a couple of month ago I was able to quickly regain my abilities on measurable fitness performances.
