If your GC was never a conditional one or condition was removed at the time of your divorce, you do not have a break. The 5 years start from your first I-485 approval date. If you were still conditional LPR when you divorce the sponsoring citizen spouse, your status stopped, and the 5 years should start at the date you REGAIN LPR status through the second citizen spouse.
The 3 years of citizen's spouse also reset when you become unmarried, and should start counting from 0 day again at your second marriage.