“An affair outside of a marriage” is an accrued asset on your emotional balance sheet and you have to expense it in a later time.
If you accumulated a lot with your spouse in the past and your principal would be high, therefore you would have higher TMV when you are older.
If the opportunity cost is just a diamond ring, I would give it up.