It doesn't sound like you

It doesn't sound like you should go back, especially 兩年來初有起色我也很喜歡。
"中止我喜歡的去做一個我不喜歡的" is a big no no.  Especially there is no garuantee of a future with 我不喜歡的

but don't understand why 想到孩子還有幾年上大學，我還要回去工作很多不情願,  Can't you contunue your own business even after your child go to collegue?  Is going back to work is something that will eventually happen?  Then it may be a different story.