Please just think if you were your previous manager,

how would you feel when his staff jumped to other team and what would you do when this person tell him want back? Think about his personality and management style would help you make decision. Do you think he would trust you as before?