Don't let him take the baby. She needs her mother more than her father;although, I agree she needs both parent. Call the police if he is going to force you to let go of your daughter. It's irrelevant whose fault at this time.
I am a mother of two and I am getting a divorce. I would give up my life for my children.I would never let my EX raise my children with the other woman. No other woman will love my children more than I do.
It's best if you can work things out but it's not the end of the world if you do get a divorce. The most important thing is to be financially independent if you seek a divorce.
Best of luck!!!