I don't think his idea will work. I am a mom and I know what you are talking about. Unless he can capture you physically abuse your child, e.g, beat/shake them really hard, starve them long term and etc., no judge will think a two-kids mom could be perfect 24x7. (Not to say leaving kids cry when they throw tantrum or something like that is not abuse at all. Even spanking is legal.)
I hope you have a few mom friends and get together regularly. They would be the perfect "character witness" for you. (Also school teachers, neighbors, co-workers.)
Again, I think you and your hu*****and should go to some consoling. If your hu*****and loves the kids, he should at least know what is best for them -- take them away from mom is close to abuse.
Another issue, I, and some of my friends, sent our kids to daycare when they were really young (
Good luck.