keep it under control.
All he heard was the tone, not the words. So, you need to remove the tone, which makes you sound weak.
By removing the tone and focusing on your words, calmly repeated, you're telling him: My words are sufficient.
(Do not give him the power to agitate you.)