"順手就拔掉他一邊的耳機" --- You use force on him first!

I hate to tell you this. Regardless how you try to make
light of your action, you use force on him first.
"用力往我頭上一拍", I image if he really use that much
force, you would not be able able to talk for a while.