Yeah, you're right 文青 ：－）

I overthought. I was only pictured senarios like "This drives me nuts!!!"  (said it angrily) situation 哈哈

It apparently also used in situations like "Hey, if we do this, it probably gonna to drive him nuts"  kind of senarios :)

But you're right, it can be used in "drive someone nuts" without limiting "you", "me", "him", etc..