I overthought. I was only pictured senarios like "This drives me nuts!!!" (said it angrily) situation 哈哈
It apparently also used in situations like "Hey, if we do this, it probably gonna to drive him nuts" kind of senarios :)
But you're right, it can be used in "drive someone nuts" without limiting "you", "me", "him", etc..
