It can be a problem even for me, in some occasions. An funny experience: years back, I was having a conversation with someone in Mandarin. I thought he said 四書(五經), yet he was actually referring to his 四叔. The context happened to make sense in both cases...