Let them know that you do not expect more money but feel very confused with the rating. Provide some good examples that you had no time to respond to that project manager but preoccupied with other more important projects or you were just in a meeting when email inquiry came in, blablabla.
You would say something like: I would work with him more closely and spend some time together to improve communition between us, but he has never brought up the issue to me so i was very surprised that I did not meet your expectation. If he had addressed the concern to me earlier, probably we would have already worked out an efficient way to handle email traffic, etc.
Also make sure that a "not meet" is not acceptable unless they show you their exact expectation of you was "responding emails within 24 hours" or so.
