I don't understand why you took offense when he gave you his notes.
He probably just want to make sure that "in case" you miss anything in your notes when you work on the summary, his notes might be helpful.
Personally, I think it is a bit unprofessional that you comment his handwriting in work related email.