Since your boss is not friend, give him an estimate

as soon as he gives you an assignment. You need to give a disclaimer it may take longer if there are too many side assignments.

Also, like sigma said, keep an note of other side work that he assigned (duration, content etc).

This can be used to back you your assignment