A couple of things you may want to consider:

1. let you boss realize your real value is in handling finacials for him - not only can you help him do basic book keepings and payroll, but help him set up standard financial management processes to better compete in the market - cost reduction/control, expense planning, tax planning and even pricing benchmark/analysis (yes, if possible, you can say to your boss that you can help your coworker to do pricing or invoicing, and let her focus on picking up the phone:). By doing so, you're helping out and don't have to do what you don't want to. Hopefully by doing so this will also further your career goals.
2). If you still have to answer phones, talk to your boss that you'll pick up the phone first (so that your customers or whoever will reach a human being) and then you will transfer it to voice message except for urgent issues. This way you don't have to take note on every call and will save you some time.
3). If worst comes to worst, you cannot change anything and you don't want to quit, just try to take a different perspective. For example you can take this as an oppotunity for you to learn business, see what day to day operation is like, improve your customer facing skills, and improve oral English etc:).

No one's job is perfect. In this economy it's wise to stay low and hang in there. Try to pick on some skills/experience in the current job, and always on the look out for better oppotunities.