It should be the rear brake system, it is one of the problem for toyota car with rear drum brake system, there is no good way to fix it, but it will not cause any problem either. If you want to do some thing about it, you can take apart the rear brake drum every six month, open the drum and use the brake clearing fluit to clean out all the dust inside, that will make it a little better.