I kind of agree what your manager said

In my group people grabs their tasks from the backlog. The process is:

1. There is a pool of tasks that this sprint should finish
2. A team member grabs a task from the pool
3. After the task is done, check in code, send an email to team, as for code review
4. While waiting for code review response(s), grab another task

It could be true that your manager doesn't like you, but I would suggest you do your best to change your image. This will also help in your next position if you do need to find a new job