Be willing to be responsible for what you say or do and not say or not do.
If you are unwilling, there's a price to pay.
A benefit of knowing your responsibility is that, at the same time, you would know the responsibility of others. Thus to establish the boundary, which seems to be the issue in your scenario.
The uneasiness provoked by a conflict has a lot to do with such unwillingness. Neuroticism may be another factor. Google "ocean test" if interested.
Godspeed.