You are not wrong.
Setting aside the whole issue of titles, the leaders/elders of a church do have authority over what happens within the church, but it is limited by Scripture. Where a pastor attempts to exert control beyond those limits, that pastor has no authority, and it is appropriate to...