A pastor can only be a pastor of a person or persons. Their calling is to shepherd those whom God has placed under their care. So, it is unbiblical to have pastors of nebulous groups or areas of ministry (eg. "Youth", "Children's", "Elderly", "Head", "Associate", etc.) Doing so, creates a culture wherein believers cannot discern the Body of Christ nor how each believer fits into Him. Early death and sickness result from such an arrangement.
2nd, having the grace of a pastor is a gift not something obtained by study. Certainly pastors should study but Biblically, no one is a pastor unless the Lord has made him one. You cannot graduate from seminary to force the Lord's hand. I have nothing against seminary, per se, but merely obtaining a degree does not make one a pastor in the Lord. A seminary degree might make one qualified to be a pastor in many churches but that is at the expense of the standard shown us in scripture.
Lastly, you cannot give to others what you don't have. If you do not have someone ruling over your soul then you cannot be a pastor. Who do you look to for correction, for guidance, for teaching, and for encouragement. If you say "Well, Jesus can do all that for me" not only do you ignore the connections show to us in scripture (including Jesus and His 12) but you take on a double-standard: one for you and another for the others. You mustn't do this. If you do, you will get the results you already wrote about;
This is not to say that none will leave you if you follow God's pattern (Jesus lost 1 out of the 12) but you cannot think to change the trajectory of the current church culture by embedding yourself in it.
Such things, because they run so opposed to the way "church is done", require wisdom and humility to understand.
Blessings