He made a statment of fact
people who were of us. but left. and now deny Christ, were never of us..
They left to reveal the fact they were never of us, for if they were they never would have left.
The people he is talking about left Yahshua not a church building, same thing I said at the beginning. Someone can leave a church and go to one they believe teaches more truth or be led by the spirit without a church building.
I had family member that used to go to a church with a famous leader, and they would always tell me this person left that person left, and they couldnt figure out why. and the church leader would always say "they fell away" and insinuate they were "faithless". a few years later my family members left because they found out the church leader ws crooked, like very very bad. Im sure he said the same about them when they left.
Not saying this is your experience, I have no clue of your experience, but my point remains, people can leave a church and not fall away. Moving, finding one they deem better, learning more truth, etc.
John was in the truth more than any "church" today, so people leaving their ekklesia was much different than leaving a church today.
I tried to fully explain, hope I worded it well.