Actually these Scriptures (below) confirm that the Church (consisting of the redeemed of the Lord) is the spiritual "building" or "house" or "temple" of God where each member is a "living stone": What? know ye not that your body is the temple of the Holy Ghost which is in you, which ye have of God, and ye are not your own? For ye are bought with a price: therefore glorify God in your body, and in your spirit, which are God's... Ye also, as lively [living] stones, are built up a spiritual house, an holy priesthood, to offer up spiritual sacrifices, acceptable to God by Jesus Christ. (1 Cor 6:19,20; 1 Pet 2:5)
These verses put a tremendous responsibility on every Christian, since God is expecting spiritual sacrifices (including self-sacrifice) from those who are a spiritual house.
Also the Bible uses three metaphors for the Church. It is (a) the Body, (b) the Bride, and (c) the Building of Christ. We do not see the spiritual dimension in this life, but it will all come together eventually.