Hello Troubled65!
Your concern regarding those outside the city, is simply a matter of process of elimination. Regarding this, scripture states the following for those who are judged at the great white throne judgment, which is made up of all of the unrighteous dead throughout all of history:
"And if anyone was found whose name was not written in the book of life, he was thrown into the lake of fire."
From the scripture above, we can deduce that anyone whose name is not written in the book of life, that person will be thrown into the lake of fire. While everyone who took part in the first resurrection will enter into the kingdom of God. And since they will be in the lake of fire, then they cannot be meandering around outside of the new Jerusalem.
Therefore, the reference to those "outside the city" would have to refer to those in outer darkness, i.e. the lake of fire. At first appearance it would make the reader believe that there will people loitering outside the city gates, which would not be in keeping with God's nature, nor His word. When we read "outside the city" our minds interpret it as being in close proximity to the city gates. However, scripture gives no measurement of distance or location by the words "outside the city." How far is outside? It doesn't say.
In addition, they will not be able to enter in, because they will be in the lake of fire, which they can't escape from. Once again, our minds interpret "nothing unclean will ever enter it" as them having opportunity to do so, but being kept out. However, once again the reason that they will not be able to enter into it, is because they will be in the lake of fire and will have no opportunity to do so.
Hope this was helpful.