I'd be interested in knowing why you believe it was the Temple. The text doesn't preclude it, but the last reference to their whereabouts was the house where they were sitting...Acts 2:2.
The apostles "abode" in the upper room (Acts 1:13). That's where they lived. Women would not be in a room where men lived, it was culturally taboo. They continued in prayer with the women (v14), but that would not have been in the upper room. Luke 24:53 says they were continually in the Temple. That's where I believe they were meeting together and praying. Houses in Jerusalem were generally quite small, likely not big enough to hold 120 people. Also, "the house" (Acts 2:2) was a common name for the Temple.
Concerning why I believe only the twelve received the initial outpouring, to find the the antecedant for the pronouns in the first few verses of Acts 2, you have to look back to chapter 1.
Acts 1:26 And they gave forth their lots; and the lot fell upon Matthias; and
he was numbered with the eleven apostles.
Acts 2:1 And when the day of Pentecost was fully come,
they [Matthias and the eleven] were all with one accord in one place.
Acts 2:2 And suddenly there came a sound from heaven as of a rushing mighty wind, and it filled all the house where
they were sitting.
Acts 2:3 And there appeared unto
them cloven tongues like as of fire, and it sat upon each of
them.
Acts 2:4 And
they were all filled with the Holy Ghost, and began to speak with other tongues, as the Spirit gave
them utterance.
...
Acts 2:8 And how hear we every
man in our own tongue, wherein we were born?
The people speaking were men, further evidence that the initial outpouring was only on the twelve apostles.
..heading for bed. God bless.