Jesus died so that it would be possible for all people to be saved (sufficient for all, efficient for the elect)
Those who refuse Jesus' atonement are blamed because it was offered to them, but they refused. If it was never offered to the unbelievers, then the unbelievers should not be punished for refusing it. I know we have many other sins to be punished for, but Hebrews 10:29 specifically mentions punishment for rejecting Christ's blood.
Hebrews 10:29 NKJV Of how much worse punishment, do you suppose, will he be thought worthy who has trampled the Son of God underfoot, counted the blood of the covenant by which he was sanctified a common thing, and insulted the Spirit of grace?