And this is the point Paul is making in Romans 5. There was no law in place between Adam and Moses, yet people were dying. Since there was only 1 human sin accrued at that point, it must have been Adam's sin imputed to them that brought about their deaths.
Look again. It specifically says in Romans 5:12 that all have sinned (See below). Verse 8 also suggests this. In fact other scriptures also point blank say it. Realize that there is a distinction being made in the chapter between sin and trespass. This is most evident in verse 14.
You also need to be careful about not confusing the natural consequences of sin with judgement for trespass, physical death with spiritual death which is related, and origination of source of consequences with totality of source of consequences as it relates to Adam's sin.
To clarify the last of these, just because sin entered the world through Adam, doesn't mean he is the only one who "accrued" sinned. For one thing, you are forgetting, or maybe not understanding about the Tree of the Knowledge of Good and Evil. The tree was deception - mixing up truth and error. When Adam took the fruit the whole world changed, and everyone down stream has reaped the confusion that followed, which makes it essentially impossible not to sin.
Romans 5:12 NASB1995
"Therefore, just as through one man sin entered into the world, and death through sin, and so death spread to all men, because all sinned— "
Romans 5:14 NIV
"Nevertheless, death reigned from the time of Adam to the time of Moses, even over those who did not sin by breaking a command, as did Adam, who is a pattern of the one to come."
Romans 5:8 NIV
"But God demonstrates his own love for us in this: While we were still sinners, Christ died for us."
Romans 3:23 NIV
"for all have sinned and fall short of the glory of God..."
Also possibly pertinent:
Romans 6:20-23 NIV
"When you were slaves to sin, you were free from the control of righteousness. What benefit did you reap at that time from the things you are now ashamed of? Those things result in death! But now that you have been set free from sin and have become slaves of God, the benefit you reap leads to holiness, and the result is eternal life. For the wages of sin is death, but the gift of God is eternal life in Christ Jesus our Lord."