Agreed; Scripture tells us:
"Therefore there is now no condemnation for those who are in Christ Jesus. For the law of the Spirit of life in Christ Jesus has set you free from the law of sin and of death. For what the Law could not do, weak as it was through the flesh, God did: sending His own Son in the likeness of sinful flesh and as an offering for sin, He condemned sin in the flesh, so that the requirement of the Law might be fulfilled in us, who do not walk according to the flesh but according to the Spirit. For those who are according to the flesh set their minds on the things of the flesh, but those who are according to the Spirit, the things of the Spirit. For the mind set on the flesh is death, but the mind set on the Spirit is life and peace, because the mind set on the flesh is hostile toward God; for it does not subject itself to the law of God, for it is not even able to do so, and those who are in the flesh cannot please God." (Romans 8:1-8, NASB, emphasis mine)
I think people read too deeply into the term "servant", not realizing what the entire parable entails is that in the it, all the servants are subject to their master and cannot escape it. In likewise stead, all mankind is subject to the Lord Jesus Christ and cannot escape His judgment. And with that in mind, the Lord is letting folks know that punishment is not going to be "one size fits all"; after all, righteousness and justice are the foundation of God's throne (Ps. 89:14), and to punish everyone with the exact same level when their offenses and knowledge vary would not be just.
God is not some vindictive tyrant waiting to beat a human being to a bloody pulp and then raise His foot to smash them into the tiled floor of His Throne room; that's not the God of the Bible. Otherwise, Christ's blood didn't save anyone from sin. Is the Lord Jesus serious at the Bema Seat when He judges us? Yes, He is, and we will receive or lose rewards as He deems.
But He's not going to pummel us into ground beef; that's Satan whispering in our ears, trying to make us all doubt the Lord's mercy, grace and love.
-Sojo414