The context is also in Hebrews 3, where people perished due to their unbelief and disobedience 40 years in the wilderness. 40 years in the wilderness, they were offered a literal 7th day rest, not an eternal rest. Therefore the Sabbath-rest is in reference to a literal 7th day Sabbath.
So do you believe in the same Sabbath rest that those who, because of unbelief and disobedience, perished in the wilderness for? The same Sabbath they perished for was a literal day; they weren't offered a Sabbath rest in Christ. So "Jesus is my Sabbath" is an invalid point for Hebrews 3-4.