At first glance, I would agree with you, but the counter-argument is this:
What about married couples who cannot engage in sexual intercourse? Like if the husband is paralyzed from the waist down or something? Are they not considered to be married in God's sight?
I do believe that the whole "one flesh" thing is tied to sexual intercourse, and this would seem to be a reason why God would consider sexual immorality as a violation and seeming breaking of the wedding vow or marriage covenant itself. I'm not sure, but that's sort of how it seems to be to me. I definitely don't have all of the answers. This whole topic has been a giant head-scratcher for me for years.
Well, I think I if me make an oath then God actually expects us to keep them and doesn’t forget about them. The thing is, is people aren’t too good at keeping our promises, so Jesus pointed out that just saying yes or no is better. That might be why Paul said it’s better to not marry, too. Jesus also spoke to a woman at a well and informed her he knew about her many marriages. So I agree marriage is more then just sex.
A lot of this isn’t just plain in the scripture though, in my opinion, hence the confusion about it all. It’s a bit of a head-scratcher to me too.