There are many sources that would disagree. But since you wish to make an argument from Gen 6: 1-3, did you think that up until v. 1, women only produced sons and when they produced daughters, God became angry when 'nobles' desired them? That is about the weakest and silliest idea yet. How did these normal human men produce giants then?
v.4 does not make any sense when viewed in the way you suggest. But I suppose it does satisfy certain mind sets.
7And the Lord said, "I will blot out man, whom I created, from upon the face of the earth, from man to cattle to creeping thing, to the fowl of the heavens, for I regret that I made them."
So God destroyed the earth because men had sex with women. That, is what you are saying. Guess we are all in for it now.
The Nephilim (נְפִילִים; a word only ever found in plural form in OT) only appears three times in Gen 6:4 and Num 13:33 (twice). The word in 1 Ch 20:8 (also 1 Ch 20:6 and 1 Ch 8:2; cf. also 1 Ch 4:12) is Rapha (רָפָא; singular), also only found these three times (though that depends on interpretation of homonyms, see below). In Josh 15:8 is the word Rephaim (רְפָאִים; plural), which is always found its eight times (also Josh 18:16; 2 Sam 5:18, 22; 23:13; Isa 17:5; 1 Ch 11:15; 14:9) in connection with the word for valley, Emeq (עֵמֶק), hence in the translations either translated "valley of giants" or transliterated "valley of Rephaim."
Second, many major lexicons essentially uphold the word "giant(s)" as the translation in these spots, though you will find some differences as to what "giant" even should "mean." Many choose not to
translate the words because of issues in knowing what translation is best, and so they
transliterate the Hebrew sounds into English (hence, Nephilim and Rephaim).