Seems an easy issue. Be friends with anyone who can be a friend back.
If anyone brings you down in any way, move on without them. This goes for Christian and non-Christian friends alike.
Not all Christians, or claiming to be, are good people to be around. And not all unsaved are out to bring Christians, or people in general, down.
Make the decision on a person by person basis, not on beliefs.