Body for the body, soul for the soul.
I know that I have a number of biases regarding body types. But I recognize my associations: for instance, I dislike tons of fat on people because it is indicitive of laziness and an unwillingness to be physically active. I dislike strong, bad odours for obvious reasons. Things like that.
The other part of it has to do with social theories and personal space theories. We judge one another because we are careful about who we want to open up to. We are careful about how much we open up to others, and for those people that we do become friendly with, we still divide them into categories based upon how comfortable we are with them: associates, barely friends, friends, close friends, best friends, lover or sibling type friends. On a purely physical level, this can be easily justified by way of protecting yourself from STDs or violent people. Personality wise, this is justifiable to protect yourself from people who are incapable of keeping your secrets safe, or from people who want to emotionally control you, or who are just plain psychopaths that like to hurt people.
is it not such a selfish and narcissistic concept where you connect only with people on the similar level of consciousness?
Different people are in different situations in life. We are not all at equal points. So I would suggest that you use your feelings of love to get to know individuals better, but without necessarily opening yourself up to the possibility of being hurt, while trying to help them in some way.
Also, just because you like someone doesn't mean you have to have sex with them
Just like you can have sex with someone you don't like. Sex and intimacy are nice when they go together, but each can be done apart from one another too.
EDIT: And if you feel feelings for people that you don't want to get really intimate with, I would suggest trying to guide those feelings towards compassion, and wanting to help those people in their lives. This will still allow you to interact with them in line with your desires that are being created by your feelings, without needing to judge other people, and thus you won't need to feel the guilt either.