I can't say I've ever ran into this as a problem. Rather than worry about who, out of the 500 people that might be on, to avoid I concentrate on building a quality friends list. I farm with people I know, level with people I know, and if I'm playing runs with people I don't know I just wait a game or so before I decide to potion up.

Is getting booted really such a bad thing? Oh well, it's always easy to join a game, and if you feel so inclined, start one yourself.