Socializing with the "sinners"

Many christians today have a problem socializing with other people that drink, or smoke, or have sex, etc. Sometimes a pastor or fellow believer might hold them accountable by saying something like "during Jesus' ministry on earth he hung out with sinners. He spent time with tax collectors and prostitutes and drunks, so its ok to spend time with them as well". If you think about it though we are all sinners. We all struggle with some sort of sin, whether it be porn, or lying, or stealing, or whatever it is that has a hold on you. So isn't this kind just like saying "Jesus hung out"? So why not just hang out with people. Get to know them for who they are, not for what they do. As much as we'd all like to be, we're not perfect and we never will be so there is no point in holding yourself above anyone else. All that usually does is push unbelievers farther away.