When I know that it is not true and I know myself that I won't do such things, I won't get affected. But when they are intentionally doing it to hurt me, I will tell them to stop. But I won't stoop down on their level by trying to explain things because I don't owe it to them, since they judge me even before they ask to know the truth. I would also stay from this negative people because they won't help me to be a better person.
I simply consider the source and let the person know that it does not bother me. That normally bothers them because they know that they aren't affecting me :)