The dry skin that sheds off of you at night will build up in the fibers of the fabric. It may not smell for a while but you'll probably start feeling itchy and grimy.
They get dirtier and may start to smell. Some people wash them once a week, other's once a year or less.
They stay dirty and stinky