Some people are born with "color vision deficiency", mostly males because the genes need are carried on the X chromosome - which men only have one of. So if the part that carries the genes that produce photopigments is damage - their is no backup chromosome. Also, color blindness can be cause by an injury to the eye, optic nerve, or brain. This could happen later in life to lead to the same result as someone born with color blindness.
I think you have to be born that way