What do the three colors symbolize in "Desiree's Baby"?

Since you don't specify the colors you are referring to, I'm going to assume you mean skin colors. Skin colors are referred to as white, black, and yellow. White stands for the ruling class of white people in the Southern plantation society. Black stands for the class of slaves who worked on the plantations, who are powerless in their society. Yellow stands for those who have both black and white ancestry or parentage; having mixed...

Since you don't specify the colors you are referring to, I'm going to assume you mean skin colors. Skin colors are referred to as white, black, and yellow. White stands for the ruling class of white people in the Southern plantation society. Black stands for the class of slaves who worked on the plantations, who are powerless in their society. Yellow stands for those who have both black and white ancestry or parentage; having mixed blood, they were considered tainted whether they were slaves or free. Thus when Armand notices his child has African features, he interprets that to mean that Desiree is "yellow," even though her hands are whiter than his, and he rejects her and the child. In the ironic twist ending, we learn that it is Armand himself who is "yellow" because his mother was of mixed blood. But the reader is left to consider that what taints him is not his genetics, but his cruelty.

There is another passage in the story that brings out three colors. As Desiree leaves Valmonde and walks into the swamp, she is described like this: 

Desiree had not changed the thin white garment nor the slippers which she wore. Her hair was uncovered and the sun’s rays brought a golden gleam from its brown meshes.

This passage foreshadows the ending of the story by emphasizing again how "white" Desiree looked, even having hair that had golden undertones. This should lead the reader to question whether Armand's assumption about her ancestry is correct. The other place where three colors are mentioned is this passage that Desiree speaks to Armand:

"I am white! Look at my hair, it is brown; and my eyes are gray, Armand, you know they are gray."

Again, this emphasizes Desiree's "white" appearance and serves to set up the ending where we find out that it is Armand's side of the family that has African heritage, not Desiree's.