This great cartoon infographic by Korwin Briggs of Veritable Hokum is an entertaining overview of the history of human color production, from mummy brown to synthetic indigo.
As it turns out, for most of the last 2,000 years, most of our pigments have come from grinding up toxic metals. Orpiment and Realgar, two yellow and orange colors used by the Egyptians? They're full of arsenic. Greek vermillion? It'll give you mercury poison. Napes yellow? It's full of lead. Emerald green? So poisonous, it was sold as rat poison.
And if your early pigments didn't contain poison, chances are they were full of creepy-crawlies. Tyrian purple is made from ground up snail shells. (And available for purchase, if you can afford it Carmine is produced by boiling thousands of beetles at once. Indian yellow? It's cow urine. And mummy brown, not so surprisingly, takes its name from the ground-up corpses of pharaohs and their courts.