Your brain guesses the illumination color and subtracts it, so objects appear to keep their "true" color. Different assumptions about lighting = different perceived colors.
This is why people disagreed about The Dress (2015): different brains assumed different illumination.