I would say that pigments are breaking down to the point of no return! the more it now gets polished, the shorter the period of time before the spots return, damage would have been done by raindrops magnifying UV deep into the pigments causing one of them to break down, Its possible Nick, that what you're seeing, maybe damage that was "polished out" before you bought the car, making an unwanted return! a quality sealer, and a good waxing, may put it to bed, but only for a short while... eventually, I would expect the panel to turn White
Thom