A little experimenting has lead me to the cause.
Checked & double checked the hoses leading to the servo (all ok). Then decided to do the good old test on it: engine of & pump the break till the vacuum is exhausted, then holding the pedal down turn the ignition on, pedal should give as the vacuum builds up. Which it did until the revs dropped to 500rpm.
So the good news is that it is the engine speed that's the cause. The bad news is I've being fighting this rough running for a while...