BMW's most basic diesel produces 150bhp out of 2.0 litres, easily pushed up over 200bhp if tuning is your thing. Refinement wise it is leagues ahead of the "ground breaking" PD diesel engines (even their ole 2.5 litre turbodiesel units that got phased out in 97 were smoother than the PD's.). The fascination with BMW? Apart from them being a country mile ahead of anything else coming out of Germany (911's possibly excepted depending on whether you like them or loathe them I suppose), they make pretty much the best powerplants on sale today. I merely compared the PD engine to BMW's efforts as you seemed to be of the opinion they were the peak of diesel innovation, which clearly, they aren't.
I didn't only single out the PD for its clattery note. If you'll notice, I referenced it dodgy handling (because at the end of the day, the PD is still a Mk4, and inherits the wallowy vague handling of its siblings - GTi models included) and limited rev range (it's still a diesel). The 150 may leave the 8v for dust (though hardly by as much as you are suggesting - not until third gear do they start pulling away, however their lead normally lasts til the next corner), but thats what I have my 16v brethren for. In the mean time, very few 130s or 115's get away, and I am rarely stuck in the black fog behind a silver TDi (do they make any other colours?) as you suggest (I'm normally past them and off).
Why do diesel Golfs outsell petrol golfs? Have you seen the current petrol engine lineup? An anemic 1.4 and 1.6 and then nothing til the GTi models. And we all know how horrendous the Mk4 petrol GTi's are. That and the increasing number of fleets going diesel only purely on cost concerns add many to the diesel sales tally.
Anyway, I'm going to go out and rev my engine to 5000rpm just for the hell of it. Because I can, you see, and also so I can marvel at the 2000rpm I STILL HAVE LEFT. Man, life must suck when it finishes at 4500 :/