They are built absolutely terribly.
I've had 3 now (different engines, expected different things) the trim was loose, they are ridiculously uncomfortable and badly designed inside, they are just terrible. Plus the fact that there are more E46's on the road than Mondeos means you have to really be in the company of a BMW snob to know the difference between your 46 M3 and the school run Mum's 320d m sport.
They're quite fun if you turn the traction control ALL the way off, but they're so heavy, when you get the grip back in the rear wheels, it's really severe, plus you're so uncomfortable it's hard to enjoy driving it unless you're wrecking the rear tyres.
M5 is a much more refined drive, not to mention comfy as hell and imposing to look at.