I personally don't think the build quality is too bad, particularly when you compare it with its' comparables from say Ford, Vauxhall, Honda, Renault, etc.
What I did notice, was after spending a solid month on my RS4 (when I did my "european tour" back in May), and then getting home and jumping in the Golf, was that the Audi was a much "nicer" place to be. Some of the switchgear in the Golf, particularly the indicators/wipers and electric windows just didn't feel "quality" in the same way that Audi make them.
However, what was/is perfectly clear, is the "major mechanicals", ie, the structural rigidity of the chassis, the quality of the suspension (ride and handling), the engine (refinement and advanced technology), and the DSG 'box - these are far, far better than any of the competition above.
I think everyone agrees that some of the interior plastics on the Mk5 are no better, indeed, maybe even a step backwards, compared to the Mk4s. However, the chassis engineering on the Mk5 is a quantum leap better than the Mk4, as is much of the general technology.