If not the ISV, look at the idle switch. Had a similiar problem on a Passat years ago, the contacts tend to wear away.
Yeah worth eliminating.
Get a multimeter out, no need to switch the car on.
Connect to each pin.
Without touching the throttle at all you should have a closed ciruit (or 0 resitance)
Open it till you hear the first 'click' (at about 5% open), you should have infinite resitance (or an open circuit if you will)
Keep opening it all the way til you hear the next 'click' (at probably 95% open), and you shoul have a closed circuit again.