if you have movement then you have a air leak!! and yes this will alter your idle,there is a o-ring seal on your idle screw, thats either badly worn or the screw is out to far ,"un screwed from the throttle body" you really need to set the idle up correctly,
get the engine to normal temp ! un clip the spade connector under coil, take off the electric connector to your isv,pull of the crank hose where it attaches to your air box and plug hole, then your revs should plummit ,
adjust idle on idle screw to around 1000 rpm +-50,if in doubt put some grease in around idle screw ,connect all back up and bob should be your uncle.if that still does not work you have air leak else wear ,"happy hunting"
regards kevin