There will no doubt be dust, road grime and other contaminants lying on the paint (otherwise you wouldn't be wiping it over) and if your just wiping it over with no lubrication then you'll be inflicting tiny scratches, this is what causes swirl marks, ideally if your going to do that then you could do with some form of quick detailer.