Mcmaddy, I've learned from experience that is advisable to put a brand new towel through the washing machine first before using it, as it gets rid of much of the fluff and lint, that otherwise ends up on your nice clean car as you've just found out
I only know this because I used to get white/yellow/purple, (whatever colour the towel was) fluff and bits all over my car if I used the towel straight out of the packet. In my case it was yellow bits all over a black car.
Give it a good wash and I'm sure it will get rid of any loose fabric for next time.