Yes, a poor drying technique can inflict damage. I like to minimise contact with the paintwork during the drying stage.
I’ve got a warm air dryer that I use to blow the final rinse water off the car. If the car is well protected, blowing off what rinse water is left leaves very few droplets to pat dry with a good quality absorbent drying towel. A warm air dryer is also very good at removing the water from all those annoying water traps too, some of which can continue to drip for some time after you’ve finished drying the car (door mirror housings, grilles, door handles, window rubbers, wheel bolt holes etc.).