I always make sure I get the car as clean as I can before actually touching it by mit too. If it's really dirty (most washes..), it gets a pressure washer rinse, then an a pre-rinse (super snow foam atm), then another rinse all before I've touched it. Then per the above, start from the top, use different mits for the bits below bump strips, and another (attached to a pokey stick!), for the wheels. And then towels to dry rather than a chamois. If you've got a good layer of wax on before the car even gets dirty it'll help too.
