If you were just polishing it then you would be fine, however SRP is very filler heavy and leaves some product on the car to hide/mask the swirl marks, and for this reason then yes you would probably remove most of this by washing without any protection over the top.
After claying, see how much time you have got left, if you have time to polish and seal then carry on, if not then just wait till the next day.