I'm not going mad then!
I have to do the same pretty much every time I get in to the car. I thought I was imagining it.
It's not as if the seat fully reclines, mine would maybe move 1-2 inches each time then I give the knob a turn and it's back to where I want it.
I am 6ft and weight just under 80kg so I'm not massive.