I wonder if the metal clip has broken inside?
IIRC, to remove the gaitor, push the base of the gaitor (from the back) towards the dash, & then you should be able to get the base out. Then lift the gaitor up over the gear knob so it's 'inside-out'. There should be a metal securing clip at top of the gear stick.
Let me know if you need a piccy or 2?