It won't move because you're not using the right tools.
Get it in a vice, as suggested, to hold it still
You need a punch, not a drill bit.
A drill bit has a pointed tip, which will splay open the end you're trying to punch out. A punch has a flat end that will knock out this pin.
If it won't shift with a correctly sized punch then you need a bigger hammer.
