The red and yellow wires that you swap are part of the stereo loom, not the car loom.
The plugs shown in your photo's are the termination of the car loom.
They then plug into the ISO socket which is the termination (car end) of the stereo loom.
Do you have a loom for the stereo?
One end will be small (that plugs into head unit)
The other end will be large - that accepts those plugs as shown in your photos.
This loom will have red and yellow wires.
These are the wires to swap.
For the sake of the next owner of your car, please dont bugger about with the car wiring - there's nothing wrong with it.
The red and yellow wires of your stereo loom will be fitted with bullet connectors mid-way for this specific purpose.