Sounds like you've wired the remote wire (thin blue one) to perm live on back of stereo thus making the amp never switch off!
Do you find you have to switch the stereo on/off by hand as even when taking key out, it doesn't switch off? If so, swap red and yellow over as you have them wired wrong!
You shouldn't have to take a fuse out and your battery shouldn't go flat!