I set up my own company last august with a friend. Similar situation to you. We're both in full time employment and are doing this in our spare time.
Company name: Try someone like companies made simple. Small fee, but you can check availability of names and they sort it out for you.
Bank account: All much of a muchness tbh. Most are offering 2 years free banking.
Accounting: We use a free online software called Quick File. Simple to use. Does invoices, can upload bank statements etc. We plan to get an accountant to tidy things up for us when required.
Taxes: As far as I was aware, if you are PAYE with your current employer the only tax required is corporation tax. VAT is not mandatory until your turnover is £70k
Good luck with the business. If you don't try you'll never know.