The 3 dr Golf has pretty good for access to the back seats. The only way to check is to try it.
However, it's not just getting in and out that is the difference. The 5 dr obviously has its own opening windows and feels much more light and airy in the back. So better for passengers.(They can honk out of the window if your driving gets too much

)
In terms of performance, the 5dr is slightly heavier, but I can't imagine its noticeable.ÂÂÂ
You will have just as much fun in a 5 as a 3 
If the kids are getting in and out on a daily basis, I would go for 5. If it's only at weekends then you might get away with 3, but I'm sure the wife will prefer 5.
Definitely a case for try before you buy, and you don't even need a GTi model to test that.
My gut feeling is go for 5. Best of both worlds-practicality & performance. (as Mr Clarkson commented last year)