We have both, but mostly drive from the lower helm. Up top everything you want is down below, be it food, drink, sunglasses, the local chart, etc. But maybe even more importantly, and aside from the boat handling, docking, access issues others have raised, especially if like me ones other half is not as keen, and loves to read when out, it can get pretty lonely being the only one up there because you're driving.
Not a lot of folk who aren't mad keen like to just sit up there keeping the skipper company. If you're both downstairs, you can at least exchange comments, draw attention to a passing things of interest, and discuss proposed routes, anchorages, and when and what you'd like for a drink, meal, whatever. I would never buy a boat without a lower inside helm. I guess a pilothouse set-up is the best compromise all round.