I use my boat in a similar fashion, fortunately I can be fishing within 20 minutes of leaving my slip in spite of it's slow speed because of the location of my marina. Occasionally, I might travel a little further, maybe about an hour before starting to put lines in, but it is usually 30 minutes or less. I am fine with not always being on the hottest bite and the tradeoff in lowered operating costs with a slow, single engine boat works for me as well as my guests. If this is not the situation for you, and you need to travel long distances, I expect you find that your friends rarely have enough time to spend fishing with you.
I was originally planning to buy a sportfish and you really can get a lot of boat for the money but be very mindful of the state of the engines when shopping. This is true for all boats of course but most sportfish boats are run pretty hard and you will find in many listings details about how recently they were rebuilt, lookup the costs of these rebuilds and be honest about how you plan to use the boat and the likely hood of having to repower or rebuild the powerplants. There is no universal right decision to be made, you just need to find the best fit for you.
I have learned to appreciate my boat, how little I burn in fuel, how relatively simple it is to maintain and when my friends join me for a morning or afternoon of fishing, we are more concerned with visiting than how great the fishing is. If your intended guests are more serious fishermen and your performance compared to everyone else on the dock is important to you, a sportfish would be a better fit.