Ok, as to single handing and "feeling safe." Feeling safe is a lot different from "wanting to do it regularly." Also, I see most of the responses addressing docking and just helm issues.
First, I do agree with the consensus that when it comes to docking and actually being a helmsman, length is largely irrelevant. In enough wind, a 12' inflatable can be difficult to put perfectly in a spot. On the other hand, the largest boat I've handled is 130'. Give me a 130' boat, twin engines, excellent bow and stern thrusters, a joy stick for docking and decks and controls on both sides and the stern and I can put that boat sideways into a spot that you wouldn't think of putting a single engine 44' boat with no thrusters.
I do believe though boat selection and equipment become very important in single handing or in the second person being older or less able to exert strength or reach or move quickly. Can you dock the single engine with no thrusters? Yes. But do you want to be docking that day after day after day after day? I don't. That's where I'd absolutely want thrusters. If I'm alone do I want to have to dock from the bridge? No. Our friends at Hatteras on current models seem to have a strong dislike for lower helms. Now some do have controls along the side or stern, so if all you have is an upper helm some other controls can save you.
To me then size enters into it in other ways. Convenience of anchoring or docking. Cost of docking. And maintenance. If I'm single handing, I want my entire job to be as easy as possible, not just safe. The more boat there is, the more I have to wash down. The more to wax or paint. Then we get to the engine room. The reason I wouldn't ever single hand a 130' is I couldn't take care of engine issues and other things of that nature. So single handing you do need to think of your ability to handle problems. Can you look quickly below? Can you quickly anchor and then address issues? You can check the engine room quicker on a 32' where it's accessed through a hatch in the cockpit than you can a 100' with a stern passage or steps down from the aft deck.
Very important to this is how much you'll be single handing. I think if it's occasional the answer is far different from all the time. Then how many and how frequent guests. Size brings you three things, assuming design and other factors are equal. 1-Seaworthiness, 2-More staterooms, 3-More salon and deck area for people to spread out. If you're boating alone the staterooms and area to spread out are non issues.
Do you need seaworthiness for the OP's goals? I think a certain amount for the Great Lakes and for options in some of the other areas. You need either some seaworthiness to go to the Bahamas or some speed to make sure your weather window is long enough. Most any boat can go to Bimini though and then island hop. Even at 7 knots you can make it to Bimini in a day.
If single handing I'd think you would be more cautious is selecting weather as well. Rough conditions can cause things to happen that are unlikely in calm.
So ultimately there are very few trawlers or trawler type boats, such as Bayliner and Mainship, that can't meet your needs. Single handing I would absolutely look for bow thrusters and access to side decks. Really any of the boats from 32' to 60' would work. However, the larger boats just aren't needed for your intended use. That's why I'd probably aim in the 30-40' range. As to single handing a 48' Kadey Krogen, easy. But unless you have some needs you haven't expressed, then you just don't have the need for 48'.
