I've seen you mention this a couple of times, so I have to mention it. There are only two harbors on Fishers island.
West Harbor: Has a few moorings and a restaurant, and an ice cream shop.
East Harbor: Tiny anchorage. No means to go ashore.
I personally wouldn't bother with either harbors on Fisher's island. Mystic, CT and Watch Hill are significantly better. If I were doing a stop-over I'd take a mooring in West Harbor. There is more to do on Bank Street in New London, even if just a stopover. The New London Moorings are admittedly somewhat urban and there is train and ferry noise. However, there are a ton of restaurants and bars if you want to leave the boat.
If you're going to go through Fishers Island Sound, I would focus on New London, Mystic or Watch Hill. Plenty of room in the Watch Hill anchorage.
You can also anchor at Rams Island, but you can't go ashore there either.