I used to operate in that stretch for 15 years as an assistance tower.
I would estimate 90% of reports by transiting, even local boaters were really incorrect. Bet they still are. Most aground claimed they were in the channel, till I showed them they weren't and there was plenty of water in the channel. No different than parts of the ACIW.
There are always areas on any inland waterway that are troublesome, but dredging
and good local knowledge made the passage possible.
The best data available is probably from the local Sea Tow in Cape May. Boat US may be OK...but historically the Sea Tow operation was an all year, not just seasonal operation, had more boats, etc. It may have changed in the last couple years so, check with both if you feel uncertain.
Greater than 4 foot draft hurts, but depending on tide cycles, traveling on the top 6 hrs or even 4 hrs makes it less so. At 50 statute miles or so, doing it in one day may be tough as a couple draw bridges can slow you down, especially if they break as they are old.
Townsend inlet is an easy to use with a quick brief from a local captain thats about 15 miles north of Cape May. Most NJ inlets get snotty with a strong easterly and outgoing current, but probably not dangerous for your trawler if less than 15 knots.