Well, since everyone else is throwing out opinions I will too.
Personally, around here I see way more small boats than large except on the river. I've fished out of a Jon boat and lots of other small aluminum boats and never felt like it was a waste of time or anything. Personally I am probably going to restore a 12' Jon here this year to use on the small restricted lakes around here. Now, I actually have a full size Blazer that I'm working on, so towing wouldn't be an issue for me even with a large boat. That said, I fully expect that even my Subaru would tow the little Jon around and be fine with it (won't weigh more than 4-500lbs loaded) and as someone who tows 10's of thousands of pounds for a living I would feel pretty comfortable doing so. Adding weight behind you, especially with no trailer brakes requires you to do everything a little slower and increase your stopping distance in general. Lane change= slower, braking= slower, etc. It'll be tougher on any engine, and transmission so good maintenance is key and expect something to wear out a little quicker. That's the trade off unless you buy something designed from the get go for towing. As for any Toyota product, they're usually over engineered so I would think it would hold up fine if you're staying well under the maximum tow weight.
As to why you don't see many small Utes/vans towing boats (even little ones); Why do we see people with 3/4 ton pick-ups just getting groceries and the occasional run to Lowes to pick up a couple hundred pounds of material for that weekend project? Most people go way overboard on vehicles that way. I can't even name the number of people I work with that have 60-90k dollar trucks in the parking lot at work that could tow 20k plus that will never see a trailer or more than 500lbs in the bed. Half have off-road packages too for that drive up a gravel road in buck season or the 10 days of serious snow we usually get. Most people over buy on their trucks in my experience/opinion. I'm guessing with the fuel prices continuing to rocket up we're going to see this change and you're going to see more and more smaller trucks, vans, Utes used going forward.