I did it the 'difficult' way with my kids when they were 7 and 9. I was nervous. VERY nervous. I just made sure they didn't go running around, but it was fine. Oh yeah. We did it with full packs. There were a couple of places where we had to remove the packs, I went down, kids passed me the packs, then I 'carried' them down.
The kids found it FUN. Most kids seem to likd scrambling on bare rock. Yes, rain makes if more tricky, and ice makes it downright dangerous, but if it's a nice day, with YOUR kids, I wouldn't worry at all.
Camping around the Johns Brook ranger station is nice. It isn't that much of a pack in. Mostly flat. A short walk down and the kids can play in the brook while you're cooking supper. Sure, you'd have no problem doing it in a day, but if you're thinking of camping, it can be nice and relaxing. You can take a bit of time to look at Bushnell Falls, instead of rushing past it.