Clarksburg is close to the Upper and eastern trailheads.. It is popular with the folks starting the Long Trail/AT at the VT/MA border just because of it's location.
I've camped at the Historic valley campground when Clarksburg was full. If you get the walkin sites, they are generally quiet and there is the lake that you can swim in. It's run by the town of North Adams so it's kind of off the radar for that area. I think I would try to avoid Windsor Lake if you can't get the walk ins.. I was there on a weekend and it wasn't what I would call quiet, outside the walkin sites but then again, I had the entire walk in site area to myself and I was on my bike so it wasn't even walk in either as I simply rode my bicycle to my campsite.
You could also camp in Pittsfield State Forest at Berry Pond though not sure if you're allowed to swim at Berry Pond, it's pretty small. I think midweek would be fairly quiet at any of the sites.. The upper campsites in Pittsfield SF are also BYO water as only the lower sites have potable water i.e. the sites on the top of the loop have no potable water. If you don't mind a drive, I've camped at Beartown SF down by Great Barrington and those sites are pretty nice and fairly quiet. October Mtn in Lee has some sites closer to greylock too...
I guess I'll toss this out there too that Sperry Campground isn't truly car camping but it's a 1.4 mile hike from the official parking lot for campers there... And there isn't swimming unless you jump into the march cataract falls...
Jay