For kayaking in the Groton/New London area, you can put in at Bluff Point, and paddle out to Ledge Light (dodging the ferries). But if you've got time, you're better off heading a little further down I-95 S to Selden Neck. That's where the CT river enters Long Island Sound and there's a big salt marsh. It's the only major river in the US that doesnt have a port at it's mouth due to the big sand bar.
Another favorite is to go to Mystic, and put in under the I-95 bridge, and then you can paddle down the Mystic river and around Mystic Seaport. That's a real nice paddle if you do it at night, especially if there's a full moon (weekend of the 20th). There's a public dock just down stream of the drawbridge in Mystic, so you can get out & get some ice cream... for some reason, ice cream seems to figure in a lot of local kayaking.
Last kayak trip would be to go a bit further east and put in at Barn Island. From there, you can paddle to Watch Hill, RI. Put in near the end of beach, and yes, ice cream.
For details, check
the CT Coastal Access Guide and
the CT Boaters Guide
For hiking, as noted Bluff Piont's pretty good. You can get on a beach & through some marsh, although it gets a little crowded on weekends. If you want "rugged," I suggest Lantern Hill & the Narragensett Trail. It starts just east of Foxwoods, so there's plenty of signs to follow. Not the most difficult trail in the world, but it has it's points.