The perpetual ice that I am aware of is in King Ravine. If you take the "subway" once in the ravine, you will likely spot some ice below the boulders. I always have in this area.
I'm not sure there's much to explore. I went by them in August on my way up Great Gully, and remember that blast of cold air, but not much else. I don't recall there being any caves, but maybe they're hidden in the dense undergrowth.
Chemin des Dames has a section of boulder caves called Tunnel Rock according to the AMC Guide. I have not done this trail so I can't say if it is a true cave but I suspect it is just a hollow like the large boulders at the base of The Fan in Huntington Ravine. (i.e. cave-like but not a true cave).
The AMC guide also differentiates the Subway from the Ice Caves Loop further up the ravine. It references the Ice Caves Loop as the location to find ice year round, although I suspect most of those hidden nooks and crannies can hold ice year round. It is just after the Great Gully junction.
After the subway and the main trail rejoin, and after the Great Gully junction, the trail divides again. Take the right branch to the "caves" and start to explore amoung the boulders. I've always found ice(5 times). VFTT member Trail Trotter found ice in August this year and also in Ice Gulch just to the north of King Ravine in Sept. so there should be some. There are some small boulder caves to explore.
Edit: Daytrip beat me, I got distracted by the Sox game.
King Ravine has always been a special place for me. As a teenager living in NJ, I attended a hiking camp that Killington ran. My first introduction to the Whites was hiking up the King Ravine trail. As we emerged into the open floor of the ravine, my jaw just dropped. Many years later I made my 1st and only trip to the world famous Tuckermans Ravine and I still like King Ravine a lot more. And it's not just the lack of the crowds!