I will speak from my personal experience in owning a second home.
We are three hours from home, when travelling with our family of four, this is right on the edge of tolerable. With just myself or with my wife, three hours is a snap. We have owned our second home since our second child was born and she is now 17 so we went through the whole gamut of travelling experiences.
I grew up in a family with a second home, also in Maine. My wife & I used that house for many years before we searched for our own house; my family's was on a seasonal road and it was not winterized. We eventually opted to build our own house as nothing we could find matched our needs as we perceived them. We now have a house on a smaller pond (180ac) in Southern Oxford county Maine. This is a smaller, year round house on a private dirt road. Our taxes are not too bad as there are a couple of larger lakes in town that support a great portion of the tax base in town. When we bought, a comparable lot on one of the larger lakes was 2x the purchase price with 2x the taxes in the same town. My guess is that smaller lakes still cost less.
Location.....a lot of considerations here. This issue is one big compromise for us. The various factors for us included waterfront, skiing, snowmobiling and hiking. We are 45 min to 1 hr from skiing. 1 hr from N. Conway, 1 hr from the ME/NH RT 2 border. We can snowmobile from the yard. I have also learned that being reasonably close to services is also very helpful. It is awfully nice to have most everything we need available to us in a 7-10 mile drive from home. This includes food shopping, hospital, lumber yard, Wal-Mart, Tractor Supply and most other services that I could possibly want; over the years, this has become surprisingly important. Also, more important with time, is the proximity to the highway. I have many different ways to go from the highway to my house, each has their own benefits. If you are going to be a frequent traveler, to me, it is a big disadvantage to have a long drive off highway as a regular part of your trip (think bad weather)
Costs; I get my driveway plowed for $15, by the same Co. that plows our road. My heating bill is about $1000 per year (temp at 50° or we heat with wood when we are there) I have a temp alarm that calls me if the temp were to drop below 40° You do need a network of friends or a caretaker to feel most comfortable. We are in a small development with some full time residents and some from away as we are. I enjoy the work around the house, sometimes that is my excuse to go to Maine while the family stays home. We did build with simplicity and low maintenance in mind. Heat with hot air, it has the quickest recovery and it is not susceptible to water freezing or leaking. Assume that you will lose power and you may freeze the house. If you can, build the plumbing to drain. All of our plumbing is concentrated on the first floor, in one small area over an unfinished portion of our basement. The basement even has floor drains.
Gear/toys: The big surprise for me has been the amount of gear that we have accumulated. We have dedicated a surprising amount of storage for this gear; do not underestimate this need. Alpine ski gear, nordic ski gear, snowmobile gear, hiking gear, skates, pond toys, snowmobiles, kayaks, canoe, sailboat, ski boat, bikes, boat trailers, snowmobile trailers and all of the accompanying clothing
Buying: research and patience...We completely changed our focus through our initial search many years ago. We started looking further north in the mountains of Maine. Ended up on a pond 1 hour closer to home. Look, look and then look some more. Do your due diligence, one of my favorite tips is to use a fishing guide to look at the quality of the body of water. There is a pond local to us that has over 200ac of surface but the depth is only 8' and it is supported by a beaver dam. Over the last 10 years, they have developed 50% of the shoreline on that pond by selling to people out of state, buyer beware. Also, be cautious about road associations including covenants that might be part of any deed; often covenants were used to make up for a lack of zoning. We have continued to add to our land and in each case, the transactions have benefited from our patience; we have assumed that we were in the drivers seat for each transaction and each time worked that angle to our advantage. Most agents do not know crap about what they are selling, at least in Maine. Our experiences have been comical including buying property that the agent never even set foot on and also buying property from a listing agent that didn't even know that they were listing the property. We did all of the leg work in each of our transactions, leaving no stone unturned. Learn about the seller, in 2 cases we bought from distressed sellers and in the third case, the seller was living in Switzerland and we dealt with him directly. Knowledge is power.
Most importantly, install a washer/dryer right from the start!! No excuses, leave the vacation clothes in the vacation home. No going home early to do laundry!!
We do not use the house as often as I would hope over the last few years. However, this has been due to the various things that my kids have been engaged in at home. This was not much of an issue when the kids were younger and it will become less of an issue as our kids get older. It is simply the stage of life that we are in right now.
I often fantasize about not owning the second home, it would be so much easier to rent in different places throughout the year. But, in the long run, I have come to truly enjoy the flexibility afforded by owning our own vacation home. It is pretty sweet arriving on a friday night with nothing much to unload other than a cooler of fresh food for the weekend.