I totally agree! The "Oh my god, you're taking down!! What if it gets went??" rant drives me nuts. There's a simple sollution... don't get it wet.
I think a down jacket is necessary to carry in the winter while hiking, skiing, snowshoeing etc. Unless you can still see your car, take it...