Yup. Puffy over everything the instant I stop; for longer stops, or when in camp, get rid of the shell, so it's base layer>mid insulation>puffy, and possible a shell over all that.
That said, it can be tough to keep down from getting damp anyway. Even just throwing it on at breaks, etc., its...