Yes, it does damage!
I played a wedding years ago. The bugs came out, and since playing fast fiddle tunes and getting stabbed by mosquitos doesn't really mix, I grabbed my Deet. Days later, where my arm was rubbing against my mandolin, all the finish was peeling off. I still use the stuff, but...