I face a simliar drive, I would say it's doable. The trail that reaches to the Hunter/Rusk col is a fire road, you're likely to make good time, in fact most of the trail walking will be on the easy side of things.
For me I would be cautious on timing if the weather was hot and humid I slow down...