Make sure we only call chmod when pull_path or pub_path is used. Add regression tests for this behavior.