[SPARK-40765] Optimize redundant fs operations in `CommandUtils#calculateSingleLocationSize#getPathSize` method

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: SQL
• Assignee: Yang Jie
• Reporter: Yang Jie
• Created: 12/Oct/22 04:02
• Updated: 12/Dec/22 18:10
• Resolved: 13/Oct/22 02:21

Description

def getPathSize(fs: FileSystem, path: Path): Long = {
      val fileStatus = fs.getFileStatus(path)
      val size = if (fileStatus.isDirectory) {
        fs.listStatus(path)
          .map { status =>
            if (isDataPath(status.getPath, stagingDir)) {
              getPathSize(fs, status.getPath)
            } else {
              0L
            }
          }.sum
      } else {
        fileStatus.getLen
      }      size
    } 

Change input parameter from `Path` to `FileStatus`,  there is no need to do `fs.getFileStatus(path)` after each recursive call.

Issue Links

• links to: [Github] Pull Request #38214 (LuciferYang)