Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31591

namePrefix could be null in Utils.createDirectory

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Not A Problem
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: Spark Core
    • Labels:
      None

      Description

      In our production, we find that many shuffle files could be located in
      /hadoop/2/yarn/local/usercache/b_carmel/appcache/application_1586487864336_4602/null-107d4e9c-d3c7-419e-9743-a21dc4eaeb3f/3a

      The Util.createDirectory() uses a default parameter "spark"

        def createDirectory(root: String, namePrefix: String = "spark"): File = {
      

      But in some cases, the actual namePrefix is null. If the method is called with null, then the default value would not be applied.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              cltlfcjin Lantao Jin
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: