Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26529

Add debug logs for confArchive when preparing local resource

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: 2.3.2, 2.4.0
    • Fix Version/s: 3.0.0
    • Component/s: Deploy, Spark Core
    • Labels:
      None

      Description

      Currently, `Client#createConfArchive` do not handle IOException, and some detail info is not provided in logs. Sometimes, this may delay the time of locating the root cause of io error.

      A case happened in our production environment is that local disk is full, and the following exception is thrown but no detail path info provided. we have to investigate all the local disk of the machine to find out the root cause.

      Exception in thread "main" java.io.IOException: No space left on device
      at java.io.FileOutputStream.writeBytes(Native Method)
      at java.io.FileOutputStream.write(FileOutputStream.java:345)
      at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:253)
      at java.util.zip.ZipOutputStream.closeEntry(ZipOutputStream.java:238)
      at java.util.zip.ZipOutputStream.finish(ZipOutputStream.java:343)
      at java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:238)
      at java.util.zip.ZipOutputStream.close(ZipOutputStream.java:360)
      at org.apache.spark.deploy.yarn.Client.createConfArchive(Client.scala:769)
      at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:657)
      at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:895)
      at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:177)
      at org.apache.spark.deploy.yarn.Client.run(Client.scala:1202)
      at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1261)
      at org.apache.spark.deploy.yarn.Client.main(Client.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:767)
      at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:189)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:214)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:128)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
      

      It make sense for us to catch the IOException and print some useful information.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                liupengcheng liupengcheng
                Reporter:
                liupengcheng liupengcheng
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: