Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21714

SparkSubmit in Yarn Client mode downloads remote files and then reuploads them again

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 2.2.0
    • 2.2.1, 2.3.0
    • Spark Submit
    • None

    Description

      SPARK-10643 added the ability for spark-submit to download remote file in client mode.

      However in yarn mode this introduced a bug where it downloads them for the client but then yarn client just reuploads them to HDFS and uses them again. This should not happen when the remote file is HDFS. This is wasting resources and its defeating the distributed cache because if the original object was public it would have been shared by many users. By us downloading and reuploading, it becomes private.

      Attachments

        Issue Links

          is broken by

          New Feature - A new feature of the product, which has yet to be developed. SPARK-10643 Support remote application download in client mode spark submit

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-21689 Spark submit will not get kerberos token token when hbase class not found

          • Major - Major loss of function.
          • Closed
          links to

          Pull request #19074 [Github] Pull Request #19074 (jerryshao)

          Web Link https://github.com/apache/spark/pull/18962

          Activity

            People

              jerryshao Saisai Shao
              tgraves Thomas Graves
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: