Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5342

Allow long running Spark apps to run on secure YARN/HDFS

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.2.0
    • 1.4.0
    • YARN
    • None

    Description

      Currently, Spark apps cannot write to HDFS after the delegation tokens reach their expiry, which maxes out at 7 days. We must find a way to ensure that we can run applications for longer - for example, spark streaming apps are expected to run forever.

      Attachments

        1. SparkYARN.pdf
          114 kB
          Hari Shreedharan

        Issue Links

          Activity

            People

              hshreedharan Hari Shreedharan
              hshreedharan Hari Shreedharan
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: