Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27499

Support mapping spark.local.dir to hostPath volume

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Minor
    • Resolution: Invalid
    • 3.0.0
    • None
    • Kubernetes, Spark Core
    • None

    Description

      Currently, the k8s executor builder mount spark.local.dir as emptyDir or memory, it should satisfy some small workload, while in some heavily workload like TPCDS, both of them can have some problem, such as pods are evicted due to disk pressure when using emptyDir, and OOM when using tmpfs.

      In particular on cloud environment, users may allocate cluster with minimum configuration and add cloud storage when running workload. In this case, we can specify multiple elastic storage as spark.local.dir to accelerate the spilling.

      Attachments

        Issue Links

          is cloned by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-28407 Support mapping spark.local.dir to hostPath volume

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              junjie Junjie Chen
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: