Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6112

Provide external block store support through HDFS RAM_DISK

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Block Manager
    • Labels:
      None

      Description

      HDFS Lazy_Persist policy provide possibility to cache the RDD off_heap in hdfs. We may want to provide similar capacity to Tachyon by leveraging hdfs RAM_DISK feature, if the user environment does not have tachyon deployed.

      With this feature, it potentially provides possibility to share RDD in memory across different jobs and even share with jobs other than spark, and avoid the RDD recomputation if executors crash.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                zzhan Zhan Zhang
              • Votes:
                1 Vote for this issue
                Watchers:
                30 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: