Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-9203

Make filesystem pluggable in BlockStore and BlockManager

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • None
    • None
    • Block Manager, Spark Core

    Description

      I was looking through the code in order to understand better how RDD is persisted to Tachyon off-heap filesystem. It looks like that the Tachyon filesystem is hard-coded and there is no way to switch to another in-memory filesystem. I think it would be great if the implementation of the BlockManager and BlockStore would be able to plug in another filesystem.

      For example, Apache Ignite also has an implementation of in-memory filesystem which can store data in on-heap and off-heap formats. It would be great if it could integrate with Spark.

      Apache Ignite In-Memory Filesystem: https://ignite.incubator.apache.org/features/igfs.html

      Attachments

        Issue Links

          relates to

          New Feature - A new feature of the product, which has yet to be developed. SPARK-6112 Provide external block store support through HDFS RAM_DISK

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              agoncharuk Alexey Goncharuk
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: