Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12196

Store/retrieve blocks in different speed storage devices by hierarchy way

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • None
    • None
    • Spark Core

    Description

      Motivation
      Our customers want to use SSD to speed up machine learning and SQL workload, but all SSDs are quite expensive and SSD's capacity is still smaller than HDD.

      Proposal
      Our solution is to build tiered storage: use SSDs as cache and HDDs as backup.
      When Spark core allocates blocks (either for shuffle or RDD cache), it stores blocks in SSDs first, and when the SSD’s free space is less than some threshold, starting to use HDDs.

      Performance Evaluation
      1. At the best case, our solution performs the same as all SSDs.
      2. At the worst case, like all data are spilled to HDDs, no performance regression.
      3. Compared with all HDDs, tiered store improves +x2 for machine learning workload and +x1.7 for SparkSQL workload.

      Usage
      1. Enable tiered storage in spark-default.conf.

      spark.diskStore.allocator      tiered
      

      2. Configure storage hierarchy, for Yarn user, see below example:

        <property>
          <name>yarn.nodemanager.local-dirs</name>
          <value>/mnt/DP_disk1/yucai/yarn/local,/mnt/DP_disk2/yucai/yarn/local,
                 /mnt/DP_disk3/yucai/yarn/local,/mnt/DP_disk4/yucai/yarn/local,
                 /mnt/DP_disk5/yucai/yarn/local,/mnt/DP_disk6/yucai/yarn/local,
          </value>
        </property>
        <property>
          <name>yarn.nodemanager.spark-dirs-tiers</name>
          <value>001111</value>
        </property>
      

      It means DP_disk1-2 are in tier1 and DP_disk2-6 make up tier2.

      More tiers
      In our implementation, we support to build any number tiers cross various storage medias (NVMe, SSD, HDD etc.). For example:

        <property>
          <name>yarn.nodemanager.local-dirs</name>
          <value>/mnt/DP_disk1/yucai/yarn/local,/mnt/DP_disk2/yucai/yarn/local,
                 /mnt/DP_disk3/yucai/yarn/local,/mnt/DP_disk4/yucai/yarn/local,
                 /mnt/DP_disk5/yucai/yarn/local,/mnt/DP_disk6/yucai/yarn/local,
          </value>
        </property>
        <property>
          <name>yarn.nodemanager.spark-dirs-tiers</name>
          <value>001122</value>
        </property>
      

      It means DP_disk1-2 are in tier1, DP_disk3-4 are in tier2 and DP_disk5-6 make up tier3.

      Attachments

        Activity

          People

            Unassigned Unassigned
            yucai yucai
            Votes:
            3 Vote for this issue
            Watchers:
            23 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: