Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-15352 Topology aware block replication
  3. SPARK-15355

Pro-active block replenishment in case of node/executor failures

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.2.0
    • Block Manager, Spark Core
    • None

    Description

      Spark currently does not replenish lost replicas. For resiliency and high availability, BlockManagerMasterEndpoint can proactively verify whether all cached RDDs have enough replicas, and replenish them, in case they don’t.

      Attachments

        Issue Links

          is related to

          Sub-task - The sub-task of the issue SPARK-33870 Enable spark.storage.replication.proactive by default

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #14412 [Github] Pull Request #14412 (shubhamchopra)

          Activity

            People

              shubhamc Shubham Chopra
              shubhamc Shubham Chopra
              Votes:
              2 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: