Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-811

Add metrics, failure reporting and additional tests for HDFS-457

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 0.22.0
    • 0.22.0
    • test
    • None
    • Reviewed

    Description

      HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.

      Description of HDFS-457

      Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
      This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.

      I suggest following additional tests for this improvement.

      #1 Test successive volume failures ( Minimum 4 volumes )
      #2 Test if each volume failure reports reduction in available DFS space and remaining space.
      #3 Test if failure of all volumes on a data nodes leads to the data node failure.
      #4 Test if correcting failed storage disk brings updates and increments available DFS space.

      Attachments

        1. hdfs-811-6.patch
          37 kB
          Eli Collins
        2. hdfs-811-5.patch
          37 kB
          Eli Collins
        3. hdfs-811-4.patch
          34 kB
          Eli Collins
        4. hdfs-811-3.patch
          34 kB
          Eli Collins
        5. hdfs-811-2.patch
          26 kB
          Eli Collins
        6. hdfs-811-1.patch
          23 kB
          Eli Collins

        Issue Links

          Activity

            People

              eli Eli Collins
              raviphulari Ravi Phulari
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: