Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1161

Make DN minimum valid volumes configurable

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.21.0, 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The minimum number of non-faulty volumes to keep the DN active is hard-coded to 1. It would be useful to allow users to configure this value so the DN can be taken offline when eg half of its disks fail, otherwise it doesn't get reported until it's down to it's final disk and suffering degraded performance.

      1. HDFS-1161-y20.patch
        16 kB
        Konstantin Shvachko
      2. hdfs-1161-6.patch
        5 kB
        Eli Collins
      3. hdfs-1161-5.patch
        4 kB
        Eli Collins
      4. hdfs-1161-4.patch
        12 kB
        Eli Collins
      5. hdfs-1161-3.patch
        11 kB
        Eli Collins
      6. hdfs-1161-2.patch
        9 kB
        Eli Collins
      7. hdfs-1161-1.patch
        9 kB
        Eli Collins

        Issue Links

          Activity

          Hide
          Eli Collins added a comment -

          Patch attached. Adds configuration option dfs.datanode.min.valid.volumes and test coverage in TestDiskError.

          Show
          Eli Collins added a comment - Patch attached. Adds configuration option dfs.datanode.min.valid.volumes and test coverage in TestDiskError.
          Hide
          Eli Collins added a comment -

          The diff in the previous patch is missing some additional tests.

          Show
          Eli Collins added a comment - The diff in the previous patch is missing some additional tests.
          Hide
          Eli Collins added a comment -

          Adds documentation of the parameter to hdfs-default.xml, error logging and test cases for invalid configuration values.

          Show
          Eli Collins added a comment - Adds documentation of the parameter to hdfs-default.xml, error logging and test cases for invalid configuration values.
          Hide
          Koji Noguchi added a comment -

          Thanks Eli, I think this would solve my worry mentioned in HDFS-1158.
          If possible, can we flip the meaning of configuration param to
          "dfs.datanode.min.invalid.volumes.before.shutdown" and set it to 1 so that default behavior would be to shutdown the datanode when even a single disk goes down?

          Show
          Koji Noguchi added a comment - Thanks Eli, I think this would solve my worry mentioned in HDFS-1158 . If possible, can we flip the meaning of configuration param to "dfs.datanode.min.invalid.volumes.before.shutdown" and set it to 1 so that default behavior would be to shutdown the datanode when even a single disk goes down?
          Hide
          Eli Collins added a comment -

          Hey Koji, good idea - that seems easier for administrators. How about "dfs.datanode.failed.volumes.tolerated" that defaults to 0 (ie any volume failure causes the DN to stop service)? As with the current patch an error will be logged if the value is out of bounds, in this case set to the total number of volumes or greater (the DN should have at least one volume available).

          Show
          Eli Collins added a comment - Hey Koji, good idea - that seems easier for administrators. How about "dfs.datanode.failed.volumes.tolerated" that defaults to 0 (ie any volume failure causes the DN to stop service)? As with the current patch an error will be logged if the value is out of bounds, in this case set to the total number of volumes or greater (the DN should have at least one volume available).
          Hide
          Koji Noguchi added a comment -

          How about "dfs.datanode.failed.volumes.tolerated" that defaults to 0 (ie any volume failure causes the DN to stop service)?

          +1

          HDFS-457 was introduced in 0.21, can we somehow include this as part of 0.21 as well ?

          Show
          Koji Noguchi added a comment - How about "dfs.datanode.failed.volumes.tolerated" that defaults to 0 (ie any volume failure causes the DN to stop service)? +1 HDFS-457 was introduced in 0.21, can we somehow include this as part of 0.21 as well ?
          Hide
          Eli Collins added a comment -

          Yes, we should put this in 21 so we don't revert the behavior in 22. Updating the fix version, will do so for dependent jiras as well.

          Show
          Eli Collins added a comment - Yes, we should put this in 21 so we don't revert the behavior in 22. Updating the fix version, will do so for dependent jiras as well.
          Hide
          Eli Collins added a comment -

          Patch attached, incorporates Koji's suggestion.

          Show
          Eli Collins added a comment - Patch attached, incorporates Koji's suggestion.
          Hide
          dhruba borthakur added a comment -

          Code looks good. The Unit test should ideally not hardcode the 15 second wait time (instead wait indefinitely). also, can we achieve the behaviour that Koji wants using the existing config parameters (instead of creating new config parameters)?

          Show
          dhruba borthakur added a comment - Code looks good. The Unit test should ideally not hardcode the 15 second wait time (instead wait indefinitely). also, can we achieve the behaviour that Koji wants using the existing config parameters (instead of creating new config parameters)?
          Hide
          Eli Collins added a comment -

          The Unit test should ideally not hardcode the 15 second wait time (instead wait indefinitely).

          It needs to wait at least that long to get the necessary heart beats, but I think it is reasonable to put that check in a loop so that in the case when heartbeats are delayed in the test the test is robust to that.

          also, can we achieve the behaviour that Koji wants using the existing config parameters (instead of creating new config parameters)?

          There are no existing configuration parameters for this, just a constant hard-coded to 1, so we need to add one.

          Show
          Eli Collins added a comment - The Unit test should ideally not hardcode the 15 second wait time (instead wait indefinitely). It needs to wait at least that long to get the necessary heart beats, but I think it is reasonable to put that check in a loop so that in the case when heartbeats are delayed in the test the test is robust to that. also, can we achieve the behaviour that Koji wants using the existing config parameters (instead of creating new config parameters)? There are no existing configuration parameters for this, just a constant hard-coded to 1, so we need to add one.
          Hide
          Konstantin Shvachko added a comment -
          1. This patch does not apply to current trunk anymore. Eli, could you please update it.
          2. There are several "white space" changes in the patch, like
            -    
            -    if(hasEnoughResource) {
            +    if (hasEnoughResources) {
            

            please avoid.

          3. I see you are changing the name of a public method hasEnoughResource() to hasEnoughResources(). Formally we should go through deprecation process, as we don't know all dependencies of the method. So if possible please avoid this too.
          4. Dhruba, if you have any particular existing config parameter, which can be used instead of introducing the new one, please speak up.
          Show
          Konstantin Shvachko added a comment - This patch does not apply to current trunk anymore. Eli, could you please update it. There are several "white space" changes in the patch, like - - if (hasEnoughResource) { + if (hasEnoughResources) { please avoid. I see you are changing the name of a public method hasEnoughResource() to hasEnoughResources() . Formally we should go through deprecation process, as we don't know all dependencies of the method. So if possible please avoid this too. Dhruba, if you have any particular existing config parameter, which can be used instead of introducing the new one, please speak up.
          Hide
          Eli Collins added a comment -

          Thanks for taking a look. Patch attached. It's the minimal patch that applies against trunk. Will move the unit test over to HDFS-811 and the cleanup over to HDFS-1160.

          Show
          Eli Collins added a comment - Thanks for taking a look. Patch attached. It's the minimal patch that applies against trunk. Will move the unit test over to HDFS-811 and the cleanup over to HDFS-1160 .
          Hide
          Konstantin Shvachko added a comment -

          +1. Looks good.

          Show
          Konstantin Shvachko added a comment - +1. Looks good.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12445590/hdfs-1161-5.patch
          against trunk revision 948260.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/182/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/182/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/182/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/182/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445590/hdfs-1161-5.patch against trunk revision 948260. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/182/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/182/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/182/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/182/console This message is automatically generated.
          Hide
          Eli Collins added a comment -

          I get the TestDataNodeVolumeFailure failure due to timeout on trunk without my change also. Haven't seen a jira for this go by, anyone else seeing this?

          Show
          Eli Collins added a comment - I get the TestDataNodeVolumeFailure failure due to timeout on trunk without my change also. Haven't seen a jira for this go by, anyone else seeing this?
          Hide
          Konstantin Shvachko added a comment -

          Eli, I ran TestDataNodeVolumeFailure on my machine with and without your patch.
          Without your patch it succeeds. With your patch it falls into an infinite loop waiting for replication and finally times out.
          The main difference is that after a disk error

          [junit] org.apache.hadoop.util.DiskChecker$DiskErrorException: directory is not writable: /home/shv/kryptonite/hdfs/build/test/data/dfs/data/data3/current/finalized
          [junit]     at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:96)
          [junit]     at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSDir.checkDirTree(FSDataset.java:226)
          [junit]     at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.checkDirs(FSDataset.java:412)
          [junit]     at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolumeSet.checkDirs(FSDataset.java:615)
          [junit]     at org.apache.hadoop.hdfs.server.datanode.FSDataset.checkDataDir(FSDataset.java:1667)
          [junit]     at org.apache.hadoop.hdfs.server.datanode.DataNode.checkDiskError(DataNode.java:763)
          [junit]     at org.apache.hadoop.hdfs.server.datanode.FSDataset.validateBlockFile(FSDataset.java:1543)
          [junit]     at org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:911)
          [junit]     at org.apache.hadoop.hdfs.server.datanode.FSDataset.getMetaFile(FSDataset.java:699)
          [junit]     at org.apache.hadoop.hdfs.server.datanode.FSDataset.metaFileExists(FSDataset.java:799)
          [junit]     at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:120)
          [junit]     at org.apache.hadoop.hdfs.server.datanode.DataXceiver.opReadBlock(DataXceiver.java:169)
          [junit]     at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$Receiver.opReadBlock(DataTransferProtocol.java:353)
          [junit]     at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$Receiver.processOp(DataTransferProtocol.java:325)
          [junit]     at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:113)
          [junit]     at java.lang.Thread.run(Thread.java:619)
          

          current code decides the DN should shutdown, while your patch decides to keep DN running.

           current> [junit] 2010-05-27 20:35:21,871 WARN  datanode.DataNode (DataNode.java:handleDiskError(771)) - DataNode.handleDiskError: Keep Running: true
          patched> [junit] 2010-05-27 19:22:45,636 WARN  datanode.DataNode (DataNode.java:handleDiskError(771)) - DataNode.handleDiskError: Keep Running: false
          

          This triggers different error reports to NN. Please try yourself.

          Show
          Konstantin Shvachko added a comment - Eli, I ran TestDataNodeVolumeFailure on my machine with and without your patch. Without your patch it succeeds. With your patch it falls into an infinite loop waiting for replication and finally times out. The main difference is that after a disk error [junit] org.apache.hadoop.util.DiskChecker$DiskErrorException: directory is not writable: /home/shv/kryptonite/hdfs/build/test/data/dfs/data/data3/current/finalized [junit] at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:96) [junit] at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSDir.checkDirTree(FSDataset.java:226) [junit] at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.checkDirs(FSDataset.java:412) [junit] at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolumeSet.checkDirs(FSDataset.java:615) [junit] at org.apache.hadoop.hdfs.server.datanode.FSDataset.checkDataDir(FSDataset.java:1667) [junit] at org.apache.hadoop.hdfs.server.datanode.DataNode.checkDiskError(DataNode.java:763) [junit] at org.apache.hadoop.hdfs.server.datanode.FSDataset.validateBlockFile(FSDataset.java:1543) [junit] at org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:911) [junit] at org.apache.hadoop.hdfs.server.datanode.FSDataset.getMetaFile(FSDataset.java:699) [junit] at org.apache.hadoop.hdfs.server.datanode.FSDataset.metaFileExists(FSDataset.java:799) [junit] at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:120) [junit] at org.apache.hadoop.hdfs.server.datanode.DataXceiver.opReadBlock(DataXceiver.java:169) [junit] at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$Receiver.opReadBlock(DataTransferProtocol.java:353) [junit] at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$Receiver.processOp(DataTransferProtocol.java:325) [junit] at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:113) [junit] at java.lang. Thread .run( Thread .java:619) current code decides the DN should shutdown, while your patch decides to keep DN running. current> [junit] 2010-05-27 20:35:21,871 WARN datanode.DataNode (DataNode.java:handleDiskError(771)) - DataNode.handleDiskError: Keep Running: true patched> [junit] 2010-05-27 19:22:45,636 WARN datanode.DataNode (DataNode.java:handleDiskError(771)) - DataNode.handleDiskError: Keep Running: false This triggers different error reports to NN. Please try yourself.
          Hide
          Eli Collins added a comment -

          Patch attached. Identified the issue, TestDataNodeVolumeFailure needs the number of tolerated volume failures to be 1 since that test expects the current behavior (the DN keeps working as long as there is a single failed volume remaining). The configuration parameter is further tested in the tests in HDFS-811.

          Show
          Eli Collins added a comment - Patch attached. Identified the issue, TestDataNodeVolumeFailure needs the number of tolerated volume failures to be 1 since that test expects the current behavior (the DN keeps working as long as there is a single failed volume remaining). The configuration parameter is further tested in the tests in HDFS-811 .
          Hide
          Eli Collins added a comment -

          current code decides the DN should shutdown, while your patch decides to keep DN running.

          The behavior is actually the reverse, currently the code will tolerate failures as long as there is one volume still working, my change makes that number configurable and restores the old behavior which was that any failure causes the datanode to stop running.

          Show
          Eli Collins added a comment - current code decides the DN should shutdown, while your patch decides to keep DN running. The behavior is actually the reverse, currently the code will tolerate failures as long as there is one volume still working, my change makes that number configurable and restores the old behavior which was that any failure causes the datanode to stop running.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12445707/hdfs-1161-6.patch
          against trunk revision 948908.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/382/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/382/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/382/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/382/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445707/hdfs-1161-6.patch against trunk revision 948908. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/382/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/382/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/382/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/382/console This message is automatically generated.
          Hide
          Eli Collins added a comment -

          The TestLargeDirectoryDelete timeout is HDFS-615.

          Show
          Eli Collins added a comment - The TestLargeDirectoryDelete timeout is HDFS-615 .
          Hide
          Eli Collins added a comment -

          Kick hudson to see if I can get a clean run.

          Show
          Eli Collins added a comment - Kick hudson to see if I can get a clean run.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12445707/hdfs-1161-6.patch
          against trunk revision 949043.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/383/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/383/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/383/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/383/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445707/hdfs-1161-6.patch against trunk revision 949043. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/383/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/383/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/383/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/383/console This message is automatically generated.
          Hide
          Eli Collins added a comment -

          Current patch is good to go. Putting on Tom's radar for 21.

          Show
          Eli Collins added a comment - Current patch is good to go. Putting on Tom's radar for 21.
          Hide
          Tom White added a comment -

          I've just committed this. Thanks Eli!

          Show
          Tom White added a comment - I've just committed this. Thanks Eli!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #298 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/298/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #298 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/298/ )
          Hide
          Arun C Murthy added a comment -

          Sorry to come in late - we should strive to make this config portable across clusters. Thus, %disks, not #disks would be better? Should I open a diff. jira?

          Show
          Arun C Murthy added a comment - Sorry to come in late - we should strive to make this config portable across clusters. Thus, %disks, not #disks would be better? Should I open a diff. jira?
          Hide
          Eli Collins added a comment -

          That's reasonable. The list of volumes (local dirs) is explicitly listed so the config isn't portable even when specified as a percent, but it's one less config that isn't portable.

          IIRC Koji's perspective was that an admin doesn't want to specify the count or percent of valid volumes, but that after a set number of failures the host should be considered faulty. Eg if it's lost two disks there's probably something wrong whether the host has 6 or 12 disks, ie assumes disk failures w/in a host are correlated.

          Ideally I think we should collect data (eg an X core host can still function well with Y% disks) and not require users configure this at all - it would be enabled by default and the daemons would take themselves offline when they've determined they don't have sufficient resources.

          Show
          Eli Collins added a comment - That's reasonable. The list of volumes (local dirs) is explicitly listed so the config isn't portable even when specified as a percent, but it's one less config that isn't portable. IIRC Koji's perspective was that an admin doesn't want to specify the count or percent of valid volumes, but that after a set number of failures the host should be considered faulty. Eg if it's lost two disks there's probably something wrong whether the host has 6 or 12 disks, ie assumes disk failures w/in a host are correlated. Ideally I think we should collect data (eg an X core host can still function well with Y% disks) and not require users configure this at all - it would be enabled by default and the daemons would take themselves offline when they've determined they don't have sufficient resources.
          Hide
          Koji Noguchi added a comment -

          IIRC Koji's perspective was that an admin doesn't want to specify the count or percent of valid volumes

          What I wanted was to keep the default behavior of shutting down the datanode when hitting faulty volume for 0.21 since we were seeing missing blocks after HDFS-457. I don't have preference between %disks and #disks.

          Show
          Koji Noguchi added a comment - IIRC Koji's perspective was that an admin doesn't want to specify the count or percent of valid volumes What I wanted was to keep the default behavior of shutting down the datanode when hitting faulty volume for 0.21 since we were seeing missing blocks after HDFS-457 . I don't have preference between %disks and #disks.

            People

            • Assignee:
              Eli Collins
              Reporter:
              Eli Collins
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development