Hadoop Common
  1. Hadoop Common
  2. HADOOP-4610

Always calculate mis-replicated blocks when safe-mode is turned off.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.18.0
    • Fix Version/s: 0.18.3
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      After HADOOP-4597 there is one last case when mis-replicated blocks are not processed. It is when the safe-mode extension = 0.
      We should regenerate neededReplications and excessReplicateMap whenever the safe mode is turned off.

      1. NeededReplAlways.patch
        4 kB
        Konstantin Shvachko
      2. NeededReplAlways-0.18.patch
        4 kB
        Konstantin Shvachko

        Issue Links

          Activity

          Hide
          Konstantin Shvachko added a comment -

          This patch removes the boolean parameter which would be always set to true otherwise.

          Show
          Konstantin Shvachko added a comment - This patch removes the boolean parameter which would be always set to true otherwise.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12393482/NeededReplAlways.patch
          against trunk revision 712033.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3548/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3548/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3548/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3548/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12393482/NeededReplAlways.patch against trunk revision 712033. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3548/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3548/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3548/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3548/console This message is automatically generated.
          Hide
          Raghu Angadi added a comment -

          I think this should go into 0.18.3. This is a serious regression if 'extension' is set to 0 in some config. It will be many month before 0.20. becomes the stable branch. Note that we would have thought threshold being set to 1.1 would be rare.. but it happens.

          +1 for the patch.

          Show
          Raghu Angadi added a comment - I think this should go into 0.18.3. This is a serious regression if 'extension' is set to 0 in some config. It will be many month before 0.20. becomes the stable branch. Note that we would have thought threshold being set to 1.1 would be rare.. but it happens. +1 for the patch.
          Hide
          Konstantin Shvachko added a comment - - edited

          The same manual test as in HADOOP-4597 is applicable here.

          Show
          Konstantin Shvachko added a comment - - edited The same manual test as in HADOOP-4597 is applicable here.
          Hide
          Konstantin Shvachko added a comment -

          > I think this should go into 0.18.3.

          Setting the 'extension' to 0 does not have a real use case at the moment.
          I do not think we should apply it to 0.18 until a need has been developed.
          I will commit it to 0.19 because this does not require a separate patch and more testing.

          Show
          Konstantin Shvachko added a comment - > I think this should go into 0.18.3. Setting the 'extension' to 0 does not have a real use case at the moment. I do not think we should apply it to 0.18 until a need has been developed. I will commit it to 0.19 because this does not require a separate patch and more testing.
          Hide
          Konstantin Shvachko added a comment -

          I just committed this.

          Show
          Konstantin Shvachko added a comment - I just committed this.
          Hide
          Koji Noguchi added a comment -

          Setting the 'extension' to 0 does not have a real use case at the moment.

          If it's straightforward, it would be nice to have this fixed in 0.18.3.
          We don't know how other users configure their clusters.

          Less blocks go missing, less angry users on the mailing list.

          Show
          Koji Noguchi added a comment - Setting the 'extension' to 0 does not have a real use case at the moment. If it's straightforward, it would be nice to have this fixed in 0.18.3. We don't know how other users configure their clusters. Less blocks go missing, less angry users on the mailing list.
          Hide
          Konstantin Shvachko added a comment -

          I just committed this to 0.18 branch.

          Show
          Konstantin Shvachko added a comment - I just committed this to 0.18 branch.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk #655 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/655/)
          . Reflect commit to 0.18.3 in trunk.
          . Always calculate mis-replicated blocks when safe-mode is turned off. Contributed by Konstantin Shvachko.

          Show
          Hudson added a comment - Integrated in Hadoop-trunk #655 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/655/ ) . Reflect commit to 0.18.3 in trunk. . Always calculate mis-replicated blocks when safe-mode is turned off. Contributed by Konstantin Shvachko.

            People

            • Assignee:
              Konstantin Shvachko
              Reporter:
              Konstantin Shvachko
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development