Hadoop Common
  1. Hadoop Common
  2. HADOOP-4610

Always calculate mis-replicated blocks when safe-mode is turned off.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.18.0
    • Fix Version/s: 0.18.3
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      After HADOOP-4597 there is one last case when mis-replicated blocks are not processed. It is when the safe-mode extension = 0.
      We should regenerate neededReplications and excessReplicateMap whenever the safe mode is turned off.

      1. NeededReplAlways-0.18.patch
        4 kB
        Konstantin Shvachko
      2. NeededReplAlways.patch
        4 kB
        Konstantin Shvachko

        Issue Links

          Activity

          Konstantin Shvachko created issue -
          Hide
          Konstantin Shvachko added a comment -

          This patch removes the boolean parameter which would be always set to true otherwise.

          Show
          Konstantin Shvachko added a comment - This patch removes the boolean parameter which would be always set to true otherwise.
          Konstantin Shvachko made changes -
          Field Original Value New Value
          Attachment NeededReplAlways.patch [ 12393482 ]
          Konstantin Shvachko made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12393482/NeededReplAlways.patch
          against trunk revision 712033.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3548/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3548/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3548/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3548/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12393482/NeededReplAlways.patch against trunk revision 712033. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3548/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3548/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3548/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3548/console This message is automatically generated.
          Hide
          Raghu Angadi added a comment -

          I think this should go into 0.18.3. This is a serious regression if 'extension' is set to 0 in some config. It will be many month before 0.20. becomes the stable branch. Note that we would have thought threshold being set to 1.1 would be rare.. but it happens.

          +1 for the patch.

          Show
          Raghu Angadi added a comment - I think this should go into 0.18.3. This is a serious regression if 'extension' is set to 0 in some config. It will be many month before 0.20. becomes the stable branch. Note that we would have thought threshold being set to 1.1 would be rare.. but it happens. +1 for the patch.
          Konstantin Shvachko made changes -
          Link This issue is related to HADOOP-4597 [ HADOOP-4597 ]
          Hide
          Konstantin Shvachko added a comment - - edited

          The same manual test as in HADOOP-4597 is applicable here.

          Show
          Konstantin Shvachko added a comment - - edited The same manual test as in HADOOP-4597 is applicable here.
          Hide
          Konstantin Shvachko added a comment -

          > I think this should go into 0.18.3.

          Setting the 'extension' to 0 does not have a real use case at the moment.
          I do not think we should apply it to 0.18 until a need has been developed.
          I will commit it to 0.19 because this does not require a separate patch and more testing.

          Show
          Konstantin Shvachko added a comment - > I think this should go into 0.18.3. Setting the 'extension' to 0 does not have a real use case at the moment. I do not think we should apply it to 0.18 until a need has been developed. I will commit it to 0.19 because this does not require a separate patch and more testing.
          Hide
          Konstantin Shvachko added a comment -

          I just committed this.

          Show
          Konstantin Shvachko added a comment - I just committed this.
          Konstantin Shvachko made changes -
          Fix Version/s 0.20.0 [ 12313438 ]
          Resolution Fixed [ 1 ]
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.19.1 [ 12313473 ]
          Hide
          Koji Noguchi added a comment -

          Setting the 'extension' to 0 does not have a real use case at the moment.

          If it's straightforward, it would be nice to have this fixed in 0.18.3.
          We don't know how other users configure their clusters.

          Less blocks go missing, less angry users on the mailing list.

          Show
          Koji Noguchi added a comment - Setting the 'extension' to 0 does not have a real use case at the moment. If it's straightforward, it would be nice to have this fixed in 0.18.3. We don't know how other users configure their clusters. Less blocks go missing, less angry users on the mailing list.
          Konstantin Shvachko made changes -
          Attachment NeededReplAlways-0.18.patch [ 12393545 ]
          Hide
          Konstantin Shvachko added a comment -

          I just committed this to 0.18 branch.

          Show
          Konstantin Shvachko added a comment - I just committed this to 0.18 branch.
          Konstantin Shvachko made changes -
          Fix Version/s 0.19.1 [ 12313473 ]
          Fix Version/s 0.18.3 [ 12313494 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk #655 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/655/)
          . Reflect commit to 0.18.3 in trunk.
          . Always calculate mis-replicated blocks when safe-mode is turned off. Contributed by Konstantin Shvachko.

          Show
          Hudson added a comment - Integrated in Hadoop-trunk #655 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/655/ ) . Reflect commit to 0.18.3 in trunk. . Always calculate mis-replicated blocks when safe-mode is turned off. Contributed by Konstantin Shvachko.
          Nigel Daley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Owen O'Malley made changes -
          Component/s dfs [ 12310710 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          4m 49s 1 Konstantin Shvachko 07/Nov/08 02:27
          Patch Available Patch Available Resolved Resolved
          18h 22m 1 Konstantin Shvachko 07/Nov/08 20:50
          Resolved Resolved Closed Closed
          83d 23h 24m 1 Nigel Daley 30/Jan/09 20:14

            People

            • Assignee:
              Konstantin Shvachko
              Reporter:
              Konstantin Shvachko
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development