Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7197 Enhancements to Mover
  3. HDFS-8540

Mover should exit with NO_MOVE_BLOCK if no block can be moved

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: balancer & mover
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When there are files not satisfying their storage policy and no move is possible, Mover exits with SUCCESS. It should exit with NO_MOVE_BLOCK.

      The bug seems in the following code. When StorageTypeDiff is not empty and scheduleMoves4Block return false, it does not update hasRemaining. Also, there is no indication of "No block can be moved" for the entire iteration.

      //Mover.processFile(..)
              if (!diff.removeOverlap(true)) {
                if (scheduleMoves4Block(diff, lb, ecSchema)) {
                  hasRemaining |= (diff.existing.size() > 1 &&
                      diff.expected.size() > 1);
                }
              }
      
      1. HDFS-8540_1.patch
        13 kB
        Surendra Singh Lilhore
      2. HDFS-8540_2.patch
        13 kB
        Surendra Singh Lilhore
      3. HDFS-8540_3.patch
        13 kB
        Surendra Singh Lilhore
      4. HDFS-8540.patch
        10 kB
        Surendra Singh Lilhore

        Issue Links

          Activity

          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Thanks Tsz Wo Nicholas Sze for reporting this issue.

          Attached patch, Please review..

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Thanks Tsz Wo Nicholas Sze for reporting this issue. Attached patch, Please review..
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 49s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 31s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 16s The applied patch generated 2 new checkstyle issues (total was 18, now 20).
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          -1 findbugs 3m 17s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings.
          +1 native 3m 15s Pre-build of native portion
          -1 hdfs tests 161m 44s Tests failed in hadoop-hdfs.
              208m 6s  



          Reason Tests
          FindBugs module:hadoop-hdfs
          Failed unit tests hadoop.hdfs.TestAppendSnapshotTruncate
            hadoop.hdfs.server.mover.TestStorageMover



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12738157/HDFS-8540.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a6cb489
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11265/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11265/artifact/patchprocess/whitespace.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/11265/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11265/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11265/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11265/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 49s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 31s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 16s The applied patch generated 2 new checkstyle issues (total was 18, now 20). -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 3m 17s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings. +1 native 3m 15s Pre-build of native portion -1 hdfs tests 161m 44s Tests failed in hadoop-hdfs.     208m 6s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.hdfs.server.mover.TestStorageMover Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12738157/HDFS-8540.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a6cb489 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11265/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11265/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/11265/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11265/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11265/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11265/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Thanks Surendra Singh Lilhore for the patch.
          Idea looks good.
          Instead of creating separate instances of Result everytime while processing recursively, why dont use Only one instance thoughout and update the same result? Anyway you are updating the current result always with the returned result.

          Show
          vinayrpet Vinayakumar B added a comment - Thanks Surendra Singh Lilhore for the patch. Idea looks good. Instead of creating separate instances of Result everytime while processing recursively, why dont use Only one instance thoughout and update the same result? Anyway you are updating the current result always with the returned result.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          The patch computes failToScheduleBlock but failToScheduleBlock is different from NO_MOVE_BLOCK since we could have both success and fail to schedule blocks. In this case, it should not return NO_MOVE_BLOCK.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - The patch computes failToScheduleBlock but failToScheduleBlock is different from NO_MOVE_BLOCK since we could have both success and fail to schedule blocks. In this case, it should not return NO_MOVE_BLOCK.
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Thanks Vinayakumar B and Tsz Wo Nicholas Sze for reviewing and suggestion.

          @Tsz Wo Nicholas Sze : you mean it should exit with NO_MOVE_BLOCK when in entire iteration no blocks are moved.

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Thanks Vinayakumar B and Tsz Wo Nicholas Sze for reviewing and suggestion. @ Tsz Wo Nicholas Sze : you mean it should exit with NO_MOVE_BLOCK when in entire iteration no blocks are moved.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Yes, NO_MOVE_BLOCK means that "No block can be moved". If some blocks can be moved in current iteration, space may be freed up so that the blocks failed to move may possibly be moved in the next iteration.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Yes, NO_MOVE_BLOCK means that "No block can be moved". If some blocks can be moved in current iteration, space may be freed up so that the blocks failed to move may possibly be moved in the next iteration.
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Thanks Tsz Wo Nicholas Sze

          I attached new patch, Please review.

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Thanks Tsz Wo Nicholas Sze I attached new patch, Please review.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 48s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 31s There were no new javac warning messages.
          +1 javadoc 9m 33s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 16s The applied patch generated 4 new checkstyle issues (total was 18, now 22).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          -1 findbugs 3m 19s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 native 3m 15s Pre-build of native portion
          -1 hdfs tests 163m 13s Tests failed in hadoop-hdfs.
              209m 30s  



          Reason Tests
          FindBugs module:hadoop-hdfs
          Failed unit tests hadoop.hdfs.server.namenode.TestStartup



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12738818/HDFS-8540_1.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 6785661
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11301/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/11301/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11301/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11301/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11301/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 48s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 31s There were no new javac warning messages. +1 javadoc 9m 33s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 16s The applied patch generated 4 new checkstyle issues (total was 18, now 22). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 3m 19s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 15s Pre-build of native portion -1 hdfs tests 163m 13s Tests failed in hadoop-hdfs.     209m 30s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.hdfs.server.namenode.TestStartup Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12738818/HDFS-8540_1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6785661 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11301/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/11301/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11301/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11301/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11301/console This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          The patch looks good. Just some minor comments:

          • remove "public" from the methods in Result.
          • add a Result.getExitStatus() method as below.
                /**
                 * @return SUCCESS if all moves are success and there is no remaining move.
                 *         Return NO_MOVE_BLOCK if there moves available but all the moves
                 *         cannot be scheduled. Otherwise, there some success moves and some
                 *         remaining moves, return IN_PROGRESS
                 */
                ExitStatus getExitStatus() {
                  return isNoBlockMoved()? ExitStatus.NO_MOVE_BLOCK
                      : isHasRemaining()? ExitStatus.IN_PROGRESS: ExitStatus.SUCCESS;
                }
            

            Then, run() can return directly

                  return new Processor().processNamespace().getExitStatus();
            
          • there is a unused import org.apache.hadoop.fs.BlockStoragePolicySpi. Could you remove it as well?
          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - The patch looks good. Just some minor comments: remove "public" from the methods in Result. add a Result.getExitStatus() method as below. /** * @ return SUCCESS if all moves are success and there is no remaining move. * Return NO_MOVE_BLOCK if there moves available but all the moves * cannot be scheduled. Otherwise, there some success moves and some * remaining moves, return IN_PROGRESS */ ExitStatus getExitStatus() { return isNoBlockMoved()? ExitStatus.NO_MOVE_BLOCK : isHasRemaining()? ExitStatus.IN_PROGRESS: ExitStatus.SUCCESS; } Then, run() can return directly return new Processor().processNamespace().getExitStatus(); there is a unused import org.apache.hadoop.fs.BlockStoragePolicySpi. Could you remove it as well?
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -
          • The Result class should be static.
          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - The Result class should be static.
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Thanks Tsz Wo Nicholas Sze for review..

          I have one doubt

              ExitStatus getExitStatus(){
                return isNoBlockMoved()? ExitStatus.NO_MOVE_BLOCK
                    : isHasRemaining()? ExitStatus.IN_PROGRESS: ExitStatus.SUCCESS;
              }
          

          This will return "NO_MOVE_BLOCK" when user execute mover but nothing to move and all the file in proper storage policy.

          I think this should be

              ExitStatus getExitStatus() {
                return isHasRemaining() ? isNoBlockMoved()
                    ? ExitStatus.NO_MOVE_BLOCK : ExitStatus.IN_PROGRESS
                    : ExitStatus.SUCCESS;
              }
          

          We should first check isHasRemaining(), if it is false we should return SUCCESS.

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Thanks Tsz Wo Nicholas Sze for review.. I have one doubt ExitStatus getExitStatus(){ return isNoBlockMoved()? ExitStatus.NO_MOVE_BLOCK : isHasRemaining()? ExitStatus.IN_PROGRESS: ExitStatus.SUCCESS; } This will return "NO_MOVE_BLOCK" when user execute mover but nothing to move and all the file in proper storage policy. I think this should be ExitStatus getExitStatus() { return isHasRemaining() ? isNoBlockMoved() ? ExitStatus.NO_MOVE_BLOCK : ExitStatus.IN_PROGRESS : ExitStatus.SUCCESS; } We should first check isHasRemaining() , if it is false we should return SUCCESS.
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          One more think

          Show
          surendrasingh Surendra Singh Lilhore added a comment - One more think
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Sorry mistakenly pressed enter key

          One more thing, this check style we no need to fix I think.

          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java:733:44: 'hasRemaining' hides a field.
          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java:737:41: 'noBlockMoved' hides a field.

          It is very common in constructors and setters that the set field name is the same as the setter parameter name.

          Please give your suggestion

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Sorry mistakenly pressed enter key One more thing, this check style we no need to fix I think. ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java:733:44: 'hasRemaining' hides a field. ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java:737:41: 'noBlockMoved' hides a field. It is very common in constructors and setters that the set field name is the same as the setter parameter name. Please give your suggestion
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          You are right about getExitStatus(). Good point! Let's return SUCCESS first for better readability. The javadoc I posted previously also need to be fixed. See below.

            /**
             * @return SUCCESS if all moves are success and there is no remaining move.
             *         Return NO_MOVE_BLOCK if there are moves available but all the moves
             *         cannot be scheduled. Otherwise, return IN_PROGRESS since there must
             *         be some remaining moves.
             */
            ExitStatus getExitStatus() {
              return !isHasRemaining()? ExitStatus.SUCCESS
                  : isNoBlockMoved()? ExitStatus.NO_MOVE_BLOCK: ExitStatus.IN_PROGRESS;
            }
          

          Let's ignore the check style warnings.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - You are right about getExitStatus(). Good point! Let's return SUCCESS first for better readability. The javadoc I posted previously also need to be fixed. See below. /** * @ return SUCCESS if all moves are success and there is no remaining move. * Return NO_MOVE_BLOCK if there are moves available but all the moves * cannot be scheduled. Otherwise, return IN_PROGRESS since there must * be some remaining moves. */ ExitStatus getExitStatus() { return !isHasRemaining()? ExitStatus.SUCCESS : isNoBlockMoved()? ExitStatus.NO_MOVE_BLOCK: ExitStatus.IN_PROGRESS; } Let's ignore the check style warnings.
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Thanks Tsz Wo Nicholas Sze

          I attached updated patch. Please review..

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Thanks Tsz Wo Nicholas Sze I attached updated patch. Please review..
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 5s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 33s There were no new javac warning messages.
          +1 javadoc 9m 43s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 16s The applied patch generated 3 new checkstyle issues (total was 18, now 20).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 3m 21s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 14s Pre-build of native portion
          -1 hdfs tests 161m 9s Tests failed in hadoop-hdfs.
              207m 53s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12739468/HDFS-8540_2.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / b8341f1
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11342/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11342/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11342/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11342/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 5s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 33s There were no new javac warning messages. +1 javadoc 9m 43s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 16s The applied patch generated 3 new checkstyle issues (total was 18, now 20). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 21s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 14s Pre-build of native portion -1 hdfs tests 161m 9s Tests failed in hadoop-hdfs.     207m 53s   Reason Tests Failed unit tests hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12739468/HDFS-8540_2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / b8341f1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11342/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11342/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11342/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11342/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 7s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 36s There were no new javac warning messages.
          +1 javadoc 9m 43s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 20s The applied patch generated 2 new checkstyle issues (total was 18, now 19).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 38s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 3m 14s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 16s Pre-build of native portion
          -1 hdfs tests 159m 56s Tests failed in hadoop-hdfs.
              206m 50s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.namenode.TestFileTruncate



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12739483/HDFS-8540_3.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / b8341f1
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11346/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11346/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11346/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11346/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 7s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 36s There were no new javac warning messages. +1 javadoc 9m 43s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 20s The applied patch generated 2 new checkstyle issues (total was 18, now 19). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 38s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 14s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 16s Pre-build of native portion -1 hdfs tests 159m 56s Tests failed in hadoop-hdfs.     206m 50s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.TestFileTruncate Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12739483/HDFS-8540_3.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / b8341f1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11346/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11346/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11346/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11346/console This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          +1 the new patch looks good.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - +1 the new patch looks good.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8020 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8020/)
          HDFS-8540. Mover should exit with NO_MOVE_BLOCK if no block can be moved. Contributed by surendra singh lilhore (szetszwo: rev 321940cf19375febe9660e96d905360cfcc15f5f)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestStorageMover.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8020 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8020/ ) HDFS-8540 . Mover should exit with NO_MOVE_BLOCK if no block can be moved. Contributed by surendra singh lilhore (szetszwo: rev 321940cf19375febe9660e96d905360cfcc15f5f) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestStorageMover.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, surendra!

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, surendra!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8021 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8021/)
          Move HDFS-8540 to 2.8 in CHANGES.txt. (szetszwo: rev 1b6695a4c0d76fe18d6524cc1379bc1185708c6f)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8021 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8021/ ) Move HDFS-8540 to 2.8 in CHANGES.txt. (szetszwo: rev 1b6695a4c0d76fe18d6524cc1379bc1185708c6f) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #960 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/960/)
          HDFS-8540. Mover should exit with NO_MOVE_BLOCK if no block can be moved. Contributed by surendra singh lilhore (szetszwo: rev 321940cf19375febe9660e96d905360cfcc15f5f)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestStorageMover.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            Move HDFS-8540 to 2.8 in CHANGES.txt. (szetszwo: rev 1b6695a4c0d76fe18d6524cc1379bc1185708c6f)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #960 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/960/ ) HDFS-8540 . Mover should exit with NO_MOVE_BLOCK if no block can be moved. Contributed by surendra singh lilhore (szetszwo: rev 321940cf19375febe9660e96d905360cfcc15f5f) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestStorageMover.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Move HDFS-8540 to 2.8 in CHANGES.txt. (szetszwo: rev 1b6695a4c0d76fe18d6524cc1379bc1185708c6f) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #230 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/230/)
          HDFS-8540. Mover should exit with NO_MOVE_BLOCK if no block can be moved. Contributed by surendra singh lilhore (szetszwo: rev 321940cf19375febe9660e96d905360cfcc15f5f)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestStorageMover.java
            Move HDFS-8540 to 2.8 in CHANGES.txt. (szetszwo: rev 1b6695a4c0d76fe18d6524cc1379bc1185708c6f)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #230 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/230/ ) HDFS-8540 . Mover should exit with NO_MOVE_BLOCK if no block can be moved. Contributed by surendra singh lilhore (szetszwo: rev 321940cf19375febe9660e96d905360cfcc15f5f) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestStorageMover.java Move HDFS-8540 to 2.8 in CHANGES.txt. (szetszwo: rev 1b6695a4c0d76fe18d6524cc1379bc1185708c6f) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2158 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2158/)
          HDFS-8540. Mover should exit with NO_MOVE_BLOCK if no block can be moved. Contributed by surendra singh lilhore (szetszwo: rev 321940cf19375febe9660e96d905360cfcc15f5f)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestStorageMover.java
            Move HDFS-8540 to 2.8 in CHANGES.txt. (szetszwo: rev 1b6695a4c0d76fe18d6524cc1379bc1185708c6f)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2158 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2158/ ) HDFS-8540 . Mover should exit with NO_MOVE_BLOCK if no block can be moved. Contributed by surendra singh lilhore (szetszwo: rev 321940cf19375febe9660e96d905360cfcc15f5f) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestStorageMover.java Move HDFS-8540 to 2.8 in CHANGES.txt. (szetszwo: rev 1b6695a4c0d76fe18d6524cc1379bc1185708c6f) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #219 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/219/)
          HDFS-8540. Mover should exit with NO_MOVE_BLOCK if no block can be moved. Contributed by surendra singh lilhore (szetszwo: rev 321940cf19375febe9660e96d905360cfcc15f5f)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestStorageMover.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
            Move HDFS-8540 to 2.8 in CHANGES.txt. (szetszwo: rev 1b6695a4c0d76fe18d6524cc1379bc1185708c6f)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #219 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/219/ ) HDFS-8540 . Mover should exit with NO_MOVE_BLOCK if no block can be moved. Contributed by surendra singh lilhore (szetszwo: rev 321940cf19375febe9660e96d905360cfcc15f5f) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestStorageMover.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java Move HDFS-8540 to 2.8 in CHANGES.txt. (szetszwo: rev 1b6695a4c0d76fe18d6524cc1379bc1185708c6f) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #228 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/228/)
          HDFS-8540. Mover should exit with NO_MOVE_BLOCK if no block can be moved. Contributed by surendra singh lilhore (szetszwo: rev 321940cf19375febe9660e96d905360cfcc15f5f)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestStorageMover.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
            Move HDFS-8540 to 2.8 in CHANGES.txt. (szetszwo: rev 1b6695a4c0d76fe18d6524cc1379bc1185708c6f)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #228 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/228/ ) HDFS-8540 . Mover should exit with NO_MOVE_BLOCK if no block can be moved. Contributed by surendra singh lilhore (szetszwo: rev 321940cf19375febe9660e96d905360cfcc15f5f) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestStorageMover.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java Move HDFS-8540 to 2.8 in CHANGES.txt. (szetszwo: rev 1b6695a4c0d76fe18d6524cc1379bc1185708c6f) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2176 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2176/)
          HDFS-8540. Mover should exit with NO_MOVE_BLOCK if no block can be moved. Contributed by surendra singh lilhore (szetszwo: rev 321940cf19375febe9660e96d905360cfcc15f5f)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestStorageMover.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java
            Move HDFS-8540 to 2.8 in CHANGES.txt. (szetszwo: rev 1b6695a4c0d76fe18d6524cc1379bc1185708c6f)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2176 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2176/ ) HDFS-8540 . Mover should exit with NO_MOVE_BLOCK if no block can be moved. Contributed by surendra singh lilhore (szetszwo: rev 321940cf19375febe9660e96d905360cfcc15f5f) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestStorageMover.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java Move HDFS-8540 to 2.8 in CHANGES.txt. (szetszwo: rev 1b6695a4c0d76fe18d6524cc1379bc1185708c6f) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

            People

            • Assignee:
              surendrasingh Surendra Singh Lilhore
              Reporter:
              szetszwo Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development