Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-328

"fs -setrep" should have better error message

    Details

    • Type: Improvement Improvement
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: namenode
    • Labels:

      Description

      When the replication # is larger than dfs.replication.max (defined in conf), "fs -setrep" shows a meaningless error message. For example,

      //dfs.replication.max is 512
      
      $ hadoop fs -setrep 1000 r.txt
      setrep: java.io.IOException: file /user/tsz/r.txt.
      
      1. HDFS-328.001.patch
        7 kB
        Daniel Templeton

        Activity

        Hide
        Nigel Daley added a comment -

        The fix for this should include some new test cases added to src/test/org/apache/hadoop/cli/testConf.xml (which is run by TestCLI)

        Show
        Nigel Daley added a comment - The fix for this should include some new test cases added to src/test/org/apache/hadoop/cli/testConf.xml (which is run by TestCLI)
        Hide
        Allen Wittenauer added a comment -

        Just verified that in 3.x trunk this error message is still broken:

        $hdfs dfs -setrep 10000 /hosts
        setrep: file /hosts.
        
        Show
        Allen Wittenauer added a comment - Just verified that in 3.x trunk this error message is still broken: $hdfs dfs -setrep 10000 /hosts setrep: file /hosts.
        Hide
        Daniel Templeton added a comment -

        No recent activity, reassigning.

        Show
        Daniel Templeton added a comment - No recent activity, reassigning.
        Hide
        Daniel Templeton added a comment -

        Adding patch that includes a change to the BlockManager.verifyReplication() method as well as adding new -setrep tests.

        Show
        Daniel Templeton added a comment - Adding patch that includes a change to the BlockManager.verifyReplication() method as well as adding new -setrep tests.
        Hide
        Daniel Templeton added a comment -

        Oops. Missed some whitespace. Reposting patch.

        Show
        Daniel Templeton added a comment - Oops. Missed some whitespace. Reposting patch.
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 17m 58s Pre-patch trunk has 2 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 7m 45s There were no new javac warning messages.
        +1 javadoc 10m 19s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 22s The applied patch generated 1 new checkstyle issues (total was 195, now 190).
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 28s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 2m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 8s Pre-build of native portion
        -1 hdfs tests 160m 32s Tests failed in hadoop-hdfs.
            206m 4s  



        Reason Tests
        Failed unit tests hadoop.hdfs.tools.TestDFSAdminWithHA
          hadoop.hdfs.server.namenode.TestFSNamesystem



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12752855/HDFS-328.001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / a9c8ea7
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12173/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12173/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12173/artifact/patchprocess/whitespace.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12173/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12173/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12173/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 58s Pre-patch trunk has 2 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 45s There were no new javac warning messages. +1 javadoc 10m 19s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 22s The applied patch generated 1 new checkstyle issues (total was 195, now 190). -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 8s Pre-build of native portion -1 hdfs tests 160m 32s Tests failed in hadoop-hdfs.     206m 4s   Reason Tests Failed unit tests hadoop.hdfs.tools.TestDFSAdminWithHA   hadoop.hdfs.server.namenode.TestFSNamesystem Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752855/HDFS-328.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a9c8ea7 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12173/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12173/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12173/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12173/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12173/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12173/console This message was automatically generated.
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 17m 57s Pre-patch trunk has 2 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 7m 55s There were no new javac warning messages.
        +1 javadoc 10m 7s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 22s The applied patch generated 1 new checkstyle issues (total was 195, now 190).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 30s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 2m 33s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 17s Pre-build of native portion
        -1 hdfs tests 160m 37s Tests failed in hadoop-hdfs.
            206m 15s  



        Reason Tests
        Failed unit tests hadoop.hdfs.server.namenode.TestCacheDirectives
          hadoop.hdfs.TestLeaseRecovery2



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12752869/HDFS-328.001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / a9c8ea7
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12176/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12176/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12176/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12176/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12176/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 57s Pre-patch trunk has 2 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 55s There were no new javac warning messages. +1 javadoc 10m 7s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 22s The applied patch generated 1 new checkstyle issues (total was 195, now 190). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 30s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 2m 33s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 17s Pre-build of native portion -1 hdfs tests 160m 37s Tests failed in hadoop-hdfs.     206m 15s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.TestCacheDirectives   hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752869/HDFS-328.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a9c8ea7 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12176/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12176/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12176/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12176/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12176/console This message was automatically generated.
        Hide
        Daniel Templeton added a comment -

        The failed tests are unrelated to this patch. The checkstyle warning is about the length of the file exceeding 2k lines, which was true before the patch.

        This one is ready for review.

        Show
        Daniel Templeton added a comment - The failed tests are unrelated to this patch. The checkstyle warning is about the length of the file exceeding 2k lines, which was true before the patch. This one is ready for review.

          People

          • Assignee:
            Daniel Templeton
            Reporter:
            Tsz Wo Nicholas Sze
          • Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:

              Development