Hadoop Common
  1. Hadoop Common
  2. HADOOP-7487

DF should throw a more reasonable exception when mount cannot be determined

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0, 3.0.0, 2.0.3-alpha
    • Fix Version/s: 2.1.0-beta
    • Component/s: fs
    • Labels:
    • Hadoop Flags:
      Reviewed
    • Target Version/s:

      Description

      Currently, when using the DF class to determine the mount corresponding to a given directory, it will throw the generic exception "Expecting a line not the end of stream" if it can't determine the mount (for example if the directory doesn't exist).

      This error message should be improved in several ways:

      1. If the dir to check doesn't exist, we can see that before even execing df, and throw a better exception (or behave better by chopping path components until it exists)
      2. Rather than parsing the lines out of df's stdout, collect the whole output, and then parse. So, if df returns a non-zero exit code, we can avoid trying to parse the empty result
      3. If there's a success exit code, and we still can't parse it (eg incompatible OS), we should include the unparseable line in the exception message.
      1. hadoop-7487-1.patch
        9 kB
        Andrew Wang
      2. hadoop-7487-2.patch
        9 kB
        Andrew Wang
      3. hadoop-7487-3.patch
        9 kB
        Andrew Wang

        Activity

        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1356 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1356/)
        HADOOP-7487. DF should throw a more reasonable exception when mount cannot be determined. Contributed by Andrew Wang. (Revision 1449992)

        Result = SUCCESS
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1449992
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DF.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDFVariations.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1356 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1356/ ) HADOOP-7487 . DF should throw a more reasonable exception when mount cannot be determined. Contributed by Andrew Wang. (Revision 1449992) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1449992 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DF.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDFVariations.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1328 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1328/)
        HADOOP-7487. DF should throw a more reasonable exception when mount cannot be determined. Contributed by Andrew Wang. (Revision 1449992)

        Result = FAILURE
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1449992
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DF.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDFVariations.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1328 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1328/ ) HADOOP-7487 . DF should throw a more reasonable exception when mount cannot be determined. Contributed by Andrew Wang. (Revision 1449992) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1449992 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DF.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDFVariations.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #139 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/139/)
        HADOOP-7487. DF should throw a more reasonable exception when mount cannot be determined. Contributed by Andrew Wang. (Revision 1449992)

        Result = SUCCESS
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1449992
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DF.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDFVariations.java
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #139 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/139/ ) HADOOP-7487 . DF should throw a more reasonable exception when mount cannot be determined. Contributed by Andrew Wang. (Revision 1449992) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1449992 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DF.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDFVariations.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #3383 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3383/)
        HADOOP-7487. DF should throw a more reasonable exception when mount cannot be determined. Contributed by Andrew Wang. (Revision 1449992)

        Result = SUCCESS
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1449992
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DF.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDFVariations.java
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #3383 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3383/ ) HADOOP-7487 . DF should throw a more reasonable exception when mount cannot be determined. Contributed by Andrew Wang. (Revision 1449992) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1449992 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DF.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDFVariations.java
        Aaron T. Myers made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Target Version/s 2.0.4-beta [ 12324030 ]
        Fix Version/s 2.0.4-beta [ 12324030 ]
        Resolution Fixed [ 1 ]
        Hide
        Aaron T. Myers added a comment -

        I've just committed this to trunk and branch-2.

        Thanks a lot for the contribution, Andrew.

        Show
        Aaron T. Myers added a comment - I've just committed this to trunk and branch-2. Thanks a lot for the contribution, Andrew.
        Hide
        Aaron T. Myers added a comment -

        +1, the latest patch looks good to me. I'm going to commit this momentarily.

        Show
        Aaron T. Myers added a comment - +1, the latest patch looks good to me. I'm going to commit this momentarily.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12570681/hadoop-7487-3.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 tests included appear to have a timeout.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2224//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2224//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12570681/hadoop-7487-3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2224//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2224//console This message is automatically generated.
        Andrew Wang made changes -
        Attachment hadoop-7487-3.patch [ 12570681 ]
        Hide
        Andrew Wang added a comment -

        Forgot about the test timeouts, added them to all the tests in TestDFVariations.

        Show
        Andrew Wang added a comment - Forgot about the test timeouts, added them to all the tests in TestDFVariations .
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12570635/hadoop-7487-2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        -1 one of tests included doesn't have a timeout.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2223//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2223//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12570635/hadoop-7487-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. -1 one of tests included doesn't have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2223//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2223//console This message is automatically generated.
        Andrew Wang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Affects Version/s 2.0.3-alpha [ 12323273 ]
        Affects Version/s 3.0.0 [ 12320357 ]
        Andrew Wang made changes -
        Attachment hadoop-7487-2.patch [ 12570635 ]
        Hide
        Andrew Wang added a comment -

        Thanks for the review Aaron!

        I added the checks on the expected exceptions, and reran TestDFVariation and TestNamenodeCapacityReport successfully.

        Show
        Andrew Wang added a comment - Thanks for the review Aaron! I added the checks on the expected exceptions, and reran TestDFVariation and TestNamenodeCapacityReport successfully.
        Aaron T. Myers made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Aaron T. Myers added a comment -

        I should've said: +1 once this is addressed.

        Show
        Aaron T. Myers added a comment - I should've said: +1 once this is addressed.
        Hide
        Aaron T. Myers added a comment -

        Patch looks pretty good to me. One suggestion: In the places in the test where you catch expected exceptions, use GenericTestUtils#assertExceptionContains to ensure that you're catching the exception you expect.

        Show
        Aaron T. Myers added a comment - Patch looks pretty good to me. One suggestion: In the places in the test where you catch expected exceptions, use GenericTestUtils#assertExceptionContains to ensure that you're catching the exception you expect.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12567487/hadoop-7487-1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2129//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2129//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12567487/hadoop-7487-1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2129//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2129//console This message is automatically generated.
        Andrew Wang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Andrew Wang made changes -
        Attachment hadoop-7487-1.patch [ 12567487 ]
        Hide
        Andrew Wang added a comment -

        Patch with Todd's suggested changes. We now defer parsing, and catch additional runtime errors during parsing. Added a test case for the newly checked errors.

        Tested via running TestDFVariations,TestNamenodeCapacityReport.

        Show
        Andrew Wang added a comment - Patch with Todd's suggested changes. We now defer parsing, and catch additional runtime errors during parsing. Added a test case for the newly checked errors. Tested via running TestDFVariations,TestNamenodeCapacityReport .
        Andrew Wang made changes -
        Field Original Value New Value
        Assignee Andrew Wang [ andrew.wang ]
        Todd Lipcon created issue -

          People

          • Assignee:
            Andrew Wang
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development