Hadoop Common
  1. Hadoop Common
  2. HADOOP-7487

DF should throw a more reasonable exception when mount cannot be determined

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0, 2.0.3-alpha, 3.0.0
    • Fix Version/s: 2.1.0-beta
    • Component/s: fs
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Currently, when using the DF class to determine the mount corresponding to a given directory, it will throw the generic exception "Expecting a line not the end of stream" if it can't determine the mount (for example if the directory doesn't exist).

      This error message should be improved in several ways:

      1. If the dir to check doesn't exist, we can see that before even execing df, and throw a better exception (or behave better by chopping path components until it exists)
      2. Rather than parsing the lines out of df's stdout, collect the whole output, and then parse. So, if df returns a non-zero exit code, we can avoid trying to parse the empty result
      3. If there's a success exit code, and we still can't parse it (eg incompatible OS), we should include the unparseable line in the exception message.
      1. hadoop-7487-3.patch
        9 kB
        Andrew Wang
      2. hadoop-7487-2.patch
        9 kB
        Andrew Wang
      3. hadoop-7487-1.patch
        9 kB
        Andrew Wang

        Activity

        Todd Lipcon created issue -
        Andrew Wang made changes -
        Field Original Value New Value
        Assignee Andrew Wang [ andrew.wang ]
        Hide
        Andrew Wang added a comment -

        Patch with Todd's suggested changes. We now defer parsing, and catch additional runtime errors during parsing. Added a test case for the newly checked errors.

        Tested via running TestDFVariations,TestNamenodeCapacityReport.

        Show
        Andrew Wang added a comment - Patch with Todd's suggested changes. We now defer parsing, and catch additional runtime errors during parsing. Added a test case for the newly checked errors. Tested via running TestDFVariations,TestNamenodeCapacityReport .
        Andrew Wang made changes -
        Attachment hadoop-7487-1.patch [ 12567487 ]
        Andrew Wang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12567487/hadoop-7487-1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2129//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2129//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12567487/hadoop-7487-1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2129//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2129//console This message is automatically generated.
        Hide
        Aaron T. Myers added a comment -

        Patch looks pretty good to me. One suggestion: In the places in the test where you catch expected exceptions, use GenericTestUtils#assertExceptionContains to ensure that you're catching the exception you expect.

        Show
        Aaron T. Myers added a comment - Patch looks pretty good to me. One suggestion: In the places in the test where you catch expected exceptions, use GenericTestUtils#assertExceptionContains to ensure that you're catching the exception you expect.
        Hide
        Aaron T. Myers added a comment -

        I should've said: +1 once this is addressed.

        Show
        Aaron T. Myers added a comment - I should've said: +1 once this is addressed.
        Aaron T. Myers made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Andrew Wang added a comment -

        Thanks for the review Aaron!

        I added the checks on the expected exceptions, and reran TestDFVariation and TestNamenodeCapacityReport successfully.

        Show
        Andrew Wang added a comment - Thanks for the review Aaron! I added the checks on the expected exceptions, and reran TestDFVariation and TestNamenodeCapacityReport successfully.
        Andrew Wang made changes -
        Attachment hadoop-7487-2.patch [ 12570635 ]
        Andrew Wang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Affects Version/s 2.0.3-alpha [ 12323273 ]
        Affects Version/s 3.0.0 [ 12320357 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12570635/hadoop-7487-2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        -1 one of tests included doesn't have a timeout.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2223//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2223//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12570635/hadoop-7487-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. -1 one of tests included doesn't have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2223//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2223//console This message is automatically generated.
        Hide
        Andrew Wang added a comment -

        Forgot about the test timeouts, added them to all the tests in TestDFVariations.

        Show
        Andrew Wang added a comment - Forgot about the test timeouts, added them to all the tests in TestDFVariations .
        Andrew Wang made changes -
        Attachment hadoop-7487-3.patch [ 12570681 ]
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12570681/hadoop-7487-3.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 tests included appear to have a timeout.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2224//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2224//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12570681/hadoop-7487-3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2224//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2224//console This message is automatically generated.
        Hide
        Aaron T. Myers added a comment -

        +1, the latest patch looks good to me. I'm going to commit this momentarily.

        Show
        Aaron T. Myers added a comment - +1, the latest patch looks good to me. I'm going to commit this momentarily.
        Hide
        Aaron T. Myers added a comment -

        I've just committed this to trunk and branch-2.

        Thanks a lot for the contribution, Andrew.

        Show
        Aaron T. Myers added a comment - I've just committed this to trunk and branch-2. Thanks a lot for the contribution, Andrew.
        Aaron T. Myers made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Target Version/s 2.0.4-beta [ 12324030 ]
        Fix Version/s 2.0.4-beta [ 12324030 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #3383 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3383/)
        HADOOP-7487. DF should throw a more reasonable exception when mount cannot be determined. Contributed by Andrew Wang. (Revision 1449992)

        Result = SUCCESS
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1449992
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DF.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDFVariations.java
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #3383 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3383/ ) HADOOP-7487 . DF should throw a more reasonable exception when mount cannot be determined. Contributed by Andrew Wang. (Revision 1449992) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1449992 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DF.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDFVariations.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #139 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/139/)
        HADOOP-7487. DF should throw a more reasonable exception when mount cannot be determined. Contributed by Andrew Wang. (Revision 1449992)

        Result = SUCCESS
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1449992
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DF.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDFVariations.java
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #139 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/139/ ) HADOOP-7487 . DF should throw a more reasonable exception when mount cannot be determined. Contributed by Andrew Wang. (Revision 1449992) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1449992 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DF.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDFVariations.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1328 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1328/)
        HADOOP-7487. DF should throw a more reasonable exception when mount cannot be determined. Contributed by Andrew Wang. (Revision 1449992)

        Result = FAILURE
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1449992
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DF.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDFVariations.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1328 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1328/ ) HADOOP-7487 . DF should throw a more reasonable exception when mount cannot be determined. Contributed by Andrew Wang. (Revision 1449992) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1449992 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DF.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDFVariations.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1356 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1356/)
        HADOOP-7487. DF should throw a more reasonable exception when mount cannot be determined. Contributed by Andrew Wang. (Revision 1449992)

        Result = SUCCESS
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1449992
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DF.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDFVariations.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1356 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1356/ ) HADOOP-7487 . DF should throw a more reasonable exception when mount cannot be determined. Contributed by Andrew Wang. (Revision 1449992) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1449992 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DF.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDFVariations.java
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Patch Available Patch Available Open Open
        21d 19h 42m 1 Aaron T. Myers 22/Feb/13 21:29
        Open Open Patch Available Patch Available
        555d 5h 20m 2 Andrew Wang 24/Feb/13 02:39
        Patch Available Patch Available Resolved Resolved
        1d 23h 4m 1 Aaron T. Myers 26/Feb/13 01:44
        Resolved Resolved Closed Closed
        182d 20h 22m 1 Arun C Murthy 27/Aug/13 23:06

          People

          • Assignee:
            Andrew Wang
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development