Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3047

FileOutputCommitter throws wrong type of exception when calling abortTask() to handle a directory without permission

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      When FileOutputCommitter calls abortTask() to create a temp directory, if the user has no permission to access the directory, or a file with the same name has existed, of course it will fail, however the system will output the error information into the log file instead of throwing an exception.As a result, when the temp directory is needed later, since the temp directory hasn't been created yet, system will throw an exception to tell user that the temp directory doesn't exist.In my opinion, the exception is not exact and the error infomation will confuse users.

      1. MAPREDUCE-3047.patch
        2 kB
        JiangKai
      2. MAPREDUCE-3047-003.patch
        1 kB
        Brahma Reddy Battula
      3. MAPREDUCE-3047-004.patch
        1 kB
        Brahma Reddy Battula
      4. MAPREDUCE-3047-1.patch
        0.9 kB
        madhukara phatak
      5. MAPREDUCE-3047-2.patch
        0.9 kB
        madhukara phatak

        Activity

        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 17m 54s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 8m 12s There were no new javac warning messages.
        +1 javadoc 9m 47s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 50s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 1m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 mapreduce tests 1m 43s Tests passed in hadoop-mapreduce-client-core.
            42m 24s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12742492/MAPREDUCE-3047-004.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / c815344
        hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5850/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5850/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5850/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 54s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 8m 12s There were no new javac warning messages. +1 javadoc 9m 47s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 50s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 1m 43s Tests passed in hadoop-mapreduce-client-core.     42m 24s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12742492/MAPREDUCE-3047-004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c815344 hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5850/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5850/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5850/console This message was automatically generated.
        Hide
        Devaraj K added a comment -

        Thanks Brahma Reddy Battula for rebasing the patch.

        Can you also add a test case for the change as part of the patch?

        Show
        Devaraj K added a comment - Thanks Brahma Reddy Battula for rebasing the patch. Can you also add a test case for the change as part of the patch?
        Hide
        Brahma Reddy Battula added a comment -

        Addressing the checkstyle issue..

        Show
        Brahma Reddy Battula added a comment - Addressing the checkstyle issue..
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 1s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 33s There were no new javac warning messages.
        +1 javadoc 9m 34s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 0m 46s The applied patch generated 1 new checkstyle issues (total was 22, now 23).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 37s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 1m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 mapreduce tests 1m 43s Tests passed in hadoop-mapreduce-client-core.
            39m 39s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12742450/MAPREDUCE-3047-003.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / a95d39f
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5849/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
        hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5849/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5849/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5849/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 1s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 33s There were no new javac warning messages. +1 javadoc 9m 34s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 46s The applied patch generated 1 new checkstyle issues (total was 22, now 23). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 1m 43s Tests passed in hadoop-mapreduce-client-core.     39m 39s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12742450/MAPREDUCE-3047-003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a95d39f checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5849/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5849/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5849/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5849/console This message was automatically generated.
        Hide
        Brahma Reddy Battula added a comment -

        Devaraj K Rebased patch kindly review!!

        Show
        Brahma Reddy Battula added a comment - Devaraj K Rebased patch kindly review!!
        Hide
        Devaraj K added a comment -

        Patch doesn't apply anymore, Cancelling patch to rebase it.

        Show
        Devaraj K added a comment - Patch doesn't apply anymore, Cancelling patch to rebase it.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12530013/MAPREDUCE-3047-2.patch
        against trunk revision 3241fc2.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5246//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12530013/MAPREDUCE-3047-2.patch against trunk revision 3241fc2. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5246//console This message is automatically generated.
        Hide
        madhukara phatak added a comment -

        updated the patch with space and granted the licence for inclusion

        Show
        madhukara phatak added a comment - updated the patch with space and granted the licence for inclusion
        Hide
        Robert Joseph Evans added a comment -

        I like the patch, I just have two comments before I can give it a +1.

        1. I need you to make sure you mark the contribute to apache link when submitting the patch. Without it I cannot take the patch.
        2. Our codeing convention calls for spaces instead of tabs.
        Show
        Robert Joseph Evans added a comment - I like the patch, I just have two comments before I can give it a +1. I need you to make sure you mark the contribute to apache link when submitting the patch. Without it I cannot take the patch. Our codeing convention calls for spaces instead of tabs.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12522132/MAPREDUCE-3047-1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 javac. The applied patch generated 508 javac compiler warnings (more than the trunk's current 507 warnings).

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
        org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
        org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
        org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2186//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2186//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522132/MAPREDUCE-3047-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 508 javac compiler warnings (more than the trunk's current 507 warnings). +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2186//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2186//console This message is automatically generated.
        Hide
        madhukara phatak added a comment -

        Updated patch according to trunk

        Show
        madhukara phatak added a comment - Updated patch according to trunk
        Hide
        Robert Joseph Evans added a comment -

        This patch is out of date. the FileOutputCommitter has had major changes to it, and this patch needs to at least be upmerged for trunk

        Show
        Robert Joseph Evans added a comment - This patch is out of date. the FileOutputCommitter has had major changes to it, and this patch needs to at least be upmerged for trunk
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12495197/MAPREDUCE-3047.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/797//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/797//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12495197/MAPREDUCE-3047.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/797//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/797//console This message is automatically generated.

          People

          • Assignee:
            Brahma Reddy Battula
            Reporter:
            JiangKai
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:

              Development