Hadoop Common
  1. Hadoop Common
  2. HADOOP-10380 Cleanup javac warnings
  3. HADOOP-10392

Use FileSystem#makeQualified(Path) instead of Path#makeQualified(FileSystem)

    Details

    • Type: Sub-task Sub-task
    • Status: Patch Available
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.3.0
    • Fix Version/s: None
    • Component/s: fs
    • Labels:
    • Target Version/s:

      Description

      There're some methods calling Path.makeQualified(FileSystem), which causes javac warning.

      1. HADOOP-10392.2.patch
        38 kB
        Akira AJISAKA
      2. HADOOP-10392.3.patch
        39 kB
        Akira AJISAKA
      3. HADOOP-10392.4.patch
        37 kB
        Akira AJISAKA
      4. HADOOP-10392.4.patch
        37 kB
        Akira AJISAKA
      5. HADOOP-10392.5.patch
        38 kB
        Akira AJISAKA
      6. HADOOP-10392.6.patch
        39 kB
        Akira AJISAKA
      7. HADOOP-10392.7.patch
        37 kB
        Akira AJISAKA
      8. HADOOP-10392.7.patch
        37 kB
        Akira AJISAKA
      9. HADOOP-10392.8.patch
        36 kB
        Akira AJISAKA
      10. HADOOP-10392.patch
        39 kB
        Akira AJISAKA

        Activity

        Hide
        Akira AJISAKA added a comment -

        Attaching a patch.

        Show
        Akira AJISAKA added a comment - Attaching a patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12633516/HADOOP-10392.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 24 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        -1 release audit. The applied patch generated 1 release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archives hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-openstack hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3647//testReport/
        Release audit warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3647//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3647//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12633516/HADOOP-10392.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 24 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. -1 release audit . The applied patch generated 1 release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archives hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-openstack hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3647//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3647//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3647//console This message is automatically generated.
        Hide
        Akira AJISAKA added a comment -

        The audit warnings is caused by HDFS-3405. GetImageServlet.java became empty file by the commit.

        Show
        Akira AJISAKA added a comment - The audit warnings is caused by HDFS-3405 . GetImageServlet.java became empty file by the commit.
        Hide
        Daryn Sharp added a comment -

        At least to me, Path#makeQualified(URI fsUri, String workingDir) seems rather cumbersome. I'd suggest using FileSystem#makeQualified(Path).

        Show
        Daryn Sharp added a comment - At least to me, Path#makeQualified(URI fsUri, String workingDir) seems rather cumbersome. I'd suggest using FileSystem#makeQualified(Path) .
        Hide
        Akira AJISAKA added a comment -

        Thanks, I'll update the patch shortly.

        Show
        Akira AJISAKA added a comment - Thanks, I'll update the patch shortly.
        Hide
        Akira AJISAKA added a comment -

        Attaching a patch.

        Show
        Akira AJISAKA added a comment - Attaching a patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12633772/HADOOP-10392.2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 24 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archives hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-openstack hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming:

        org.apache.hadoop.mapred.gridmix.TestPseudoLocalFs

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3654//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3654//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12633772/HADOOP-10392.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 24 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archives hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-openstack hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming: org.apache.hadoop.mapred.gridmix.TestPseudoLocalFs +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3654//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3654//console This message is automatically generated.
        Hide
        Akira AJISAKA added a comment -

        Attaching a patch to skip FileSystem#checkPath() in PseudoLocalFs.

        Show
        Akira AJISAKA added a comment - Attaching a patch to skip FileSystem#checkPath() in PseudoLocalFs.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12633823/HADOOP-10392.3.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 24 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archives hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-openstack hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming:

        org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3656//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3656//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12633823/HADOOP-10392.3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 24 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archives hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-openstack hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming: org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3656//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3656//console This message is automatically generated.
        Hide
        Akira AJISAKA added a comment -

        The test timeout looks unrelated to the patch because the v2 patch passed the test.

        Show
        Akira AJISAKA added a comment - The test timeout looks unrelated to the patch because the v2 patch passed the test.
        Hide
        Akira AJISAKA added a comment -

        Filed MAPREDUCE-5793 to track the timeout.

        Show
        Akira AJISAKA added a comment - Filed MAPREDUCE-5793 to track the timeout.
        Hide
        Daryn Sharp added a comment -

        Although the change seems innocuous, I'd suggest running the yarn/MR test suites to make sure nothing breaks. Otherwise it looks simple and straightforward.

        Show
        Daryn Sharp added a comment - Although the change seems innocuous, I'd suggest running the yarn/MR test suites to make sure nothing breaks. Otherwise it looks simple and straightforward.
        Hide
        Akira AJISAKA added a comment -

        Thanks. I'll run the tests locally.

        Show
        Akira AJISAKA added a comment - Thanks. I'll run the tests locally.
        Hide
        Akira AJISAKA added a comment -

        I ran all the test in yarn and mapreduce project locally and all of them were successful.

        Show
        Akira AJISAKA added a comment - I ran all the test in yarn and mapreduce project locally and all of them were successful.
        Hide
        Daryn Sharp added a comment -

        +1 Good work.

        Show
        Daryn Sharp added a comment - +1 Good work.
        Hide
        Akira AJISAKA added a comment -

        Thanks for the review, Daryn Sharp.

        Show
        Akira AJISAKA added a comment - Thanks for the review, Daryn Sharp .
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12633823/HADOOP-10392.3.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 24 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archives hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-openstack hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming:

        org.apache.hadoop.streaming.TestStreamingTaskLog

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3706//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3706//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12633823/HADOOP-10392.3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 24 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archives hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-openstack hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming: org.apache.hadoop.streaming.TestStreamingTaskLog +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3706//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3706//console This message is automatically generated.
        Hide
        Akira AJISAKA added a comment -

        The failure is not related to the patch. The test fails in trunk also (HADOOP-10424).

        Show
        Akira AJISAKA added a comment - The failure is not related to the patch. The test fails in trunk also ( HADOOP-10424 ).
        Hide
        Akira AJISAKA added a comment -

        Rebased the patch for the latest trunk.

        Show
        Akira AJISAKA added a comment - Rebased the patch for the latest trunk.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12637082/HADOOP-10392.4.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 24 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archives hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-openstack hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming:

        org.apache.hadoop.streaming.TestStreamingTaskLog

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3723//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3723//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12637082/HADOOP-10392.4.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 24 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archives hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-openstack hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming: org.apache.hadoop.streaming.TestStreamingTaskLog +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3723//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3723//console This message is automatically generated.
        Hide
        Akira AJISAKA added a comment -

        The test failures was reported by MAPREDUCE-5810. Attaching the same patch to re-run Jenkins.

        Show
        Akira AJISAKA added a comment - The test failures was reported by MAPREDUCE-5810 . Attaching the same patch to re-run Jenkins.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12644781/HADOOP-10392.4.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 24 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The following test timeouts occurred in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archives hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-openstack hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming:

        org.apache.hadoop.mapred.pipes.TestPipeApplication

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4204//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4204//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12644781/HADOOP-10392.4.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 24 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The following test timeouts occurred in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archives hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-openstack hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming: org.apache.hadoop.mapred.pipes.TestPipeApplication +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4204//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4204//console This message is automatically generated.
        Hide
        Benjamin Zhitomirsky added a comment -

        Please pay attention that using FileSystem.makeQualified(path) is a bug in some cases. Consider the following:
        In conf: fs.defaultName="hdfs://nn1:8020"
        FileSystem.get(conf).makeQualified("hdfs://nn2:8021/dir/subdir/file");

        This will fail with the exception:
        org.apache.hadoop.fs.InvalidPathException: Invalid path name Wrong FS: hdfs://nn2:8021/dir/subdir/file, expected: hdfs://nn1:8020

        Please consider this in your fix.
        The following call does it right:
        FileContext.getFileContext(conf).makeQualified(path);

        Show
        Benjamin Zhitomirsky added a comment - Please pay attention that using FileSystem.makeQualified(path) is a bug in some cases. Consider the following: In conf: fs.defaultName="hdfs://nn1:8020" FileSystem.get(conf).makeQualified("hdfs://nn2:8021/dir/subdir/file"); This will fail with the exception: org.apache.hadoop.fs.InvalidPathException: Invalid path name Wrong FS: hdfs://nn2:8021/dir/subdir/file, expected: hdfs://nn1:8020 Please consider this in your fix. The following call does it right: FileContext.getFileContext(conf).makeQualified(path);
        Hide
        Benjamin Zhitomirsky added a comment -

        Example of possible issue in your patch:
        .../util/MRAsyncDiskService.java:

        • return (new Path(path)).makeQualified(this.localFileSystem)
          + return this.localFileSystem.makeQualified(new Path(path))
          .toUri().getPath();

        The original would accept path which is already fully qualified and references another file system. However your fix will cause the exception that I mentioned above.

        On the bright side all the fixes that you made in file systems and FileUtil seems right.

        Show
        Benjamin Zhitomirsky added a comment - Example of possible issue in your patch: .../util/MRAsyncDiskService.java: return (new Path(path)).makeQualified(this.localFileSystem) + return this.localFileSystem.makeQualified(new Path(path)) .toUri().getPath(); The original would accept path which is already fully qualified and references another file system. However your fix will cause the exception that I mentioned above. On the bright side all the fixes that you made in file systems and FileUtil seems right.
        Hide
        Akira AJISAKA added a comment -

        Benjamin Zhitomirsky, thanks for your review!
        I understand FileSystem.makeQualified(Path) will fail if the Path does not belong to the FileSystem. I'll fix them.

        Show
        Akira AJISAKA added a comment - Benjamin Zhitomirsky , thanks for your review! I understand FileSystem.makeQualified(Path) will fail if the Path does not belong to the FileSystem. I'll fix them.
        Hide
        Akira AJISAKA added a comment -

        .../util/MRAsyncDiskService.java:

        - return (new Path(path)).makeQualified(this.localFileSystem)
        + return this.localFileSystem.makeQualified(new Path(path))
              .toUri().getPath();
        

        Since this is a test code, the path is supposed to belong to localFS. I prefer to check if the path belongs to the FIleSystem in the test, and we should fix the input path when the test fails.

        I'll fix them.

        Looked around the patch, now I don't think the fix is needed. Please correct me if I am wrong.

        Show
        Akira AJISAKA added a comment - .../util/MRAsyncDiskService.java: - return ( new Path(path)).makeQualified( this .localFileSystem) + return this .localFileSystem.makeQualified( new Path(path)) .toUri().getPath(); Since this is a test code, the path is supposed to belong to localFS. I prefer to check if the path belongs to the FIleSystem in the test, and we should fix the input path when the test fails. I'll fix them. Looked around the patch, now I don't think the fix is needed. Please correct me if I am wrong.
        Hide
        Daryn Sharp added a comment -

        To summarize, this should be the expected behavior. Does this patch meet this criteria?

        FileSystem#makeQualified(Path) - verifies path belongs to the fs:

        • Fail if the path is qualified for a different fs
        • Accept qualified path on this fs
        • Qualify relative path against this fs

        Path#makeQualified(FileSystem) - uses given fs to qualify relative paths:

        • Accept if the path is qualified for any fs
        • Qualify relative path against the given fs
        Show
        Daryn Sharp added a comment - To summarize, this should be the expected behavior. Does this patch meet this criteria? FileSystem#makeQualified(Path) - verifies path belongs to the fs: Fail if the path is qualified for a different fs Accept qualified path on this fs Qualify relative path against this fs Path#makeQualified(FileSystem) - uses given fs to qualify relative paths: Accept if the path is qualified for any fs Qualify relative path against the given fs
        Hide
        Benjamin Zhitomirsky added a comment -

        IMHO, there are still issues:
        1. I believe that ioPath in Gridmix could be fully qualified path pointing to another file system.
        2. In Rumen test.tools.input.dir won't support fully qualified path anymore
        3. Similar with StreamingJob
        4. Swift must be checked as well. I am not familiar with it.
        5. You are right about MRAsyncDiskService.java, but right now I am developing a new mapreduce test which uses two HDFS file systems and will exploit this If you want you may leave it as it and I will fix it in my patch later...

        Please pay attention that:
        new FileContext(conf).makeQualified(path)

        • does the same but without any issues. It helps when source file system is not provided with the path.
        Show
        Benjamin Zhitomirsky added a comment - IMHO, there are still issues: 1. I believe that ioPath in Gridmix could be fully qualified path pointing to another file system. 2. In Rumen test.tools.input.dir won't support fully qualified path anymore 3. Similar with StreamingJob 4. Swift must be checked as well. I am not familiar with it. 5. You are right about MRAsyncDiskService.java, but right now I am developing a new mapreduce test which uses two HDFS file systems and will exploit this If you want you may leave it as it and I will fix it in my patch later... Please pay attention that: new FileContext(conf).makeQualified(path) does the same but without any issues. It helps when source file system is not provided with the path.
        Hide
        Akira AJISAKA added a comment -

        Thanks Daryn Sharp for the summary and Benjamin Zhitomirsky for the clarification!
        1 (Gridmix) and 3 (StreamingJob) are not the test code, so I need to update the patch. I'll update the patch to use Path.makeQualified(URI, Path) since I don't want to create new FileContext instance.

        Show
        Akira AJISAKA added a comment - Thanks Daryn Sharp for the summary and Benjamin Zhitomirsky for the clarification! 1 (Gridmix) and 3 (StreamingJob) are not the test code, so I need to update the patch. I'll update the patch to use Path.makeQualified(URI, Path) since I don't want to create new FileContext instance.
        Hide
        Akira AJISAKA added a comment -

        Rebased the first patch for the latest trunk.

        Show
        Akira AJISAKA added a comment - Rebased the first patch for the latest trunk.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12658383/HADOOP-10392.5.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 24 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archives hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-openstack hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming:

        org.apache.hadoop.ipc.TestRPCCallBenchmark
        org.apache.hadoop.net.TestNetUtils
        org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
        org.apache.hadoop.security.TestSecurityUtil
        org.apache.hadoop.ipc.TestRPC
        org.apache.hadoop.security.TestDoAsEffectiveUser
        org.apache.hadoop.conf.TestConfiguration
        org.apache.hadoop.ipc.TestIPC
        org.apache.hadoop.mapreduce.TestMapReduce
        org.apache.hadoop.mapreduce.lib.input.TestLineRecordReaderJobs
        org.apache.hadoop.mapred.jobcontrol.TestLocalJobControl
        org.apache.hadoop.mapred.TestJobCounters
        org.apache.hadoop.mapred.TestLocalMRNotification
        org.apache.hadoop.mapred.lib.TestDelegatingInputFormat
        org.apache.hadoop.mapred.TestReduceFetch
        org.apache.hadoop.mapreduce.TestMapReduceLazyOutput
        org.apache.hadoop.mapreduce.lib.join.TestJoinProperties
        org.apache.hadoop.mapred.lib.TestMultithreadedMapRunner
        org.apache.hadoop.mapred.TestClusterMRNotification
        org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner
        org.apache.hadoop.mapreduce.lib.chain.TestSingleElementChain
        org.apache.hadoop.mapreduce.TestMapperReducerCleanup
        org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
        org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler
        org.apache.hadoop.fs.TestFileSystem
        org.apache.hadoop.mapreduce.TestLargeSort
        org.apache.hadoop.mapred.join.TestDatamerge
        org.apache.hadoop.mapreduce.lib.input.TestMultipleInputs
        org.apache.hadoop.mapred.TestLazyOutput
        org.apache.hadoop.mapred.TestTaskCommit
        org.apache.hadoop.mapreduce.TestMRJobClient
        org.apache.hadoop.mapreduce.security.TestMRCredentials
        org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
        org.apache.hadoop.mapred.lib.TestChainMapReduce
        org.apache.hadoop.mapreduce.lib.fieldsel.TestMRFieldSelection
        org.apache.hadoop.mapreduce.lib.partition.TestMRKeyFieldBasedComparator
        org.apache.hadoop.mapreduce.lib.db.TestDataDrivenDBInputFormat
        org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
        org.apache.hadoop.mapreduce.v2.TestMRJobs
        org.apache.hadoop.mapred.TestMapRed
        org.apache.hadoop.mapred.lib.TestKeyFieldBasedComparator
        org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat
        org.apache.hadoop.mapreduce.v2.TestNonExistentJob
        org.apache.hadoop.mapreduce.lib.input.TestDelegatingInputFormat
        org.apache.hadoop.mapred.TestMiniMRChildTask
        org.apache.hadoop.fs.slive.TestSlive
        org.apache.hadoop.mapred.TestComparators
        org.apache.hadoop.mapreduce.v2.TestUberAM
        org.apache.hadoop.mapred.TestMiniMRClasspath
        org.apache.hadoop.mapred.TestMapOutputType
        org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
        org.apache.hadoop.mapred.lib.aggregate.TestAggregates
        org.apache.hadoop.ipc.TestMRCJCSocketFactory
        org.apache.hadoop.mapreduce.TestValueIterReset
        org.apache.hadoop.mapred.TestMRCJCFileInputFormat
        org.apache.hadoop.mapreduce.lib.aggregate.TestMapReduceAggregates
        org.apache.hadoop.mapred.TestReporter
        org.apache.hadoop.mapred.TestFileOutputFormat
        org.apache.hadoop.mapreduce.lib.chain.TestMapReduceChain
        org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
        org.apache.hadoop.mapreduce.TestMapCollection
        org.apache.hadoop.mapreduce.TestLocalRunner
        org.apache.hadoop.mapreduce.lib.output.TestMRMultipleOutputs
        org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
        org.apache.hadoop.mapred.TestMerge
        org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
        org.apache.hadoop.mapred.TestCollect
        org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle
        org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser
        org.apache.hadoop.fs.TestDFSIO
        org.apache.hadoop.mapred.TestUserDefinedCounters
        org.apache.hadoop.mapreduce.TestNewCombinerGrouping
        org.apache.hadoop.conf.TestNoDefaultsJobConf
        org.apache.hadoop.mapred.TestJobName
        org.apache.hadoop.mapreduce.v2.TestMRAMWithNonNormalizedCapabilities
        org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
        org.apache.hadoop.mapred.TestLineRecordReaderJobs
        org.apache.hadoop.mapreduce.lib.chain.TestChainErrors
        org.apache.hadoop.mapred.lib.TestMultipleOutputs
        org.apache.hadoop.mapred.TestJobCleanup
        org.apache.hadoop.mapreduce.TestMROutputFormat
        org.apache.hadoop.mapred.TestMiniMRBringup
        org.apache.hadoop.mapred.TestOldCombinerGrouping
        org.apache.hadoop.mapred.TestClusterMapReduceTestCase
        org.apache.hadoop.mapreduce.TestChild
        org.apache.hadoop.mapreduce.lib.join.TestJoinDatamerge
        org.apache.hadoop.mapred.TestNetworkedJob
        org.apache.hadoop.mapred.TestMiniMRClientCluster
        org.apache.hadoop.mapred.TestClientRedirect
        org.apache.hadoop.mapreduce.v2.TestRMNMInfo
        org.apache.hadoop.mapred.TestJavaSerialization
        org.apache.hadoop.mapred.TestFieldSelection
        org.apache.hadoop.mapred.TestJobSysDirWithDFS
        org.apache.hadoop.mapreduce.lib.map.TestMultithreadedMapper
        org.apache.hadoop.mapreduce.lib.db.TestDBJob
        org.apache.hadoop.examples.TestWordStats
        org.apache.hadoop.tools.TestHadoopArchives
        org.apache.hadoop.tools.TestDistCh
        org.apache.hadoop.tools.TestCopyFiles
        org.apache.hadoop.mapred.gridmix.TestSleepJob
        org.apache.hadoop.mapred.gridmix.TestLoadJob
        org.apache.hadoop.mapred.gridmix.TestGridmixSubmission
        org.apache.hadoop.mapred.gridmix.TestCompressionEmulationUtils
        org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation
        org.apache.hadoop.streaming.TestMultipleArchiveFiles
        org.apache.hadoop.streaming.TestStreamingBadRecords
        org.apache.hadoop.streaming.TestStreamingOutputKeyValueTypes
        org.apache.hadoop.streaming.TestStreamAggregate
        org.apache.hadoop.streaming.TestStreamReduceNone
        org.apache.hadoop.streaming.TestUnconsumedInput
        org.apache.hadoop.streaming.mapreduce.TestStreamXmlRecordReader
        org.apache.hadoop.streaming.TestGzipInput
        org.apache.hadoop.streaming.TestStreaming
        org.apache.hadoop.streaming.TestDumpTypedBytes
        org.apache.hadoop.streaming.TestLoadTypedBytes
        org.apache.hadoop.streaming.TestStreamingSeparator
        org.apache.hadoop.streaming.TestStreamingCounters
        org.apache.hadoop.streaming.TestFileArgs
        org.apache.hadoop.streaming.TestStreamDataProtocol
        org.apache.hadoop.streaming.TestStreamingExitStatus
        org.apache.hadoop.streaming.TestSymLink
        org.apache.hadoop.streaming.TestRawBytesStreaming
        org.apache.hadoop.streaming.TestStreamingCombiner
        org.apache.hadoop.streaming.TestStreamingStderr
        org.apache.hadoop.streaming.TestTypedBytesStreaming
        org.apache.hadoop.streaming.TestStreamingBackground
        org.apache.hadoop.streaming.TestStreamXmlRecordReader
        org.apache.hadoop.streaming.TestStreamingKeyValue
        org.apache.hadoop.streaming.TestStreamXmlMultipleRecords
        org.apache.hadoop.streaming.TestMultipleCachefiles
        org.apache.hadoop.streaming.TestStreamingOutputOnlyKeys

        The following test timeouts occurred in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archives hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-openstack hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming:

        org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl
        org.apache.hadoop.mapred.pipes.TestPipeApplication

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4382//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4382//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12658383/HADOOP-10392.5.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 24 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archives hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-openstack hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming: org.apache.hadoop.ipc.TestRPCCallBenchmark org.apache.hadoop.net.TestNetUtils org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl org.apache.hadoop.security.TestSecurityUtil org.apache.hadoop.ipc.TestRPC org.apache.hadoop.security.TestDoAsEffectiveUser org.apache.hadoop.conf.TestConfiguration org.apache.hadoop.ipc.TestIPC org.apache.hadoop.mapreduce.TestMapReduce org.apache.hadoop.mapreduce.lib.input.TestLineRecordReaderJobs org.apache.hadoop.mapred.jobcontrol.TestLocalJobControl org.apache.hadoop.mapred.TestJobCounters org.apache.hadoop.mapred.TestLocalMRNotification org.apache.hadoop.mapred.lib.TestDelegatingInputFormat org.apache.hadoop.mapred.TestReduceFetch org.apache.hadoop.mapreduce.TestMapReduceLazyOutput org.apache.hadoop.mapreduce.lib.join.TestJoinProperties org.apache.hadoop.mapred.lib.TestMultithreadedMapRunner org.apache.hadoop.mapred.TestClusterMRNotification org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner org.apache.hadoop.mapreduce.lib.chain.TestSingleElementChain org.apache.hadoop.mapreduce.TestMapperReducerCleanup org.apache.hadoop.mapreduce.security.TestBinaryTokenFile org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler org.apache.hadoop.fs.TestFileSystem org.apache.hadoop.mapreduce.TestLargeSort org.apache.hadoop.mapred.join.TestDatamerge org.apache.hadoop.mapreduce.lib.input.TestMultipleInputs org.apache.hadoop.mapred.TestLazyOutput org.apache.hadoop.mapred.TestTaskCommit org.apache.hadoop.mapreduce.TestMRJobClient org.apache.hadoop.mapreduce.security.TestMRCredentials org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers org.apache.hadoop.mapred.lib.TestChainMapReduce org.apache.hadoop.mapreduce.lib.fieldsel.TestMRFieldSelection org.apache.hadoop.mapreduce.lib.partition.TestMRKeyFieldBasedComparator org.apache.hadoop.mapreduce.lib.db.TestDataDrivenDBInputFormat org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath org.apache.hadoop.mapreduce.v2.TestMRJobs org.apache.hadoop.mapred.TestMapRed org.apache.hadoop.mapred.lib.TestKeyFieldBasedComparator org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat org.apache.hadoop.mapreduce.v2.TestNonExistentJob org.apache.hadoop.mapreduce.lib.input.TestDelegatingInputFormat org.apache.hadoop.mapred.TestMiniMRChildTask org.apache.hadoop.fs.slive.TestSlive org.apache.hadoop.mapred.TestComparators org.apache.hadoop.mapreduce.v2.TestUberAM org.apache.hadoop.mapred.TestMiniMRClasspath org.apache.hadoop.mapred.TestMapOutputType org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter org.apache.hadoop.mapred.lib.aggregate.TestAggregates org.apache.hadoop.ipc.TestMRCJCSocketFactory org.apache.hadoop.mapreduce.TestValueIterReset org.apache.hadoop.mapred.TestMRCJCFileInputFormat org.apache.hadoop.mapreduce.lib.aggregate.TestMapReduceAggregates org.apache.hadoop.mapred.TestReporter org.apache.hadoop.mapred.TestFileOutputFormat org.apache.hadoop.mapreduce.lib.chain.TestMapReduceChain org.apache.hadoop.mapred.TestReduceFetchFromPartialMem org.apache.hadoop.mapreduce.TestMapCollection org.apache.hadoop.mapreduce.TestLocalRunner org.apache.hadoop.mapreduce.lib.output.TestMRMultipleOutputs org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService org.apache.hadoop.mapred.TestMerge org.apache.hadoop.mapreduce.v2.TestMROldApiJobs org.apache.hadoop.mapred.TestCollect org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser org.apache.hadoop.fs.TestDFSIO org.apache.hadoop.mapred.TestUserDefinedCounters org.apache.hadoop.mapreduce.TestNewCombinerGrouping org.apache.hadoop.conf.TestNoDefaultsJobConf org.apache.hadoop.mapred.TestJobName org.apache.hadoop.mapreduce.v2.TestMRAMWithNonNormalizedCapabilities org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution org.apache.hadoop.mapred.TestLineRecordReaderJobs org.apache.hadoop.mapreduce.lib.chain.TestChainErrors org.apache.hadoop.mapred.lib.TestMultipleOutputs org.apache.hadoop.mapred.TestJobCleanup org.apache.hadoop.mapreduce.TestMROutputFormat org.apache.hadoop.mapred.TestMiniMRBringup org.apache.hadoop.mapred.TestOldCombinerGrouping org.apache.hadoop.mapred.TestClusterMapReduceTestCase org.apache.hadoop.mapreduce.TestChild org.apache.hadoop.mapreduce.lib.join.TestJoinDatamerge org.apache.hadoop.mapred.TestNetworkedJob org.apache.hadoop.mapred.TestMiniMRClientCluster org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapreduce.v2.TestRMNMInfo org.apache.hadoop.mapred.TestJavaSerialization org.apache.hadoop.mapred.TestFieldSelection org.apache.hadoop.mapred.TestJobSysDirWithDFS org.apache.hadoop.mapreduce.lib.map.TestMultithreadedMapper org.apache.hadoop.mapreduce.lib.db.TestDBJob org.apache.hadoop.examples.TestWordStats org.apache.hadoop.tools.TestHadoopArchives org.apache.hadoop.tools.TestDistCh org.apache.hadoop.tools.TestCopyFiles org.apache.hadoop.mapred.gridmix.TestSleepJob org.apache.hadoop.mapred.gridmix.TestLoadJob org.apache.hadoop.mapred.gridmix.TestGridmixSubmission org.apache.hadoop.mapred.gridmix.TestCompressionEmulationUtils org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation org.apache.hadoop.streaming.TestMultipleArchiveFiles org.apache.hadoop.streaming.TestStreamingBadRecords org.apache.hadoop.streaming.TestStreamingOutputKeyValueTypes org.apache.hadoop.streaming.TestStreamAggregate org.apache.hadoop.streaming.TestStreamReduceNone org.apache.hadoop.streaming.TestUnconsumedInput org.apache.hadoop.streaming.mapreduce.TestStreamXmlRecordReader org.apache.hadoop.streaming.TestGzipInput org.apache.hadoop.streaming.TestStreaming org.apache.hadoop.streaming.TestDumpTypedBytes org.apache.hadoop.streaming.TestLoadTypedBytes org.apache.hadoop.streaming.TestStreamingSeparator org.apache.hadoop.streaming.TestStreamingCounters org.apache.hadoop.streaming.TestFileArgs org.apache.hadoop.streaming.TestStreamDataProtocol org.apache.hadoop.streaming.TestStreamingExitStatus org.apache.hadoop.streaming.TestSymLink org.apache.hadoop.streaming.TestRawBytesStreaming org.apache.hadoop.streaming.TestStreamingCombiner org.apache.hadoop.streaming.TestStreamingStderr org.apache.hadoop.streaming.TestTypedBytesStreaming org.apache.hadoop.streaming.TestStreamingBackground org.apache.hadoop.streaming.TestStreamXmlRecordReader org.apache.hadoop.streaming.TestStreamingKeyValue org.apache.hadoop.streaming.TestStreamXmlMultipleRecords org.apache.hadoop.streaming.TestMultipleCachefiles org.apache.hadoop.streaming.TestStreamingOutputOnlyKeys The following test timeouts occurred in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archives hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-openstack hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming: org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl org.apache.hadoop.mapred.pipes.TestPipeApplication +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4382//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4382//console This message is automatically generated.
        Hide
        Daryn Sharp added a comment -

        Please clarify if FileSystem isn't currently working as I described?

        Show
        Daryn Sharp added a comment - Please clarify if FileSystem isn't currently working as I described?
        Hide
        Akira AJISAKA added a comment -

        Re-reading the code, I've changed my mind.

        In StreamJob.java#parseArgv(), If someone uses StreamJob pointing another FileSystem (such as hdfs://...) by '-file' option, the path is changed to LocalFileSystem (file://...) by Path.makeQualified(FileSystem).

          packageFiles_.add(file);
          try {
            Path path = new Path(file);
            FileSystem localFs = FileSystem.getLocal(config_);
            String finalPath = path.makeQualified(localFs).toString();
        

        However, the code later checks if the file exists, so StreamJob fails if a user sets another FileSystem by -file option.

           validate(packageFiles_);
        

        Therefore, I'm thinking it's okay to use localFs.makeQualified(path). If the code is changed to use it, StreamJob fails there in the example but it's not changed whether StreamJob fails or not.

        In Gridmix.java,

          final FileSystem inputFs = ioPath.getFileSystem(conf);
          ioPath = ioPath.makeQualified(inputFs);
          boolean succeeded = false;
          try {
            succeeded = FileSystem.mkdirs(inputFs, ioPath, new FsPermission((short)0777));
        

        ioPath.getFileSystem(conf) returns the FileSystem which owns ioPath. Therefore if ioPath.makeQualified(inputFs) is changed to inputFs.makeQualified(ioPath), it doesn't fail.

        Show
        Akira AJISAKA added a comment - Re-reading the code, I've changed my mind. In StreamJob.java#parseArgv(), If someone uses StreamJob pointing another FileSystem (such as hdfs://...) by '-file' option, the path is changed to LocalFileSystem ( file:// ...) by Path.makeQualified(FileSystem) . packageFiles_.add(file); try { Path path = new Path(file); FileSystem localFs = FileSystem.getLocal(config_); String finalPath = path.makeQualified(localFs).toString(); However, the code later checks if the file exists, so StreamJob fails if a user sets another FileSystem by -file option. validate(packageFiles_); Therefore, I'm thinking it's okay to use localFs.makeQualified(path) . If the code is changed to use it, StreamJob fails there in the example but it's not changed whether StreamJob fails or not. In Gridmix.java, final FileSystem inputFs = ioPath.getFileSystem(conf); ioPath = ioPath.makeQualified(inputFs); boolean succeeded = false ; try { succeeded = FileSystem.mkdirs(inputFs, ioPath, new FsPermission(( short )0777)); ioPath.getFileSystem(conf) returns the FileSystem which owns ioPath. Therefore if ioPath.makeQualified(inputFs) is changed to inputFs.makeQualified(ioPath), it doesn't fail.
        Hide
        Akira AJISAKA added a comment -

        In Rumen test.tools.input.dir won't support fully qualified path anymore

        test.tools.input.dir supports file://... for pointing LocalFileSystem. Users can use it.

        Swift must be checked as well. I am not familiar with it.

        I searched Path.makeQualified(FileSystem) pointing another FileSystem, but I couldn't find them in the test code. All the path don't have its scheme.

        Show
        Akira AJISAKA added a comment - In Rumen test.tools.input.dir won't support fully qualified path anymore test.tools.input.dir supports file:// ... for pointing LocalFileSystem. Users can use it. Swift must be checked as well. I am not familiar with it. I searched Path.makeQualified(FileSystem) pointing another FileSystem, but I couldn't find them in the test code. All the path don't have its scheme.
        Hide
        Akira AJISAKA added a comment -

        That's why now I prefer v4 patch.

        Show
        Akira AJISAKA added a comment - That's why now I prefer v4 patch.
        Hide
        Daryn Sharp added a comment -

        (while closing tabs I noticed I didn't submit this comment)

        Both examples demonstrate correct behavior that should not fail. Perhaps it's my pre-coffee brain, but I'm confused if you are proposing changes to streaming and gridmix path handling.

        In the stream job example, if path is qualified as an hdfs:// path then path#makeQualified(localFs) should return the path as-is, not rewrite it to localFs as described. That would be a horrible bug! The path should be qualified with the given fs if and only if the path isn't already qualified. Changing to localFs.makeQualified(path) is wrong because it will artificial fail for non-local paths.

        In gridmix example, it's perfectly correct: it's qualifying the path with its own fs. For relative paths, it will qualify with the default fs. It's a no-op for qualified paths because it's already qualified with the given fs.

        Please elaborate on whether the current behavior is wrong, or if the patch requires the discussed changes - in which case the patch has introduced bugs.

        Show
        Daryn Sharp added a comment - (while closing tabs I noticed I didn't submit this comment) Both examples demonstrate correct behavior that should not fail. Perhaps it's my pre-coffee brain, but I'm confused if you are proposing changes to streaming and gridmix path handling. In the stream job example, if path is qualified as an hdfs:// path then path#makeQualified(localFs) should return the path as-is, not rewrite it to localFs as described. That would be a horrible bug! The path should be qualified with the given fs if and only if the path isn't already qualified. Changing to localFs.makeQualified(path) is wrong because it will artificial fail for non-local paths. In gridmix example, it's perfectly correct: it's qualifying the path with its own fs. For relative paths, it will qualify with the default fs. It's a no-op for qualified paths because it's already qualified with the given fs. Please elaborate on whether the current behavior is wrong, or if the patch requires the discussed changes - in which case the patch has introduced bugs.
        Hide
        Akira AJISAKA added a comment -

        Thanks Daryn Sharp for the comment.

        Changing to localFs.makeQualified(path) is wrong because it will artificial fail for non-local paths.

        Hadoop Streaming '-file' option only accepts local files. If non-local path is set, StreamJob will fail in

           validate(packageFiles_);
        

        method, which validates if the file path exists in localFs.

          private void validate(final List<String> values)
          throws IllegalArgumentException {
            for (String file : values) {
              File f = new File(file);
              if (!FileUtil.canRead(f)) {
                fail("File: " + f.getAbsolutePath()
                  + " does not exist, or is not readable.");
              }
            }
          }
        

        Therefore, I'm thinking it's okay to fail artificially in LocalFs.makeQualified(path) because the job eventually fails with non-local paths.
        In addition, it's also fine for me to use Path.makeQualified(URI, Path) in StreamingJob, that does not change the behavior at all.

        Show
        Akira AJISAKA added a comment - Thanks Daryn Sharp for the comment. Changing to localFs.makeQualified(path) is wrong because it will artificial fail for non-local paths. Hadoop Streaming '-file' option only accepts local files. If non-local path is set, StreamJob will fail in validate(packageFiles_); method, which validates if the file path exists in localFs. private void validate( final List< String > values) throws IllegalArgumentException { for ( String file : values) { File f = new File(file); if (!FileUtil.canRead(f)) { fail( "File: " + f.getAbsolutePath() + " does not exist, or is not readable." ); } } } Therefore, I'm thinking it's okay to fail artificially in LocalFs.makeQualified(path) because the job eventually fails with non-local paths. In addition, it's also fine for me to use Path.makeQualified(URI, Path) in StreamingJob, that does not change the behavior at all.
        Hide
        Akira AJISAKA added a comment -

        Attaching v6 patch to use Path.makeQualified(URI, Path) in StreamJob.

        Show
        Akira AJISAKA added a comment - Attaching v6 patch to use Path.makeQualified(URI, Path) in StreamJob.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12660604/HADOOP-10392.6.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 24 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archives hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-openstack hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4442//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4442//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12660604/HADOOP-10392.6.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 24 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archives hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-openstack hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4442//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4442//console This message is automatically generated.
        Hide
        Steve Loughran added a comment -

        Quite a fiew of the MR tests go {{System.getProperty("test.build.data", "/tmp"))) ... they should fall back to something cross platform, maybe just "target" or the system tmp dir property. Even though this is on the same line as these patches, I think it's something to keep separate to reduce the delta of this patch

        What I would recommend, though is that in TestHistograms the call System.getProperty("test.tools.input.dir", "") is hardened to something like System.getProperty("test.tools.input.dir", "target/input") to give it some kind of fallback on an IDE-run that doesn't set the system properties right.

        Otherwise I'm happy with it. Let's see what happens against trunk now

        Show
        Steve Loughran added a comment - Quite a fiew of the MR tests go {{System.getProperty("test.build.data", "/tmp"))) ... they should fall back to something cross platform, maybe just "target" or the system tmp dir property. Even though this is on the same line as these patches, I think it's something to keep separate to reduce the delta of this patch What I would recommend, though is that in TestHistograms the call System.getProperty("test.tools.input.dir", "") is hardened to something like System.getProperty("test.tools.input.dir", "target/input") to give it some kind of fallback on an IDE-run that doesn't set the system properties right. Otherwise I'm happy with it. Let's see what happens against trunk now
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12660604/HADOOP-10392.6.patch
        against trunk revision 786dbdf.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5451//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12660604/HADOOP-10392.6.patch against trunk revision 786dbdf. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5451//console This message is automatically generated.
        Hide
        Akira AJISAKA added a comment -

        Thanks Steve Loughran for the comment. Rebased for the latest trunk and fixed rootInputDir to use target/input by default in TestHistograms.java.

        Show
        Akira AJISAKA added a comment - Thanks Steve Loughran for the comment. Rebased for the latest trunk and fixed rootInputDir to use target/input by default in TestHistograms.java .
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12696257/HADOOP-10392.7.patch
        against trunk revision 1154281.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 24 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 7 new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archives hadoop-tools/hadoop-aws hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-openstack hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming:

        org.apache.hadoop.mapred.TestMiniMRChildTask
        org.apache.hadoop.conf.TestJobConf
        org.apache.hadoop.mapred.TestTaskCommit
        org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
        org.apache.hadoop.mapreduce.TestLargeSort
        org.apache.hadoop.mapred.TestFileOutputFormat
        org.apache.hadoop.mapred.TestReporter

        The test build failed in hadoop-tools/hadoop-streaming hadoop-tools/hadoop-rumen hadoop-tools/hadoop-openstack hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-aws

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5570//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5570//artifact/patchprocess/newPatchFindbugsWarningshadoop-rumen.html
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5570//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12696257/HADOOP-10392.7.patch against trunk revision 1154281. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 24 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 7 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archives hadoop-tools/hadoop-aws hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-openstack hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming: org.apache.hadoop.mapred.TestMiniMRChildTask org.apache.hadoop.conf.TestJobConf org.apache.hadoop.mapred.TestTaskCommit org.apache.hadoop.mapred.TestReduceFetchFromPartialMem org.apache.hadoop.mapreduce.TestLargeSort org.apache.hadoop.mapred.TestFileOutputFormat org.apache.hadoop.mapred.TestReporter The test build failed in hadoop-tools/hadoop-streaming hadoop-tools/hadoop-rumen hadoop-tools/hadoop-openstack hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-aws Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5570//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5570//artifact/patchprocess/newPatchFindbugsWarningshadoop-rumen.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5570//console This message is automatically generated.
        Hide
        Akira AJISAKA added a comment -

        The findbugs warnings in hadoop-rumen are not related to the patch. I'll file a jira for fixing the warnings.

        Show
        Akira AJISAKA added a comment - The findbugs warnings in hadoop-rumen are not related to the patch. I'll file a jira for fixing the warnings.
        Hide
        Akira AJISAKA added a comment -

        Filed MAPREDUCE-6243 for findbugs warnings.

        Show
        Akira AJISAKA added a comment - Filed MAPREDUCE-6243 for findbugs warnings.
        Hide
        Akira AJISAKA added a comment -

        The test failures look unrelated to the patch. Resubmitting.

        Show
        Akira AJISAKA added a comment - The test failures look unrelated to the patch. Resubmitting.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12696564/HADOOP-10392.7.patch
        against trunk revision e04e8fa.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 24 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archives hadoop-tools/hadoop-aws hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-openstack hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming:

        org.apache.hadoop.conf.TestJobConf
        org.apache.hadoop.mapreduce.TestLargeSort
        org.apache.hadoop.mapred.gridmix.TestHighRamJob

        The test build failed in hadoop-tools/hadoop-streaming hadoop-tools/hadoop-rumen hadoop-tools/hadoop-archives hadoop-mapreduce-project/hadoop-mapreduce-examples

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5587//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5587//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12696564/HADOOP-10392.7.patch against trunk revision e04e8fa. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 24 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archives hadoop-tools/hadoop-aws hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-openstack hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming: org.apache.hadoop.conf.TestJobConf org.apache.hadoop.mapreduce.TestLargeSort org.apache.hadoop.mapred.gridmix.TestHighRamJob The test build failed in hadoop-tools/hadoop-streaming hadoop-tools/hadoop-rumen hadoop-tools/hadoop-archives hadoop-mapreduce-project/hadoop-mapreduce-examples Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5587//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5587//console This message is automatically generated.
        Hide
        Akira AJISAKA added a comment -

        These failures are not related to the patch. MAPREDUCE-6223, MAPREDUCE-6234, and MAPREDUCE-6233 will fix them.

        Show
        Akira AJISAKA added a comment - These failures are not related to the patch. MAPREDUCE-6223 , MAPREDUCE-6234 , and MAPREDUCE-6233 will fix them.
        Hide
        Akira AJISAKA added a comment -

        Rebased for the latest trunk.

        Show
        Akira AJISAKA added a comment - Rebased for the latest trunk.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12707690/HADOOP-10392.8.patch
        against trunk revision af618f2.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 24 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-tools/hadoop-archives hadoop-tools/hadoop-aws hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-openstack hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming:

        org.apache.hadoop.mapred.TestMRIntermediateDataEncryption
        org.apache.hadoop.mapred.TestReduceFetch
        org.apache.hadoop.mapred.TestMiniMRClasspath
        org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
        org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
        org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
        org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler
        org.apache.hadoop.mapreduce.TestMapReduceLazyOutput
        org.apache.hadoop.mapreduce.security.TestMRCredentials
        org.apache.hadoop.ipc.TestMRCJCSocketFactory
        org.apache.hadoop.mapred.TestMiniMRBringup
        org.apache.hadoop.conf.TestNoDefaultsJobConf
        org.apache.hadoop.mapreduce.TestChild
        org.apache.hadoop.mapred.TestMRTimelineEventHandling
        org.apache.hadoop.mapred.TestLazyOutput
        org.apache.hadoop.mapred.TestJobCleanup
        org.apache.hadoop.mapred.TestJobCounters
        org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser
        org.apache.hadoop.mapred.TestJobName
        org.apache.hadoop.mapreduce.v2.TestUberAM
        org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
        org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner
        org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
        org.apache.hadoop.mapreduce.v2.TestRMNMInfo
        org.apache.hadoop.mapred.TestMerge
        org.apache.hadoop.mapreduce.v2.TestNonExistentJob
        org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
        org.apache.hadoop.mapred.TestClusterMapReduceTestCase
        org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
        org.apache.hadoop.mapred.TestJobSysDirWithDFS
        org.apache.hadoop.mapred.TestMiniMRClientCluster
        org.apache.hadoop.mapreduce.TestLargeSort
        org.apache.hadoop.mapreduce.TestMRJobClient
        org.apache.hadoop.mapreduce.v2.TestMRJobs
        org.apache.hadoop.mapreduce.v2.TestMRAMWithNonNormalizedCapabilities
        org.apache.hadoop.mapred.TestClusterMRNotification
        org.apache.hadoop.mapred.TestNetworkedJob
        org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle
        org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
        org.apache.hadoop.mapred.TestMiniMRChildTask
        org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation
        org.apache.hadoop.mapred.gridmix.TestLoadJob
        org.apache.hadoop.mapred.gridmix.TestGridmixSubmission
        org.apache.hadoop.mapred.gridmix.TestSleepJob
        org.apache.hadoop.streaming.TestFileArgs
        org.apache.hadoop.streaming.TestMultipleCachefiles
        org.apache.hadoop.streaming.TestMultipleArchiveFiles
        org.apache.hadoop.streaming.TestSymLink
        org.apache.hadoop.streaming.TestStreamingBadRecords

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/6010//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/6010//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12707690/HADOOP-10392.8.patch against trunk revision af618f2. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 24 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-tools/hadoop-archives hadoop-tools/hadoop-aws hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-openstack hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming: org.apache.hadoop.mapred.TestMRIntermediateDataEncryption org.apache.hadoop.mapred.TestReduceFetch org.apache.hadoop.mapred.TestMiniMRClasspath org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution org.apache.hadoop.mapreduce.v2.TestMROldApiJobs org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler org.apache.hadoop.mapreduce.TestMapReduceLazyOutput org.apache.hadoop.mapreduce.security.TestMRCredentials org.apache.hadoop.ipc.TestMRCJCSocketFactory org.apache.hadoop.mapred.TestMiniMRBringup org.apache.hadoop.conf.TestNoDefaultsJobConf org.apache.hadoop.mapreduce.TestChild org.apache.hadoop.mapred.TestMRTimelineEventHandling org.apache.hadoop.mapred.TestLazyOutput org.apache.hadoop.mapred.TestJobCleanup org.apache.hadoop.mapred.TestJobCounters org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser org.apache.hadoop.mapred.TestJobName org.apache.hadoop.mapreduce.v2.TestUberAM org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers org.apache.hadoop.mapreduce.v2.TestRMNMInfo org.apache.hadoop.mapred.TestMerge org.apache.hadoop.mapreduce.v2.TestNonExistentJob org.apache.hadoop.mapred.TestReduceFetchFromPartialMem org.apache.hadoop.mapred.TestClusterMapReduceTestCase org.apache.hadoop.mapreduce.security.TestBinaryTokenFile org.apache.hadoop.mapred.TestJobSysDirWithDFS org.apache.hadoop.mapred.TestMiniMRClientCluster org.apache.hadoop.mapreduce.TestLargeSort org.apache.hadoop.mapreduce.TestMRJobClient org.apache.hadoop.mapreduce.v2.TestMRJobs org.apache.hadoop.mapreduce.v2.TestMRAMWithNonNormalizedCapabilities org.apache.hadoop.mapred.TestClusterMRNotification org.apache.hadoop.mapred.TestNetworkedJob org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath org.apache.hadoop.mapred.TestMiniMRChildTask org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation org.apache.hadoop.mapred.gridmix.TestLoadJob org.apache.hadoop.mapred.gridmix.TestGridmixSubmission org.apache.hadoop.mapred.gridmix.TestSleepJob org.apache.hadoop.streaming.TestFileArgs org.apache.hadoop.streaming.TestMultipleCachefiles org.apache.hadoop.streaming.TestMultipleArchiveFiles org.apache.hadoop.streaming.TestSymLink org.apache.hadoop.streaming.TestStreamingBadRecords Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/6010//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/6010//console This message is automatically generated.

          People

          • Assignee:
            Akira AJISAKA
            Reporter:
            Akira AJISAKA
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:

              Development