Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6811

[ATS1.5] All history logs should be kept under its own User Directory.

    Details

    • Hadoop Flags:
      Reviewed

      Description

      ATS1.5 allows to store history data in underlying FileSystem folder path i.e /acitve-dir and /done-dir. These base directories are protected for unauthorized user access for other users data by setting sticky bit for /active-dir.

      But object store filesystems such as WASB does not have user access control on folders and files. When WASB are used as underlying file system for ATS1.5, the history data which are stored in FS are accessible to all users. This would be a security risk

      I would propose to keep history data under its own user directory i.e /active-dir/$USER. Even this do not solve basic user access from FS, but it provides capability to plugin Apache Ranger policies for each user folders. One thing to note that setting policies to each user folder is admin responsibility. But grouping all history data of one user folder allows to set policies so that user access control is achieved.

      1. YARN-6811.01.patch
        14 kB
        Rohith Sharma K S
      2. YARN-6811.02.patch
        22 kB
        Rohith Sharma K S
      3. YARN-6811-branch-2.01.patch
        22 kB
        Rohith Sharma K S

        Activity

        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Attaching patch to store entity files under its own directory.

        Show
        rohithsharma Rohith Sharma K S added a comment - Attaching patch to store entity files under its own directory.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 10s Maven dependency ordering for branch
        +1 mvninstall 13m 25s trunk passed
        +1 compile 8m 33s trunk passed
        +1 checkstyle 0m 57s trunk passed
        +1 mvnsite 1m 36s trunk passed
        +1 findbugs 2m 47s trunk passed
        +1 javadoc 1m 25s trunk passed
              Patch Compile Tests
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 9s the patch passed
        +1 compile 5m 14s the patch passed
        +1 javac 5m 14s the patch passed
        -0 checkstyle 0m 55s hadoop-yarn-project/hadoop-yarn: The patch generated 7 new + 229 unchanged - 11 fixed = 236 total (was 240)
        +1 mvnsite 1m 33s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 8s the patch passed
        +1 javadoc 1m 22s the patch passed
              Other Tests
        +1 unit 0m 31s hadoop-yarn-api in the patch passed.
        +1 unit 2m 27s hadoop-yarn-common in the patch passed.
        +1 unit 1m 6s hadoop-yarn-server-timeline-pluginstorage in the patch passed.
        +1 asflicense 0m 27s The patch does not generate ASF License warnings.
        55m 7s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6811
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879617/YARN-6811.01.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 970ca0826447 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0fd6d0f
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16619/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16619/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16619/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 13m 25s trunk passed +1 compile 8m 33s trunk passed +1 checkstyle 0m 57s trunk passed +1 mvnsite 1m 36s trunk passed +1 findbugs 2m 47s trunk passed +1 javadoc 1m 25s trunk passed       Patch Compile Tests 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 9s the patch passed +1 compile 5m 14s the patch passed +1 javac 5m 14s the patch passed -0 checkstyle 0m 55s hadoop-yarn-project/hadoop-yarn: The patch generated 7 new + 229 unchanged - 11 fixed = 236 total (was 240) +1 mvnsite 1m 33s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 8s the patch passed +1 javadoc 1m 22s the patch passed       Other Tests +1 unit 0m 31s hadoop-yarn-api in the patch passed. +1 unit 2m 27s hadoop-yarn-common in the patch passed. +1 unit 1m 6s hadoop-yarn-server-timeline-pluginstorage in the patch passed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 55m 7s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6811 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879617/YARN-6811.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 970ca0826447 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0fd6d0f Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16619/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16619/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/16619/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        djp Junping Du added a comment -

        Thanks Rohith Sharma K S for contributing the patch! The approach here looks general good to me. The only concern here is it could be a bit performance impact as it will search two directories (with User and without user). I think one improve could we don't search user directory when "keep-under-user-dir" set to false. The verse is not true because we need to handle rolling upgrade case.

        Some detail comments:

        public static final String
        +      TIMELINE_SERVICE_ENTITYGROUP_FS_STORE_KEEP_UNDER_USER_DIR =
        +      TIMELINE_SERVICE_ENTITYGROUP_FS_STORE_PREFIX + "keep-under-user-dir"
        

        The name of new added configuration is too long, can it simply be "with-user-dir"?

        We should document the new configuration in yarn-default.xml with proper explanation of how this configuration is used for.

        Like my comments offline, createUserDir(String user) should have a better name given it doesn't already create user dir (depends on configuration). May be better to call it getAppRootDir()?

        We need to handle rolling upgrade case. I think we can add a unit test here as we can write app log with "keep-under-user-dir" = false for writing to old location, and try to read it out when set "keep-under-user-dir" = true.

        Show
        djp Junping Du added a comment - Thanks Rohith Sharma K S for contributing the patch! The approach here looks general good to me. The only concern here is it could be a bit performance impact as it will search two directories (with User and without user). I think one improve could we don't search user directory when "keep-under-user-dir" set to false. The verse is not true because we need to handle rolling upgrade case. Some detail comments: public static final String + TIMELINE_SERVICE_ENTITYGROUP_FS_STORE_KEEP_UNDER_USER_DIR = + TIMELINE_SERVICE_ENTITYGROUP_FS_STORE_PREFIX + "keep-under-user-dir" The name of new added configuration is too long, can it simply be "with-user-dir"? We should document the new configuration in yarn-default.xml with proper explanation of how this configuration is used for. Like my comments offline, createUserDir(String user) should have a better name given it doesn't already create user dir (depends on configuration). May be better to call it getAppRootDir() ? We need to handle rolling upgrade case. I think we can add a unit test here as we can write app log with "keep-under-user-dir" = false for writing to old location, and try to read it out when set "keep-under-user-dir" = true.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Thanks Junping Du for the reviews. Couple of items to clear.

        1. TimelineClient: The configuration exposed is only used at client side configuration. TimelineClient can decide whether to write inside user-dir or not. It does not impact running timeline clients during rolling upgrade. Running timelineclient still write into old path after upgrade.
        2. TimelineServer : After this patch, TimelineServer supports reading data from active-dir and user-active-dir which is subfolder of active-dir. So, newly added configuration is not needed at server side.
          1. Scanning active directory and move completed application into done directory. It happens for every regular interval. Scanning happens recursively under active path which takes care of moving data from user directory as well.
          2. If read request came while application is still running which need to be cached
            • The existing behavior i.e reading active application data from active-path still continue to support. No change for this code path. It do not impact rolling upgrade
            • If server do not find application data under active-path directory then try to find under user directory which are sub folder of active path. This would affect little bit performance since active application path need to searched under active path sub folders.

        Overall

        1. Rolling upgrade do not affect with this patch. Downgrading the TimelineServer would be concern that server will not serve from user-dir. IIUC, downgrading would always result in loosing some of the existing support. I think we need not much worry about it.
        2. Performance: Given timeline clients are written inside user-dir then impact is there on performance if read request come for active application. Once application is finished, then there is NO impact on performance.

        I think one improve could we don't search user directory when "keep-under-user-dir" set to false.

        It is client side configuration as I explained above

        The name of new added configuration is too long, can it simply be "with-user-dir"?

        agree

        We should document the new configuration in yarn-default.xml

        agree

        Like my comments offline, createUserDir(String user) should have a better name given it doesn't already create user dir

        I missed to update it. I will do in next path.

        I think we can add a unit test here as we can write app log

        Yep, to read from active-path and sub folder of it as well.

        Show
        rohithsharma Rohith Sharma K S added a comment - Thanks Junping Du for the reviews. Couple of items to clear. TimelineClient : The configuration exposed is only used at client side configuration. TimelineClient can decide whether to write inside user-dir or not. It does not impact running timeline clients during rolling upgrade. Running timelineclient still write into old path after upgrade. TimelineServer : After this patch, TimelineServer supports reading data from active-dir and user-active-dir which is subfolder of active-dir. So, newly added configuration is not needed at server side. Scanning active directory and move completed application into done directory. It happens for every regular interval. Scanning happens recursively under active path which takes care of moving data from user directory as well. If read request came while application is still running which need to be cached The existing behavior i.e reading active application data from active-path still continue to support. No change for this code path. It do not impact rolling upgrade If server do not find application data under active-path directory then try to find under user directory which are sub folder of active path. This would affect little bit performance since active application path need to searched under active path sub folders. Overall Rolling upgrade do not affect with this patch. Downgrading the TimelineServer would be concern that server will not serve from user-dir. IIUC, downgrading would always result in loosing some of the existing support. I think we need not much worry about it. Performance: Given timeline clients are written inside user-dir then impact is there on performance if read request come for active application. Once application is finished, then there is NO impact on performance. I think one improve could we don't search user directory when "keep-under-user-dir" set to false. It is client side configuration as I explained above The name of new added configuration is too long, can it simply be "with-user-dir"? agree We should document the new configuration in yarn-default.xml agree Like my comments offline, createUserDir(String user) should have a better name given it doesn't already create user dir I missed to update it. I will do in next path. I think we can add a unit test here as we can write app log Yep, to read from active-path and sub folder of it as well.
        Hide
        djp Junping Du added a comment -

        Thanks Rohith Sharma K S for more clarification. That sounds like a reasonable design to me and address all my concern for rolling upgrade. Look forward to new patch to address my other comments.

        Show
        djp Junping Du added a comment - Thanks Rohith Sharma K S for more clarification. That sounds like a reasonable design to me and address all my concern for rolling upgrade. Look forward to new patch to address my other comments.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Updated the patch fixing review comments

        Show
        rohithsharma Rohith Sharma K S added a comment - Updated the patch fixing review comments
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 10s Maven dependency ordering for branch
        +1 mvninstall 14m 21s trunk passed
        +1 compile 9m 45s trunk passed
        +1 checkstyle 0m 59s trunk passed
        +1 mvnsite 1m 37s trunk passed
        +1 findbugs 3m 5s trunk passed
        +1 javadoc 1m 32s trunk passed
              Patch Compile Tests
        0 mvndep 0m 11s Maven dependency ordering for patch
        +1 mvninstall 1m 26s the patch passed
        +1 compile 5m 38s the patch passed
        +1 javac 5m 38s the patch passed
        -0 checkstyle 1m 0s hadoop-yarn-project/hadoop-yarn: The patch generated 6 new + 231 unchanged - 11 fixed = 237 total (was 242)
        +1 mvnsite 1m 35s the patch passed
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 16s the patch passed
        +1 javadoc 1m 24s the patch passed
              Other Tests
        +1 unit 0m 33s hadoop-yarn-api in the patch passed.
        +1 unit 2m 30s hadoop-yarn-common in the patch passed.
        +1 unit 2m 17s hadoop-yarn-server-timeline-pluginstorage in the patch passed.
        +1 asflicense 0m 29s The patch does not generate ASF License warnings.
        60m 22s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6811
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880260/YARN-6811.02.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 8e14ba642a8e 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c5d256c
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16691/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/16691/artifact/patchprocess/whitespace-eol.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16691/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16691/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 14m 21s trunk passed +1 compile 9m 45s trunk passed +1 checkstyle 0m 59s trunk passed +1 mvnsite 1m 37s trunk passed +1 findbugs 3m 5s trunk passed +1 javadoc 1m 32s trunk passed       Patch Compile Tests 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 26s the patch passed +1 compile 5m 38s the patch passed +1 javac 5m 38s the patch passed -0 checkstyle 1m 0s hadoop-yarn-project/hadoop-yarn: The patch generated 6 new + 231 unchanged - 11 fixed = 237 total (was 242) +1 mvnsite 1m 35s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 16s the patch passed +1 javadoc 1m 24s the patch passed       Other Tests +1 unit 0m 33s hadoop-yarn-api in the patch passed. +1 unit 2m 30s hadoop-yarn-common in the patch passed. +1 unit 2m 17s hadoop-yarn-server-timeline-pluginstorage in the patch passed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 60m 22s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6811 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880260/YARN-6811.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 8e14ba642a8e 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c5d256c Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16691/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/16691/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16691/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/16691/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        djp Junping Du added a comment -

        Thanks for updating the patch and addressing my comments, Rohith Sharma K S. Latest patch LGTM.
        +1. Will commit it tomorrow if no other comments.

        Show
        djp Junping Du added a comment - Thanks for updating the patch and addressing my comments, Rohith Sharma K S . Latest patch LGTM. +1. Will commit it tomorrow if no other comments.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12122 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12122/)
        YARN-6811. [ATS1.5] All history logs should be kept under its own User (junping_du: rev f44b349b813508f0f6d99ca10bddba683dedf6c4)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/FileSystemTimelineWriter.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClientForATS1_5.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/TestEntityGroupFSTimelineStore.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12122 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12122/ ) YARN-6811 . [ATS1.5] All history logs should be kept under its own User (junping_du: rev f44b349b813508f0f6d99ca10bddba683dedf6c4) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/FileSystemTimelineWriter.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClientForATS1_5.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/TestEntityGroupFSTimelineStore.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        Hide
        djp Junping Du added a comment -

        I have commit the patch to trunk. For branch-2, my cherry-pick has several conflicts and the build still get failed even after I fix these conflicts. Rohith Sharma K S, can you upload a patch for branch-2?

        Show
        djp Junping Du added a comment - I have commit the patch to trunk. For branch-2, my cherry-pick has several conflicts and the build still get failed even after I fix these conflicts. Rohith Sharma K S , can you upload a patch for branch-2?
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        updated branch-2 patch!

        Show
        rohithsharma Rohith Sharma K S added a comment - updated branch-2 patch!
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 12m 27s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              branch-2 Compile Tests
        0 mvndep 0m 47s Maven dependency ordering for branch
        +1 mvninstall 6m 50s branch-2 passed
        +1 compile 1m 52s branch-2 passed with JDK v1.8.0_144
        +1 compile 2m 14s branch-2 passed with JDK v1.7.0_131
        +1 checkstyle 0m 43s branch-2 passed
        +1 mvnsite 1m 21s branch-2 passed
        +1 findbugs 2m 53s branch-2 passed
        +1 javadoc 0m 54s branch-2 passed with JDK v1.8.0_144
        +1 javadoc 1m 7s branch-2 passed with JDK v1.7.0_131
              Patch Compile Tests
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 7s the patch passed
        +1 compile 1m 49s the patch passed with JDK v1.8.0_144
        +1 javac 1m 49s the patch passed
        +1 compile 2m 15s the patch passed with JDK v1.7.0_131
        +1 javac 2m 15s the patch passed
        -0 checkstyle 0m 41s hadoop-yarn-project/hadoop-yarn: The patch generated 5 new + 231 unchanged - 10 fixed = 236 total (was 241)
        +1 mvnsite 1m 19s the patch passed
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 29s the patch passed
        +1 javadoc 0m 52s the patch passed with JDK v1.8.0_144
        +1 javadoc 1m 3s the patch passed with JDK v1.7.0_131
              Other Tests
        +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_131.
        +1 unit 2m 23s hadoop-yarn-common in the patch passed with JDK v1.7.0_131.
        +1 unit 1m 51s hadoop-yarn-server-timeline-pluginstorage in the patch passed with JDK v1.7.0_131.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        63m 36s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:5e40efe
        JIRA Issue YARN-6811
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880613/YARN-6811-branch-2.01.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux ca479071305e 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / d15bac1
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_144 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16735/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/16735/artifact/patchprocess/whitespace-eol.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16735/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16735/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 12m 27s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       branch-2 Compile Tests 0 mvndep 0m 47s Maven dependency ordering for branch +1 mvninstall 6m 50s branch-2 passed +1 compile 1m 52s branch-2 passed with JDK v1.8.0_144 +1 compile 2m 14s branch-2 passed with JDK v1.7.0_131 +1 checkstyle 0m 43s branch-2 passed +1 mvnsite 1m 21s branch-2 passed +1 findbugs 2m 53s branch-2 passed +1 javadoc 0m 54s branch-2 passed with JDK v1.8.0_144 +1 javadoc 1m 7s branch-2 passed with JDK v1.7.0_131       Patch Compile Tests 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 7s the patch passed +1 compile 1m 49s the patch passed with JDK v1.8.0_144 +1 javac 1m 49s the patch passed +1 compile 2m 15s the patch passed with JDK v1.7.0_131 +1 javac 2m 15s the patch passed -0 checkstyle 0m 41s hadoop-yarn-project/hadoop-yarn: The patch generated 5 new + 231 unchanged - 10 fixed = 236 total (was 241) +1 mvnsite 1m 19s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 29s the patch passed +1 javadoc 0m 52s the patch passed with JDK v1.8.0_144 +1 javadoc 1m 3s the patch passed with JDK v1.7.0_131       Other Tests +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_131. +1 unit 2m 23s hadoop-yarn-common in the patch passed with JDK v1.7.0_131. +1 unit 1m 51s hadoop-yarn-server-timeline-pluginstorage in the patch passed with JDK v1.7.0_131. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 63m 36s Subsystem Report/Notes Docker Image:yetus/hadoop:5e40efe JIRA Issue YARN-6811 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880613/YARN-6811-branch-2.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux ca479071305e 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / d15bac1 Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_144 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16735/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/16735/artifact/patchprocess/whitespace-eol.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16735/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/16735/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        cc :/ Junping Du Updated the patch for branch-2

        Show
        rohithsharma Rohith Sharma K S added a comment - cc :/ Junping Du Updated the patch for branch-2
        Hide
        djp Junping Du added a comment -

        Thanks Rohith Sharma K S. I have commit the patch to trunk and branch-2.

        Show
        djp Junping Du added a comment - Thanks Rohith Sharma K S . I have commit the patch to trunk and branch-2.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        I don't understand why this patch is needed. Can we simply not change the active-dir configuration to have a $user.name path?

        Also, why doesn't the same logic apply for the done-directory? I don't see corresponding changes.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - I don't understand why this patch is needed. Can we simply not change the active-dir configuration to have a $user.name path? Also, why doesn't the same logic apply for the done-directory? I don't see corresponding changes.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Can we simply not change the active-dir configuration to have a $user.name path?

        This configuration is referred by TimelineSever to retrieve unfinished entities via REST service. So this can't be changed directly in timelineclient. This active base path should be same in timeline server and timeline client as well.

        why doesn't the same logic apply for the done-directory?

        DONE directory is accessed only by TimelineServer. So, it is not required to create user directory under done directory path!

        Show
        rohithsharma Rohith Sharma K S added a comment - Can we simply not change the active-dir configuration to have a $user.name path? This configuration is referred by TimelineSever to retrieve unfinished entities via REST service. So this can't be changed directly in timelineclient. This active base path should be same in timeline server and timeline client as well. why doesn't the same logic apply for the done-directory? DONE directory is accessed only by TimelineServer. So, it is not required to create user directory under done directory path!

          People

          • Assignee:
            rohithsharma Rohith Sharma K S
            Reporter:
            rohithsharma Rohith Sharma K S
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development