Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4904 YARN Log tooling enhancements
  3. YARN-4990

Re-direction of a particular log file within in a container in NM UI does not redirect properly to Log Server ( history ) on container completion

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The NM does the redirection to the history server correctly. However if the user is viewing or has a link to a particular specific file, the redirect ends up going to the top level page for the container and not redirecting to the specific file. Additionally, the start param to show logs from the offset 0 also goes missing.

      1. YARN-4990.2.patch
        9 kB
        Xuan Gong
      2. YARN-4990.1.patch
        2 kB
        Xuan Gong

        Issue Links

          Activity

          Hide
          jlowe Jason Lowe added a comment -

          Thanks, Xuan! I committed this to branch-2.8 as well.

          Show
          jlowe Jason Lowe added a comment - Thanks, Xuan! I committed this to branch-2.8 as well.
          Hide
          djp Junping Du added a comment -

          I am fine with it.

          Show
          djp Junping Du added a comment - I am fine with it.
          Hide
          jlowe Jason Lowe added a comment -

          This would be a nice fix to get into 2.8 and seems to be low risk. Any objections?

          Show
          jlowe Jason Lowe added a comment - This would be a nice fix to get into 2.8 and seems to be low risk. Any objections?
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11027 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11027/)
          YARN-4990. Re-direction of a particular log file within in a container (junping_du: rev 736f54b727c3f0ecc8fb9a594f2281c240c89cb8)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMWebAppFilter.java
          • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebFilter.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11027 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11027/ ) YARN-4990 . Re-direction of a particular log file within in a container (junping_du: rev 736f54b727c3f0ecc8fb9a594f2281c240c89cb8) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMWebAppFilter.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebFilter.java
          Hide
          djp Junping Du added a comment -

          I have commit the patch to trunk and branch-2. Thanks Hitesh Shah for reporting the issue and Xuan Gong to fix it!

          Show
          djp Junping Du added a comment - I have commit the patch to trunk and branch-2. Thanks Hitesh Shah for reporting the issue and Xuan Gong to fix it!
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 13m 3s trunk passed
          +1 compile 0m 27s trunk passed
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 25s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 39s trunk passed
          +1 javadoc 0m 17s trunk passed
          +1 mvninstall 0m 22s the patch passed
          +1 compile 0m 24s the patch passed
          +1 javac 0m 24s the patch passed
          -0 checkstyle 0m 13s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 3 new + 2 unchanged - 0 fixed = 5 total (was 2)
          +1 mvnsite 0m 22s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 44s the patch passed
          +1 javadoc 0m 15s the patch passed
          +1 unit 12m 56s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          32m 34s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-4990
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844028/YARN-4990.2.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 2ad78b092ec5 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8b042bc
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14427/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14427/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14427/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 13m 3s trunk passed +1 compile 0m 27s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 25s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 39s trunk passed +1 javadoc 0m 17s trunk passed +1 mvninstall 0m 22s the patch passed +1 compile 0m 24s the patch passed +1 javac 0m 24s the patch passed -0 checkstyle 0m 13s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 3 new + 2 unchanged - 0 fixed = 5 total (was 2) +1 mvnsite 0m 22s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 44s the patch passed +1 javadoc 0m 15s the patch passed +1 unit 12m 56s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 32m 34s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-4990 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844028/YARN-4990.2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2ad78b092ec5 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8b042bc Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14427/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14427/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14427/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 13m 18s trunk passed
          +1 compile 0m 27s trunk passed
          +1 checkstyle 0m 16s trunk passed
          +1 mvnsite 0m 25s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 0m 42s trunk passed
          +1 javadoc 0m 18s trunk passed
          +1 mvninstall 0m 22s the patch passed
          +1 compile 0m 24s the patch passed
          +1 javac 0m 24s the patch passed
          -0 checkstyle 0m 14s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 3 new + 2 unchanged - 0 fixed = 5 total (was 2)
          +1 mvnsite 0m 23s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 44s the patch passed
          +1 javadoc 0m 14s the patch passed
          +1 unit 12m 50s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          32m 49s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-4990
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844028/YARN-4990.2.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c1da71772947 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8b042bc
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14426/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14426/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14426/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 13m 18s trunk passed +1 compile 0m 27s trunk passed +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 25s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 42s trunk passed +1 javadoc 0m 18s trunk passed +1 mvninstall 0m 22s the patch passed +1 compile 0m 24s the patch passed +1 javac 0m 24s the patch passed -0 checkstyle 0m 14s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 3 new + 2 unchanged - 0 fixed = 5 total (was 2) +1 mvnsite 0m 23s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 44s the patch passed +1 javadoc 0m 14s the patch passed +1 unit 12m 50s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 32m 49s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-4990 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844028/YARN-4990.2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c1da71772947 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8b042bc Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14426/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14426/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14426/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          djp Junping Du added a comment -

          v2 patch LGTM. Verified locally that without the fix, the test will get failed.
          Looks like Jenkins doesn't kick off for some reason, will manually do. +1 pending on Jenkins report.

          Show
          djp Junping Du added a comment - v2 patch LGTM. Verified locally that without the fix, the test will get failed. Looks like Jenkins doesn't kick off for some reason, will manually do. +1 pending on Jenkins report.
          Hide
          xgong Xuan Gong added a comment -

          Thanks for the review. Junping Du

          Attached a new patch which added a testcase

          Show
          xgong Xuan Gong added a comment - Thanks for the review. Junping Du Attached a new patch which added a testcase
          Hide
          djp Junping Du added a comment -

          Thanks Xuan Gong for uploading a patch to fix this. The patch looks good in general. However, may be we can add some sanity UT here? just like what we do in TestAMFilter, etc.

          Show
          djp Junping Du added a comment - Thanks Xuan Gong for uploading a patch to fix this. The patch looks good in general. However, may be we can add some sanity UT here? just like what we do in TestAMFilter, etc.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 15m 31s trunk passed
          +1 compile 0m 32s trunk passed
          +1 checkstyle 0m 20s trunk passed
          +1 mvnsite 0m 31s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 0m 51s trunk passed
          +1 javadoc 0m 20s trunk passed
          +1 mvninstall 0m 26s the patch passed
          +1 compile 0m 26s the patch passed
          +1 javac 0m 26s the patch passed
          +1 checkstyle 0m 15s the patch passed
          +1 mvnsite 0m 24s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 54s the patch passed
          +1 javadoc 0m 18s the patch passed
          +1 unit 13m 29s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          36m 38s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-4990
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843686/YARN-4990.1.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0392e8cc1bde 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / fcbe152
          Default Java 1.8.0_111
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14352/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14352/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 15m 31s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 31s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 0m 51s trunk passed +1 javadoc 0m 20s trunk passed +1 mvninstall 0m 26s the patch passed +1 compile 0m 26s the patch passed +1 javac 0m 26s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 24s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 54s the patch passed +1 javadoc 0m 18s the patch passed +1 unit 13m 29s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 36m 38s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-4990 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843686/YARN-4990.1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0392e8cc1bde 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / fcbe152 Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14352/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14352/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xgong Xuan Gong added a comment -

          It is not easy to add a test case for it. But did a manual test.

          http://xuanmacbook-pro.home:8042/node/containerlogs/container_1481936892133_0005_01_000001/xuan
          

          would redirect to

          http://localhost:8188/applicationhistory/logs/xuanmacbook-pro.home:53565/container_1481936892133_0005_01_000001/container_1481936892133_0005_01_000001/xuan
          
          http://xuanmacbook-pro.home:8042/node/containerlogs/container_1481936892133_0005_01_000001/xuan/syslog/?start=10
          

          would redirect to

          http://localhost:8188/applicationhistory/logs/xuanmacbook-pro.home:53565/container_1481936892133_0005_01_000001/container_1481936892133_0005_01_000001/xuan/syslog?start=10
          

          Also verified that we do have the correct offset

          Show
          xgong Xuan Gong added a comment - It is not easy to add a test case for it. But did a manual test. http: //xuanmacbook-pro.home:8042/node/containerlogs/container_1481936892133_0005_01_000001/xuan would redirect to http: //localhost:8188/applicationhistory/logs/xuanmacbook-pro.home:53565/container_1481936892133_0005_01_000001/container_1481936892133_0005_01_000001/xuan http: //xuanmacbook-pro.home:8042/node/containerlogs/container_1481936892133_0005_01_000001/xuan/syslog/?start=10 would redirect to http: //localhost:8188/applicationhistory/logs/xuanmacbook-pro.home:53565/container_1481936892133_0005_01_000001/container_1481936892133_0005_01_000001/xuan/syslog?start=10 Also verified that we do have the correct offset
          Hide
          xgong Xuan Gong added a comment -

          The issue is that we did not parse/append containerLogType(logFile name)when we generate the redirect log link for the container logs.

          Attached a trivial patch to fix it

          Show
          xgong Xuan Gong added a comment - The issue is that we did not parse/append containerLogType(logFile name)when we generate the redirect log link for the container logs. Attached a trivial patch to fix it

            People

            • Assignee:
              xgong Xuan Gong
              Reporter:
              hitesh Hitesh Shah
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development