Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6266

Job#getTrackingURL should consistently return a proper URL

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      When a job is running, Job#getTrackingURL returns a proper URL like:

      http://<RM_IP>:8088/proxy/application_1424910897258_0004/

      Once a job is finished and the job has moved to the JHS, then Job#getTrackingURL returns a URL without the protocol like:

      <JHS_IP>:19888/jobhistory/job/job_1424910897258_0004

      1. MAPREDUCE-6266.001.patch
        1 kB
        Ray Chiang
      2. MAPREDUCE-6266.002.patch
        2 kB
        Ray Chiang
      3. MAPREDUCE-6266.003.patch
        1 kB
        Ray Chiang

        Activity

        Hide
        rchiang Ray Chiang added a comment -
        • Call MRWebAppUtil#getApplicationWebURLOnJHSWithScheme() instead of MRWebAppUtil#getApplicationWebURLOnJHSWithoutScheme()
        • Retry method call 3 times in order to lessen chance of returning an empty string
        Show
        rchiang Ray Chiang added a comment - Call MRWebAppUtil#getApplicationWebURLOnJHSWithScheme() instead of MRWebAppUtil#getApplicationWebURLOnJHSWithoutScheme() Retry method call 3 times in order to lessen chance of returning an empty string
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12704287/MAPREDUCE-6266.001.patch
        against trunk revision 8212877.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5284//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5284//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12704287/MAPREDUCE-6266.001.patch against trunk revision 8212877. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5284//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5284//console This message is automatically generated.
        Hide
        rchiang Ray Chiang added a comment -

        Add minor unit test update

        Show
        rchiang Ray Chiang added a comment - Add minor unit test update
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12704372/MAPREDUCE-6266.002.patch
        against trunk revision 387f271.

        -1 @author. The patch appears to contain @author tags which the Hadoop community has agreed to not allow in code contributions.

        +1 tests included. The patch appears to include new or modified test files.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5286//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12704372/MAPREDUCE-6266.002.patch against trunk revision 387f271. -1 @author . The patch appears to contain @author tags which the Hadoop community has agreed to not allow in code contributions. +1 tests included . The patch appears to include new or modified test files. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5286//console This message is automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12704372/MAPREDUCE-6266.002.patch
        against trunk revision 387f271.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs:

        org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryEntities

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5287//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5287//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12704372/MAPREDUCE-6266.002.patch against trunk revision 387f271. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs: org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryEntities Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5287//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5287//console This message is automatically generated.
        Hide
        djp Junping Du added a comment -

        Thanks Ray Chiang for reporting an issue and working on a patch here. One question: why we add a loop here in getting URL?

        Show
        djp Junping Du added a comment - Thanks Ray Chiang for reporting an issue and working on a patch here. One question: why we add a loop here in getting URL?
        Hide
        rchiang Ray Chiang added a comment -

        The call to Job.getTrackingURL() occasionally fails quietly (due to the empty catch statement and probably network hiccups). This will put in a limited retry to make the occurrence less frequent. I can take out that part if it's undesirable.

        Show
        rchiang Ray Chiang added a comment - The call to Job.getTrackingURL() occasionally fails quietly (due to the empty catch statement and probably network hiccups). This will put in a limited retry to make the occurrence less frequent. I can take out that part if it's undesirable.
        Hide
        djp Junping Du added a comment -

        Can we dig out the reason why Job.getTrackingURL() occasionally fails? It could be some bug hidden there. If so, it would be great if we can fix it also in this JIRA.

        Show
        djp Junping Du added a comment - Can we dig out the reason why Job.getTrackingURL() occasionally fails? It could be some bug hidden there. If so, it would be great if we can fix it also in this JIRA.
        Hide
        rchiang Ray Chiang added a comment -

        RE: failed unit test

        Interesting, testing on my standalone machine gives me one set of results on the TestJobHistoryEntities unit test. Testing on Jenkins seems to give a valid URL.

        Show
        rchiang Ray Chiang added a comment - RE: failed unit test Interesting, testing on my standalone machine gives me one set of results on the TestJobHistoryEntities unit test. Testing on Jenkins seems to give a valid URL.
        Hide
        rchiang Ray Chiang added a comment -

        Okay, I'll take a look and see if I can replicate the issue.

        Show
        rchiang Ray Chiang added a comment - Okay, I'll take a look and see if I can replicate the issue.
        Hide
        rchiang Ray Chiang added a comment -

        I couldn't replicate UnknownHostExceptions normally.

        • Remove retry loop. Add exception output to make it possible to debug later.
        • Remove modifications to unit test, since it results in different results on the command line and Jenkins.
        Show
        rchiang Ray Chiang added a comment - I couldn't replicate UnknownHostExceptions normally. Remove retry loop. Add exception output to make it possible to debug later. Remove modifications to unit test, since it results in different results on the command line and Jenkins.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12704871/MAPREDUCE-6266.003.patch
        against trunk revision 685dbaf.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5295//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5295//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12704871/MAPREDUCE-6266.003.patch against trunk revision 685dbaf. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5295//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5295//console This message is automatically generated.
        Hide
        rchiang Ray Chiang added a comment -

        RE: No unit tests

        Removed change since it has different values in Jenkins vs standalone

        Show
        rchiang Ray Chiang added a comment - RE: No unit tests Removed change since it has different values in Jenkins vs standalone
        Hide
        rchiang Ray Chiang added a comment -

        Junping Du, I don't see an obvious bug, but I changed the code to print out the exception if it occurs.

        Show
        rchiang Ray Chiang added a comment - Junping Du , I don't see an obvious bug, but I changed the code to print out the exception if it occurs.
        Hide
        rkanter Robert Kanter added a comment -

        The 003 patch looks good to me. +1

        Junping Du, any additional comments?

        Show
        rkanter Robert Kanter added a comment - The 003 patch looks good to me. +1 Junping Du , any additional comments?
        Hide
        rkanter Robert Kanter added a comment -

        Thanks Ray. Committed to trunk and branch-2!

        Show
        rkanter Robert Kanter added a comment - Thanks Ray. Committed to trunk and branch-2!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7553 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7553/)
        MAPREDUCE-6266. Job#getTrackingURL should consistently return a proper URL (rchiang via rkanter) (rkanter: rev c4986b2d00d327f18d0c7e2f8805b69a4e07a19b)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
        • hadoop-mapreduce-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7553 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7553/ ) MAPREDUCE-6266 . Job#getTrackingURL should consistently return a proper URL (rchiang via rkanter) (rkanter: rev c4986b2d00d327f18d0c7e2f8805b69a4e07a19b) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java hadoop-mapreduce-project/CHANGES.txt
        Hide
        rchiang Ray Chiang added a comment -

        Thanks for the review and the commit!

        Show
        rchiang Ray Chiang added a comment - Thanks for the review and the commit!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #893 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/893/)
        MAPREDUCE-6266. Job#getTrackingURL should consistently return a proper URL (rchiang via rkanter) (rkanter: rev c4986b2d00d327f18d0c7e2f8805b69a4e07a19b)

        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #893 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/893/ ) MAPREDUCE-6266 . Job#getTrackingURL should consistently return a proper URL (rchiang via rkanter) (rkanter: rev c4986b2d00d327f18d0c7e2f8805b69a4e07a19b) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #159 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/159/)
        MAPREDUCE-6266. Job#getTrackingURL should consistently return a proper URL (rchiang via rkanter) (rkanter: rev c4986b2d00d327f18d0c7e2f8805b69a4e07a19b)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
        • hadoop-mapreduce-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #159 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/159/ ) MAPREDUCE-6266 . Job#getTrackingURL should consistently return a proper URL (rchiang via rkanter) (rkanter: rev c4986b2d00d327f18d0c7e2f8805b69a4e07a19b) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java hadoop-mapreduce-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2091 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2091/)
        MAPREDUCE-6266. Job#getTrackingURL should consistently return a proper URL (rchiang via rkanter) (rkanter: rev c4986b2d00d327f18d0c7e2f8805b69a4e07a19b)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
        • hadoop-mapreduce-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2091 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2091/ ) MAPREDUCE-6266 . Job#getTrackingURL should consistently return a proper URL (rchiang via rkanter) (rkanter: rev c4986b2d00d327f18d0c7e2f8805b69a4e07a19b) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java hadoop-mapreduce-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #150 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/150/)
        MAPREDUCE-6266. Job#getTrackingURL should consistently return a proper URL (rchiang via rkanter) (rkanter: rev c4986b2d00d327f18d0c7e2f8805b69a4e07a19b)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
        • hadoop-mapreduce-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #150 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/150/ ) MAPREDUCE-6266 . Job#getTrackingURL should consistently return a proper URL (rchiang via rkanter) (rkanter: rev c4986b2d00d327f18d0c7e2f8805b69a4e07a19b) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java hadoop-mapreduce-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #160 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/160/)
        MAPREDUCE-6266. Job#getTrackingURL should consistently return a proper URL (rchiang via rkanter) (rkanter: rev c4986b2d00d327f18d0c7e2f8805b69a4e07a19b)

        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #160 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/160/ ) MAPREDUCE-6266 . Job#getTrackingURL should consistently return a proper URL (rchiang via rkanter) (rkanter: rev c4986b2d00d327f18d0c7e2f8805b69a4e07a19b) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2109 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2109/)
        MAPREDUCE-6266. Job#getTrackingURL should consistently return a proper URL (rchiang via rkanter) (rkanter: rev c4986b2d00d327f18d0c7e2f8805b69a4e07a19b)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
        • hadoop-mapreduce-project/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2109 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2109/ ) MAPREDUCE-6266 . Job#getTrackingURL should consistently return a proper URL (rchiang via rkanter) (rkanter: rev c4986b2d00d327f18d0c7e2f8805b69a4e07a19b) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java hadoop-mapreduce-project/CHANGES.txt

          People

          • Assignee:
            rchiang Ray Chiang
            Reporter:
            rchiang Ray Chiang
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development