Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2342

Set yarn.log.server.url to point to JH server

    Details

      Description

      Bigtop sets up a MapReduce JobHistoryServer, but YARN Nodemanager is unaware of the existence of the MapReduce JobHistoryServer when a MapReduce job completes. The NodeManager UI uses the undocumented yarn.log.server.url yarn-site.xml property to point back to the JobHistoryServer Web UI for completed jobs.

        Issue Links

          Activity

          Hide
          oflebbe Olaf Flebbe added a comment -

          Are you sure that this works?

          I found no reference to yarn.log.server.url in 2.7.x hadoop code.

          Show
          oflebbe Olaf Flebbe added a comment - Are you sure that this works? I found no reference to yarn.log.server.url in 2.7.x hadoop code .
          Hide
          oflebbe Olaf Flebbe added a comment -

          A forget it. I found it.

          Show
          oflebbe Olaf Flebbe added a comment - A forget it. I found it.
          Hide
          jonathak Jonathan Kelly added a comment -

          Yep, it's sometimes difficult to find config variables in Hadoop source because they use so many prefix constants.

          In this case, just to let everybody else know, it can be found here: https://github.com/apache/hadoop/blob/release-2.7.2-RC2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java#L813

          Show
          jonathak Jonathan Kelly added a comment - Yep, it's sometimes difficult to find config variables in Hadoop source because they use so many prefix constants. In this case, just to let everybody else know, it can be found here: https://github.com/apache/hadoop/blob/release-2.7.2-RC2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java#L813
          Hide
          oflebbe Olaf Flebbe added a comment -

          The patch LGTM. And I know that there are problems in the yarn webui with finished jobs... And similar problems in hue. I thought it was triggered by tez. I like to test your patch first, hopefully I can do the test until tomorrow.

          Show
          oflebbe Olaf Flebbe added a comment - The patch LGTM. And I know that there are problems in the yarn webui with finished jobs... And similar problems in hue. I thought it was triggered by tez. I like to test your patch first, hopefully I can do the test until tomorrow.
          Hide
          oflebbe Olaf Flebbe added a comment -

          I tested it, and it worked for me if I am tweaking the configuration a bit.

          Can you confirm, that one needs the hadoop-mapreduce-historyserver and the hadoop-yarn-proxyserver running?

          Can you confirm that you have the mapreduce.jobhistory.intermediate-done-dir and mapreduce.jobhistory.done-dir properties set, at least if yarn.app.mapreduce.am.staging-dir is not set to /user ?

          Show
          oflebbe Olaf Flebbe added a comment - I tested it, and it worked for me if I am tweaking the configuration a bit. Can you confirm, that one needs the hadoop-mapreduce-historyserver and the hadoop-yarn-proxyserver running? Can you confirm that you have the mapreduce.jobhistory.intermediate-done-dir and mapreduce.jobhistory.done-dir properties set, at least if yarn.app.mapreduce.am.staging-dir is not set to /user ?
          Hide
          oflebbe Olaf Flebbe added a comment -

          committed.

          Show
          oflebbe Olaf Flebbe added a comment - committed.

            People

            • Assignee:
              aartokhy Aaron Tokhy
              Reporter:
              aartokhy Aaron Tokhy
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development