Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3038

job history server not starting because conf() missing HsController

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0, 0.24.0
    • Component/s: mrv2
    • Labels:
      None

      Description

      Exception starting history server.

      Sep 19, 2011 6:51:53 PM com.google.inject.MessageProcessor visit
      INFO: An exception was caught and reported. Message: org.apache.hadoop.yarn.webapp.WebAppException: conf() not found in class org.apache.hadoop.mapreduce.v2.hs.webapp.HsController org.apache.hadoop.yarn.webapp.WebAppException: conf() not found in class org.apache.hadoop.mapreduce.v2.hs.webapp.HsController
      at org.apache.hadoop.yarn.webapp.Router.addController(Router.java:107)
      at org.apache.hadoop.yarn.webapp.Router.add(Router.java:83)
      at org.apache.hadoop.yarn.webapp.WebApp.route(WebApp.java:140)
      at org.apache.hadoop.yarn.webapp.WebApp.route(WebApp.java:146)
      at org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebApp.setup(HsWebApp.java:42)
      at org.apache.hadoop.yarn.webapp.WebApp.configureServlets(WebApp.java:121)
      at com.google.inject.servlet.ServletModule.configure(ServletModule.java:45)
      at com.google.inject.AbstractModule.configure(AbstractModule.java:59)
      at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:223)
      at com.google.inject.spi.Elements.getElements(Elements.java:101)
      at com.google.inject.InjectorShell$Builder.build(InjectorShell.java:135)
      at com.google.inject.InjectorBuilder.build(InjectorBuilder.java:102)
      at com.google.inject.Guice.createInjector(Guice.java:92)
      at com.google.inject.Guice.createInjector(Guice.java:69)
      at com.google.inject.Guice.createInjector(Guice.java:59)
      at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:166)
      at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService.initializeWebApp(HistoryClientService.java:138)
      at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService.start(HistoryClientService.java:109)
      at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
      at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.main(JobHistoryServer.java:83)

      1. MAPREDUCE-3038.patch
        3 kB
        Jeffrey Naisbitt
      2. MAPREDUCE-3038.patch
        3 kB
        Jeffrey Naisbitt
      3. MAPREDUCE-3038.patch
        3 kB
        Jeffrey Naisbitt
      4. MAPREDUCE-3038.patch
        3 kB
        Jeffrey Naisbitt
      5. MAPREDUCE-3038.java
        1.0 kB
        Thomas Graves

        Activity

        Hide
        Thomas Graves added a comment -

        it looks like the conf() function was remove from HsController but HsWebApp is still expecting it there.

        Show
        Thomas Graves added a comment - it looks like the conf() function was remove from HsController but HsWebApp is still expecting it there.
        Hide
        Thomas Graves added a comment -

        putting the conf routine back allows history server to start.

        Show
        Thomas Graves added a comment - putting the conf routine back allows history server to start.
        Hide
        Thomas Graves added a comment -

        It looks like hscontroller extends appcontroller and conf() was added there so I think the intention was for it to pick up that version.

        Show
        Thomas Graves added a comment - It looks like hscontroller extends appcontroller and conf() was added there so I think the intention was for it to pick up that version.
        Hide
        Jeffrey Naisbitt added a comment -

        It looks to me like Router.java is using reflection incorrectly and not checking whether the method was inherited. It's manually throwing that exception if the method is not found in that class.

        Show
        Jeffrey Naisbitt added a comment - It looks to me like Router.java is using reflection incorrectly and not checking whether the method was inherited. It's manually throwing that exception if the method is not found in that class.
        Hide
        Jeffrey Naisbitt added a comment -

        It looks like I caused this by removing the method. I think the correct solution is to fix the addController method in Router.java so it does the reflection correctly though - rather than adding the method back in to the HsController since it already inherits it. addController currently does not look at inherited methods. Additionally, it seems like we could just do a getMethod and get rid of the loop through all the methods to get better speed/efficiency/clarity.

        Show
        Jeffrey Naisbitt added a comment - It looks like I caused this by removing the method. I think the correct solution is to fix the addController method in Router.java so it does the reflection correctly though - rather than adding the method back in to the HsController since it already inherits it. addController currently does not look at inherited methods. Additionally, it seems like we could just do a getMethod and get rid of the loop through all the methods to get better speed/efficiency/clarity.
        Hide
        Jeffrey Naisbitt added a comment -

        I'm still testing this a bit. The jobs are passing, and the historyserver is running fine, but I'm seeing this log message repeatedly:
        11/09/19 20:00:26 INFO mapred.ClientCache: HistoryServer is not configured.

        Show
        Jeffrey Naisbitt added a comment - I'm still testing this a bit. The jobs are passing, and the historyserver is running fine, but I'm seeing this log message repeatedly: 11/09/19 20:00:26 INFO mapred.ClientCache: HistoryServer is not configured.
        Hide
        Jeffrey Naisbitt added a comment -

        It looks like the INFO message is caused by the change with MAPREDUCE-2925 and unrelated to this patch.

        Show
        Jeffrey Naisbitt added a comment - It looks like the INFO message is caused by the change with MAPREDUCE-2925 and unrelated to this patch.
        Hide
        Jeffrey Naisbitt added a comment -

        Reattaching the patch to check the radio box for granting license.

        Show
        Jeffrey Naisbitt added a comment - Reattaching the patch to check the radio box for granting license.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12495134/MAPREDUCE-3038.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/788//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/788//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12495134/MAPREDUCE-3038.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/788//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/788//console This message is automatically generated.
        Hide
        Mahadev konar added a comment -

        @jeffrey,

        the code seems to be catching NullPointerException() and throwing a new Exception. Why are we catching a NullPointerException()? Shouldnt this be taken care of in the code itself?

        Show
        Mahadev konar added a comment - @jeffrey, the code seems to be catching NullPointerException() and throwing a new Exception. Why are we catching a NullPointerException()? Shouldnt this be taken care of in the code itself?
        Hide
        Jeffrey Naisbitt added a comment -

        Removed NullPointerException catch and fixed a typo in the comment.

        Show
        Jeffrey Naisbitt added a comment - Removed NullPointerException catch and fixed a typo in the comment.
        Hide
        Jeffrey Naisbitt added a comment -

        Reattaching again with the grant license

        Show
        Jeffrey Naisbitt added a comment - Reattaching again with the grant license
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12495155/MAPREDUCE-3038.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/792//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/792//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12495155/MAPREDUCE-3038.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/792//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/792//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12495155/MAPREDUCE-3038.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/793//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/793//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12495155/MAPREDUCE-3038.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/793//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/793//console This message is automatically generated.
        Hide
        Mahadev konar added a comment -

        I just commmitted this to trunk and 0.23. thanks Jeff!

        Show
        Mahadev konar added a comment - I just commmitted this to trunk and 0.23. thanks Jeff!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #915 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/915/)
        MAPREDUCE-3038. job history server not starting because conf() missing HsController (Jeffrey Naisbitt via mahadev)

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172875
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/Router.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #915 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/915/ ) MAPREDUCE-3038 . job history server not starting because conf() missing HsController (Jeffrey Naisbitt via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172875 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/Router.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #992 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/992/)
        MAPREDUCE-3038. job history server not starting because conf() missing HsController (Jeffrey Naisbitt via mahadev)

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172875
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/Router.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #992 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/992/ ) MAPREDUCE-3038 . job history server not starting because conf() missing HsController (Jeffrey Naisbitt via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172875 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/Router.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #932 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/932/)
        MAPREDUCE-3038. job history server not starting because conf() missing HsController (Jeffrey Naisbitt via mahadev)

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172875
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/Router.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #932 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/932/ ) MAPREDUCE-3038 . job history server not starting because conf() missing HsController (Jeffrey Naisbitt via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172875 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/Router.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #806 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/806/)
        MAPREDUCE-3038. job history server not starting because conf() missing HsController (Jeffrey Naisbitt via mahadev)

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172875
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/Router.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #806 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/806/ ) MAPREDUCE-3038 . job history server not starting because conf() missing HsController (Jeffrey Naisbitt via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172875 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/Router.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #836 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/836/)
        MAPREDUCE-3038. job history server not starting because conf() missing HsController (Jeffrey Naisbitt via mahadev)

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172875
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/Router.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #836 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/836/ ) MAPREDUCE-3038 . job history server not starting because conf() missing HsController (Jeffrey Naisbitt via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172875 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/Router.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #12 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/12/)
        MAPREDUCE-3038. job history server not starting because conf() missing HsController (Jeffrey Naisbitt via mahadev) - merging r1172875 from trunk

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172879
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/Router.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #12 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/12/ ) MAPREDUCE-3038 . job history server not starting because conf() missing HsController (Jeffrey Naisbitt via mahadev) - merging r1172875 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172879 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/Router.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #14 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/14/)
        MAPREDUCE-3038. job history server not starting because conf() missing HsController (Jeffrey Naisbitt via mahadev) - merging r1172875 from trunk

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172879
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/Router.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #14 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/14/ ) MAPREDUCE-3038 . job history server not starting because conf() missing HsController (Jeffrey Naisbitt via mahadev) - merging r1172875 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172879 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/Router.java

          People

          • Assignee:
            Jeffrey Naisbitt
            Reporter:
            Thomas Graves
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development