Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2307

Exception thrown in Jobtracker logs, when the Scheduler configured is FairScheduler.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: contrib/fair-share
    • Labels:
      None

      Description

      If we try to start the job tracker with fair scheduler using the default configuration, It is giving the below exception.

       
      2010-07-03 10:18:27,142 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001: starting
      2010-07-03 10:18:27,143 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 9001: starting
      2010-07-03 10:18:27,143 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9001: starting
      2010-07-03 10:18:27,143 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9001: starting
      2010-07-03 10:18:27,143 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 9001: starting
      2010-07-03 10:18:27,143 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 9001: starting
      2010-07-03 10:18:27,143 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 9001: starting
      2010-07-03 10:18:27,143 INFO org.apache.hadoop.mapred.JobTracker: Starting RUNNING
      2010-07-03 10:18:27,143 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 9001: starting
      2010-07-03 10:18:28,037 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/linux172.site
      2010-07-03 10:18:28,090 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/linux177.site
      2010-07-03 10:18:40,074 ERROR org.apache.hadoop.mapred.PoolManager: Failed to reload allocations file - will use existing allocations.
      java.lang.NullPointerException
      at java.io.File.<init>(File.java:222)
      at org.apache.hadoop.mapred.PoolManager.reloadAllocsIfNecessary(PoolManager.java:127)
      at org.apache.hadoop.mapred.FairScheduler.assignTasks(FairScheduler.java:234)
      at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2785)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:513)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:984)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:980)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:978)
      

        Activity

        Hide
        Devaraj K added a comment -

        Added null check for allocFile instead of catching and logging NullPointerException when the allocFile is null.

        Show
        Devaraj K added a comment - Added null check for allocFile instead of catching and logging NullPointerException when the allocFile is null.
        Hide
        Todd Lipcon added a comment -

        Looks good to me, +1

        Show
        Todd Lipcon added a comment - Looks good to me, +1
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12472960/MAPREDUCE-2307.patch
        against trunk revision 1079072.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/132//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/132//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/132//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12472960/MAPREDUCE-2307.patch against trunk revision 1079072. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/132//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/132//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/132//console This message is automatically generated.
        Hide
        Devaraj K added a comment -

        Contrib unit tests failures are not because of the patch, those have been failing for the past 28 builds (Since #105).

        Tests are not required for this patch because null check is added instead of logging NullPointerException.

        Show
        Devaraj K added a comment - Contrib unit tests failures are not because of the patch, those have been failing for the past 28 builds (Since #105). Tests are not required for this patch because null check is added instead of logging NullPointerException.
        Hide
        Matei Zaharia added a comment -

        The problem seems to be because there's no longer a fair-scheduler.xml in the conf directory by default. I believe it was created from a fair-scheduler.xml.template in the past. Rather than adding this null check and ignoring the error silently, it would be better if you added back a template so that that file gets built, if this is possible after the project split.

        Show
        Matei Zaharia added a comment - The problem seems to be because there's no longer a fair-scheduler.xml in the conf directory by default. I believe it was created from a fair-scheduler.xml.template in the past. Rather than adding this null check and ignoring the error silently, it would be better if you added back a template so that that file gets built, if this is possible after the project split.
        Hide
        Devaraj K added a comment -

        If the allocFile is null, it is already logging the message in the initialize() method. When the reloadAllocsIfNecessary() invokes with allocFile is null, it encounters NullPointerException, catching and logging that NullPointerException may not be good.

        Show
        Devaraj K added a comment - If the allocFile is null, it is already logging the message in the initialize() method. When the reloadAllocsIfNecessary() invokes with allocFile is null, it encounters NullPointerException, catching and logging that NullPointerException may not be good.
        Hide
        Matei Zaharia added a comment -

        Oh OK, got it. This looks good then. I will commit it if there are no other comments.

        Show
        Matei Zaharia added a comment - Oh OK, got it. This looks good then. I will commit it if there are no other comments.
        Hide
        Matei Zaharia added a comment -

        I've committed this. Thanks Devaraj!

        Show
        Matei Zaharia added a comment - I've committed this. Thanks Devaraj!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #637 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/637/)
        MAPREDUCE-2307. Exception thrown in Jobtracker logs, when the Scheduler
        configured is FairScheduler. Contributed by Devaraj K.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #637 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/637/ ) MAPREDUCE-2307 . Exception thrown in Jobtracker logs, when the Scheduler configured is FairScheduler. Contributed by Devaraj K.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/ )

          People

          • Assignee:
            Devaraj K
            Reporter:
            Devaraj K
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development