Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2630

MR-279: refreshQueues leads to NPEs when used w/FifoScheduler

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: mrv2
    • Labels:
      None
    • Environment:

      All

    • Hadoop Flags:
      Reviewed

      Description

      The RM's admin service exposes a method refreshQueues that is used to update the queue configuration when used with the CapacityScheduler, but if it is used with the FifoScheduler, it will set the containerTokenSecretManager/clusterTracker fields on the FifoScheduler to null, which eventually leads to NPE. Since the FifoScheduler only has one queue that cannot be refreshed, the correct behavior is for the refreshQueues call to be a no-op.

      I will attach a patch that fixes this by splitting the ResourceScheduler's reinitialize method into separate initialize/updateQueues methods.

      1. MAPREDUCE-2630.patch
        12 kB
        Josh Wills
      2. MAPREDUCE-2630-simple.patch
        2 kB
        Josh Wills

        Issue Links

          Activity

          Hide
          Arun C Murthy added a comment -

          Josh, sorry, took me a while to get to this.

          Thanks for the patch.

          Could we just add a check to FifoScheduler.reinit ala CS.reinit? That way we can keep a single method in Scheduler. I'm happy to rename it 'configure' or some such. Thoughts?

          Show
          Arun C Murthy added a comment - Josh, sorry, took me a while to get to this. Thanks for the patch. Could we just add a check to FifoScheduler.reinit ala CS.reinit? That way we can keep a single method in Scheduler. I'm happy to rename it 'configure' or some such. Thoughts?
          Hide
          Josh Wills added a comment -

          No worries Arun- it was a hectic week. Really enjoyed your talk at the Summit, lots of good questions from the audience.

          My preference for the two separate methods was based on the idea that the containerTokenSecretManager and clusterTracker really should never change once the scheduler is init'd, and that having the two separate methods would prevent any other scheduler impls that came along from introducing the same kind of error. If that isn't true, i.e., there is some situation where we would want to change the value of the containerTokenSecretManager/clusterTracker, I'm fine to have the single reinit method.

          Show
          Josh Wills added a comment - No worries Arun- it was a hectic week. Really enjoyed your talk at the Summit, lots of good questions from the audience. My preference for the two separate methods was based on the idea that the containerTokenSecretManager and clusterTracker really should never change once the scheduler is init'd, and that having the two separate methods would prevent any other scheduler impls that came along from introducing the same kind of error. If that isn't true, i.e., there is some situation where we would want to change the value of the containerTokenSecretManager/clusterTracker, I'm fine to have the single reinit method.
          Hide
          Josh Wills added a comment -

          Modified patch to include new tests.

          Show
          Josh Wills added a comment - Modified patch to include new tests.
          Hide
          Josh Wills added a comment -

          Simpler patch that just updates the impl of FifoScheduler to make it differentiate between initialization and re-initialization, ala the CapacityScheduler.

          Show
          Josh Wills added a comment - Simpler patch that just updates the impl of FifoScheduler to make it differentiate between initialization and re-initialization, ala the CapacityScheduler.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12485614/MAPREDUCE-2630-simple.patch
          against trunk revision 1145424.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/456//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12485614/MAPREDUCE-2630-simple.patch against trunk revision 1145424. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/456//console This message is automatically generated.
          Hide
          Mahadev konar added a comment -

          I just pushed this. Thanks Josh.

          Show
          Mahadev konar added a comment - I just pushed this. Thanks Josh.

            People

            • Assignee:
              Josh Wills
              Reporter:
              Josh Wills
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 1h
                1h
                Remaining:
                Remaining Estimate - 1h
                1h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development