Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6000

Make AllocationFileLoaderService.Listener public

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: fairscheduler, yarn
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We removed public modifier of AllocationFileLoaderService.Listener in YARN-4997 since it trigger a findbugs warning. However it breaks Hive code in FairSchedulerShim.

      AllocationFileLoaderService allocsLoader = new AllocationFileLoaderService();
          allocsLoader.init(conf);
          allocsLoader.setReloadListener(new AllocationFileLoaderService.Listener() {
            @Override
            public void onReload(AllocationConfiguration allocs) {
              allocConf.set(allocs);
            }
          });
          try {
            allocsLoader.reloadAllocations();
          } catch (Exception ex) {
            throw new IOException("Failed to load queue allocations", ex);
          }
          if (allocConf.get() == null) {
            allocConf.set(new AllocationConfiguration(conf));
          }
          QueuePlacementPolicy queuePolicy = allocConf.get().getPlacementPolicy();
          if (queuePolicy != null) {
            requestedQueue = queuePolicy.assignAppToQueue(requestedQueue, userName);
      

      As a result we should set the modifier back to public.

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 21s trunk passed
          +1 compile 0m 33s trunk passed
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 0m 39s trunk passed
          +1 mvneclipse 0m 18s trunk passed
          +1 findbugs 1m 3s trunk passed
          +1 javadoc 0m 21s trunk passed
          +1 mvninstall 0m 33s the patch passed
          +1 compile 0m 31s the patch passed
          +1 javac 0m 31s the patch passed
          -0 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 17 unchanged - 0 fixed = 18 total (was 17)
          +1 mvnsite 0m 37s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 5s the patch passed
          +1 javadoc 0m 18s the patch passed
          +1 unit 38m 42s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          54m 49s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-6000
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843140/YARN-6000.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6e0525a60794 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / fbdbbd5
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14290/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14290/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14290/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 21s trunk passed +1 compile 0m 33s trunk passed +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 39s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 1m 3s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 33s the patch passed +1 compile 0m 31s the patch passed +1 javac 0m 31s the patch passed -0 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 17 unchanged - 0 fixed = 18 total (was 17) +1 mvnsite 0m 37s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 5s the patch passed +1 javadoc 0m 18s the patch passed +1 unit 38m 42s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 54m 49s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6000 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843140/YARN-6000.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6e0525a60794 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / fbdbbd5 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14290/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14290/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14290/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          Tao Jie Tao Jie added a comment -

          Daniel Templeton, Karthik Kambatla would you mind taking a look at it?

          Show
          Tao Jie Tao Jie added a comment - Daniel Templeton , Karthik Kambatla would you mind taking a look at it?
          Hide
          kasha Karthik Kambatla added a comment -

          The change in YARN-4997 is not incompatible by itself, since AllocationFileLoaderService is marked as Unstable. It would be good to understand the requirements from Hive, so we can expose the necessary details through Public-Stable API.

          That said, I don't see any harm in making the Listener public in the interim.

          +1, checking this in.

          Show
          kasha Karthik Kambatla added a comment - The change in YARN-4997 is not incompatible by itself, since AllocationFileLoaderService is marked as Unstable. It would be good to understand the requirements from Hive, so we can expose the necessary details through Public-Stable API. That said, I don't see any harm in making the Listener public in the interim. +1, checking this in.
          Hide
          kasha Karthik Kambatla added a comment -

          Just committed to trunk. Thanks Tao Jie for fixing this.

          Show
          kasha Karthik Kambatla added a comment - Just committed to trunk. Thanks Tao Jie for fixing this.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11034 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11034/)
          YARN-6000. Make AllocationFileLoaderService.Listener public. (Tao Jie (kasha: rev 4d3f73acc0a5cabc748132889dbe670bea178a3f)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11034 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11034/ ) YARN-6000 . Make AllocationFileLoaderService.Listener public. (Tao Jie (kasha: rev 4d3f73acc0a5cabc748132889dbe670bea178a3f) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java
          Hide
          sershe Sergey Shelukhin added a comment - - edited

          Thanks! As for Hive requirements, see the code snippet above. We are using the listener because that seems to be the only way to get the updated value out. We just need to get the allocConf that we use to get queuepolicy, and then get the queue

          Show
          sershe Sergey Shelukhin added a comment - - edited Thanks! As for Hive requirements, see the code snippet above. We are using the listener because that seems to be the only way to get the updated value out. We just need to get the allocConf that we use to get queuepolicy, and then get the queue

            People

            • Assignee:
              Tao Jie Tao Jie
              Reporter:
              Tao Jie Tao Jie
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development