Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9164

Shutdown NM may cause NPE when opportunistic container scheduling is enabled

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • None
    • 3.0.4, 3.1.2, 3.3.0, 3.2.1
    • None
    • None
    • Reviewed

    Description

      We have meeted an NPE which can crash the whole cluster

      2018-12-31 22:18:11,924 FATAL org.apache.hadoop.yarn.event.EventDispatcher: Error in handling event type APP_ATTEMPT_REMOVED to the Event Dispatcher
      java.lang.NullPointerException
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.completedContainer(AbstractYarnScheduler.java:696)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.doneApplicationAttempt(CapacityScheduler.java:1123)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1827)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:171)
      at org.apache.hadoop.yarn.event.EventDispatcher$EventProcessor.run(EventDispatcher.java:66)
      at java.lang.Thread.run(Thread.java:745)
      
      

      this bug also happens in the latest trunk!

       

      workload is 

      $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-$VERSION.jar pi -Dmapreduce.job.num-opportunistic-maps-percent="100" 50 100
      

      while job is running , shutdown one NM

       also need inject sleep before AbstractYarnScheduler.getNode()

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            xiaoheipangzi lujie Assign to me
            xiaoheipangzi lujie
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment