Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6863

job finish but yarn list status is accepted and applicationmaster is hang on Waiting for application to be successfully unregistered

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.6.0
    • Fix Version/s: None
    • Labels:
      None

      Description

      applicationmaster process log is loop on “Waiting for application to be successfully unregistered.”

      ApplicationMaster log

      2017-03-12 01:16:50,854 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1489067586592_112212Job Transitioned from RUNNING to COMMITTING
      2017-03-12 01:16:50,854 INFO [CommitterEvent Processor #2] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_COMMIT
      2017-03-12 01:16:50,884 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for JobFinishedEvent 
      2017-03-12 01:16:50,884 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1489067586592_112212Job Transitioned from COMMITTING to SUCCEEDED
      2017-03-12 01:16:50,885 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: We are finishing cleanly so this is the last retry
      2017-03-12 01:16:50,885 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true
      2017-03-12 01:16:50,885 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: RMCommunicator notified that shouldUnregistered is: true
      2017-03-12 01:16:50,885 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true
      2017-03-12 01:16:50,885 INFO [Thread-402] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true
      2017-03-12 01:16:50,885 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services
      2017-03-12 01:16:50,886 INFO [Thread-402] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 0
      2017-03-12 01:16:50,959 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:260 CompletedReds:0 ContAlloc:261 ContRel:0 HostLocal:115 RackLocal:146
      2017-03-12 01:16:51,212 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://bipcluster:8020/data/yarn/stage/master/.staging/job_1489067586592_112212/job_1489067586592_112212_1.jhist to hdfs://bipcluster:8020/data/yarn/intermediate_done/master/job_1489067586592_112212-1489252563933-master-3934_7823976%3Adw_log_app_page_event_hour_inc.sql%3As4-1489252610863-260-0-SUCCEEDED-root.bigdata.etl.hourlyetl.veryhigh-1489252568534.jhist_tmp
      2017-03-12 01:16:51,255 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://bipcluster:8020/data/yarn/intermediate_done/master/job_1489067586592_112212-1489252563933-master-3934_7823976%3Adw_log_app_page_event_hour_inc.sql%3As4-1489252610863-260-0-SUCCEEDED-root.bigdata.etl.hourlyetl.veryhigh-1489252568534.jhist_tmp
      2017-03-12 01:16:51,256 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://bipcluster:8020/data/yarn/stage/master/.staging/job_1489067586592_112212/job_1489067586592_112212_1_conf.xml to hdfs://bipcluster:8020/data/yarn/intermediate_done/master/job_1489067586592_112212_conf.xml_tmp
      2017-03-12 01:16:51,270 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://bipcluster:8020/data/yarn/intermediate_done/master/job_1489067586592_112212_conf.xml_tmp
      2017-03-12 01:16:51,274 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://bipcluster:8020/data/yarn/intermediate_done/master/job_1489067586592_112212.summary_tmp to hdfs://bipcluster:8020/data/yarn/intermediate_done/master/job_1489067586592_112212.summary
      2017-03-12 01:16:51,276 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://bipcluster:8020/data/yarn/intermediate_done/master/job_1489067586592_112212_conf.xml_tmp to hdfs://bipcluster:8020/data/yarn/intermediate_done/master/job_1489067586592_112212_conf.xml
      2017-03-12 01:16:51,277 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://bipcluster:8020/data/yarn/intermediate_done/master/job_1489067586592_112212-1489252563933-master-3934_7823976%3Adw_log_app_page_event_hour_inc.sql%3As4-1489252610863-260-0-SUCCEEDED-root.bigdata.etl.hourlyetl.veryhigh-1489252568534.jhist_tmp to hdfs://bipcluster:8020/data/yarn/intermediate_done/master/job_1489067586592_112212-1489252563933-master-3934_7823976%3Adw_log_app_page_event_hour_inc.sql%3As4-1489252610863-260-0-SUCCEEDED-root.bigdata.etl.hourlyetl.veryhigh-1489252568534.jhist
      2017-03-12 01:16:51,277 INFO [Thread-402] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop()
      2017-03-12 01:16:51,278 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1489067586592_112212_m_000217_0
      2017-03-12 01:16:51,279 INFO [Thread-402] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sh-hadoop-datanode-128-41.elenet.me:39175
      2017-03-12 01:16:51,284 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1489067586592_112212_m_000217_0 TaskAttempt Transitioned from SUCCESS_FINISHING_CONTAINER to SUCCEEDED
      2017-03-12 01:16:51,294 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Setting job diagnostics to 
      2017-03-12 01:16:51,297 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: History url is http://bigdata-rsm.elenet.me:20020/jobhistory/job/job_1489067586592_112212
      2017-03-12 01:16:51,302 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
      2017-03-12 01:16:51,803 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
      2017-03-12 01:16:52,305 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
      2017-03-12 01:16:52,806 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
      2017-03-12 01:16:53,306 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
      2017-03-12 01:16:53,808 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
      2017-03-12 01:16:54,309 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
      2017-03-12 01:16:54,810 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
      2017-03-12 01:16:55,311 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
      2017-03-12 01:16:55,821 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
      2017-03-12 01:16:56,322 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
      2017-03-12 01:16:56,823 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
      2017-03-12 01:16:57,324 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
      2017-03-12 01:16:57,825 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
      2017-03-12 01:16:58,326 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
      2017-03-12 01:16:58,828 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
      2017-03-12 01:16:59,329 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
      2017-03-12 01:16:59,830 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
      2017-03-12 01:17:00,331 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
      2017-03-12 01:17:00,832 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
      2017-03-12 01:17:01,333 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
      2017-03-12 01:17:01,835 INFO [Thread-402] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
      

        Attachments

        1. yarn resourcemanager job list status.png
          537 kB
          scott.zhai
        2. jobhistory status.png
          459 kB
          scott.zhai

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              zhaiyuyong scott.zhai
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated: