Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-8183

Fix ConcurrentModificationException inside RMAppAttemptMetrics#convertAtomicLongMaptoLongMap

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 3.0.0, 3.1.0
    • 2.10.0, 3.2.0, 3.1.1, 3.0.3
    • yarn
    • None
    • Reviewed

    Description

      yclient gets stuck in killing application with repeatedly printing following message

      INFO impl.YarnClientImpl: Waiting for application application_1523604760756_0001 to be killed.

      RM shows following exception

       ERROR resourcemanager.ResourceManager (ResourceManager.java:handle(995)) - Error in handling event type APP_UPDATE_SAVED for application application_ID
      java.util.ConcurrentModificationException
              at java.util.HashMap$HashIterator.nextNode(HashMap.java:1442)
              at java.util.HashMap$EntryIterator.next(HashMap.java:1476)
              at java.util.HashMap$EntryIterator.next(HashMap.java:1474)
              at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics.convertAtomicLongMaptoLongMap(RMAppAttemptMetrics.java:212)
              at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics.getAggregateAppResourceUsage(RMAppAttemptMetrics.java:133)
              at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.getRMAppMetrics(RMAppImpl.java:1660)
              at org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher.appFinished(TimelineServiceV2Publisher.java:178)
              at org.apache.hadoop.yarn.server.resourcemanager.metrics.CombinedSystemMetricsPublisher.appFinished(CombinedSystemMetricsPublisher.java:73)
              at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$FinalTransition.transition(RMAppImpl.java:1470)
              at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$AppKilledTransition.transition(RMAppImpl.java:1408)
              at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$AppKilledTransition.transition(RMAppImpl.java:1400)
              at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$FinalStateSavedTransition.transition(RMAppImpl.java:1177)
              at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$FinalStateSavedTransition.transition(RMAppImpl.java:1164)
              at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
              at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
              at org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
              at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
              at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:898)
              at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:118)
              at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:993)
              at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:977)
              at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
              at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
              at java.lang.Thread.run(Thread.java:748)
      

      Attachments

        1. YARN-8183.2.patch
          2 kB
          Suma Shivaprasad
        2. YARN-8183.1.patch
          2 kB
          Suma Shivaprasad

        Activity

          People

            suma.shivaprasad Suma Shivaprasad
            ssathish@hortonworks.com Sumana Sathish
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: