Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-45 [Preemption] Scheduler feedback to AM to release containers
  3. YARN-1408

Preemption caused Invalid State Event: ACQUIRED at KILLED and caused a task timeout for 30mins

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.5.0
    • Component/s: resourcemanager
    • Labels:
      None
    • Target Version/s:

      Description

      Capacity preemption is enabled as follows.

      • yarn.resourcemanager.scheduler.monitor.enable= true ,
      • yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy

      Queue = a,b
      Capacity of Queue A = 80%
      Capacity of Queue B = 20%

      Step 1: Assign a big jobA on queue a which uses full cluster capacity
      Step 2: Submitted a jobB to queue b which would use less than 20% of cluster capacity

      JobA task which uses queue b capcity is been preempted and killed.

      This caused below problem:
      1. New Container has got allocated for jobA in Queue A as per node update from an NM.
      2. This container has been preempted immediately as per preemption.

      Here ACQUIRED at KILLED Invalid State exception came when the next AM heartbeat reached RM.
      ERROR org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: Can't handle this event at current state
      org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: ACQUIRED at KILLED

      This also caused the Task to go for a timeout for 30minutes as this Container was already killed by preemption.
      attempt_1380289782418_0003_m_000000_0 Timed out after 1800 secs

        Attachments

        1. Yarn-1408.patch
          2 kB
          Sunil Govindan
        2. Yarn-1408.1.patch
          2 kB
          Sunil Govindan
        3. Yarn-1408.2.patch
          1 kB
          Sunil Govindan
        4. Yarn-1408.3.patch
          6 kB
          Sunil Govindan
        5. Yarn-1408.4.patch
          7 kB
          Sunil Govindan
        6. Yarn-1408.5.patch
          27 kB
          Sunil Govindan
        7. Yarn-1408.6.patch
          30 kB
          Sunil Govindan
        8. Yarn-1408.7.patch
          36 kB
          Sunil Govindan
        9. Yarn-1408.8.patch
          35 kB
          Sunil Govindan
        10. Yarn-1408.9.patch
          36 kB
          Sunil Govindan
        11. Yarn-1408.10.patch
          36 kB
          Sunil Govindan
        12. Yarn-1408.11.patch
          36 kB
          Sunil Govindan
        13. YARN-1408-branch-2.5-1.patch
          38 kB
          Mayank Bansal

          Issue Links

            Activity

              People

              • Assignee:
                sunilg Sunil Govindan
                Reporter:
                sunilg Sunil Govindan
              • Votes:
                0 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: