Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4464

Lower the default max applications stored in the RM and store

    Details

    • Type: Wish
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: resourcemanager
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      The maximum applications the RM stores in memory and in the state-store by default has been lowered from 10,000 to 1,000. This should ease the pressure on the state-store. However, installations relying on the default to be 10,000 are affected.

      Description

      my cluster has 120 nodes.
      I configured RM Restart feature.

      yarn.resourcemanager.recovery.enabled=true
      yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
      yarn.resourcemanager.fs.state-store.uri=/system/yarn/rmstore
      

      unfortunately I did not configure yarn.resourcemanager.state-store.max-completed-applications.
      so that property configured default value 10,000.

      I have restarted RM due to changing another configuartion.
      I expected that RM restart immediately.

      recovery process was very slow. I have waited about 20min.

      realize missing yarn.resourcemanager.state-store.max-completed-applications.

      its default value is very huge.
      need to change lower value or document notice on RM Restart page.

        Attachments

        1. YARN-4464.008.patch
          4 kB
          Daniel Templeton
        2. YARN-4464.007.patch
          4 kB
          Daniel Templeton
        3. YARN-4464.006.patch
          3 kB
          Daniel Templeton
        4. YARN-4464.005.patch
          2 kB
          Daniel Templeton
        5. YARN-4464.004.patch
          3 kB
          Daniel Templeton
        6. YARN-4464.003.patch
          2 kB
          Daniel Templeton
        7. YARN-4464.002.patch
          2 kB
          Daniel Templeton
        8. YARN-4464.001.patch
          1.0 kB
          Daniel Templeton

          Activity

            People

            • Assignee:
              templedf Daniel Templeton
              Reporter:
              magnum KWON BYUNGCHANG
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: