Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6523

Optimize system credentials sent in node heartbeat responses

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0, 2.7.3
    • Fix Version/s: 3.3.0
    • Component/s: RM
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Currently as part of heartbeat response RM sets all application's tokens though all applications might not be active on the node. On top of it NodeHeartbeatResponsePBImpl converts tokens for each app into SystemCredentialsForAppsProto. Hence for each node and each heartbeat too many SystemCredentialsForAppsProto objects were getting created.
      We hit a OOM while testing for 2000 concurrent apps on 500 nodes cluster with 8GB RAM configured for RM

        Attachments

        1. YARN-6523.001.patch
          31 kB
          Manikandan R
        2. YARN-6523.002.patch
          30 kB
          Manikandan R
        3. YARN-6523.003.patch
          30 kB
          Manikandan R
        4. YARN-6523.004.patch
          31 kB
          Manikandan R
        5. YARN-6523.005.patch
          31 kB
          Manikandan R
        6. YARN-6523.006.patch
          24 kB
          Manikandan R
        7. YARN-6523.007.patch
          37 kB
          Manikandan R
        8. YARN-6523.008.patch
          62 kB
          Manikandan R
        9. YARN-6523.009.patch
          67 kB
          Manikandan R
        10. YARN-6523.010.patch
          79 kB
          Manikandan R
        11. YARN-6523.011.patch
          79 kB
          Manikandan R
        12. YARN-6523.012.patch
          79 kB
          Manikandan R
        13. YARN-6523.013.patch
          80 kB
          Manikandan R
        14. YARN-6523.014.patch
          80 kB
          Manikandan R
        15. YARN-6523.015.patch
          80 kB
          Manikandan R

          Activity

            People

            • Assignee:
              manirajv06@gmail.com Manikandan R
              Reporter:
              Naganarasimha Naganarasimha G R
            • Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: