Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9629

Support configurable MIN_LOG_ROLLING_INTERVAL

    XMLWordPrintableJSON

    Details

      Description

      One of the log-aggregation parameter, the minimum valid value for yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds is MIN_LOG_ROLLING_INTERVAL - it has been hardcoded since its addition in YARN-2583.

      It has been empirically set as 1 hour, as lower values would too frequently put the NodeManagers under pressure. For bigger clusters that is indeed a valid limitation, but for smaller clusters it makes sense and a valid customer usecase to use lower values, even like not so lower 30 mins. At this point this can only be achieved by setting yarn.nodemanager.log-aggregation.debug-enabled, which I believe should be kept as debug purposes.

      I'm suggesting to make this min configurable, although a warning should be logged in the NodeManager startup when this value is lower than 1 hour.

        Attachments

        1. YARN-9629.001.patch
          8 kB
          Adam Antal
        2. YARN-9629.002.patch
          10 kB
          Adam Antal
        3. YARN-9629.003.patch
          11 kB
          Adam Antal
        4. YARN-9629.004.patch
          11 kB
          Adam Antal
        5. YARN-9629.005.patch
          11 kB
          Adam Antal
        6. YARN-9629.006.patch
          11 kB
          Adam Antal
        7. YARN-9629.branch-3.1.001.patch
          11 kB
          Szilard Nemeth
        8. YARN-9629.branch-3.2.001.patch
          11 kB
          Szilard Nemeth
        9. YARN-9629.branch-3.2.001.patch
          11 kB
          Szilard Nemeth

          Activity

            People

            • Assignee:
              adam.antal Adam Antal
              Reporter:
              adam.antal Adam Antal
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: