Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9914

Use separate configs for free disk space checking for full and not-full disks

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      YARN-3943 added separate configurations for the nodemanager health check disk utilization full disk check:

      max-disk-utilization-per-disk-percentage - threshold for marking a good disk full

      disk-utilization-watermark-low-per-disk-percentage - threshold for marking a full disk as not full.

      On our clusters, we do not use these configs. We instead use min-free-space-per-disk-mb so we can specify the limit in mb instead of percent of utilization. We have observed the same oscillation behavior as described in YARN-3943 with this parameter. I would like to add an optional config to specify a separate threshold for marking a full disk as not full:

      min-free-space-per-disk-mb - threshold at which a good disk is marked full

      disk-free-space-per-disk-high-watermark-mb - threshold at which a full disk is marked good.

      So for example, we could set min-free-space-per-disk-mb = 5GB, which would cause a disk to be marked full when free space goes below 5GB, and disk-free-space-per-disk-high-watermark-mb = 10GB to keep the disk in the full state until free space goes above 10GB.

      Attachments

        1. YARN-9914.001.patch
          22 kB
          Jim Brennan
        2. YARN-9914.002.patch
          22 kB
          Jim Brennan
        3. YARN-9914-branch-2.8.001.patch
          22 kB
          Jim Brennan

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            jbrennan Jim Brennan
            jbrennan Jim Brennan
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment