Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-3649

Logic error regarding storm.supervisor.medium.memory.grace.period.ms

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.0.0, 2.1.0, 2.2.0
    • 2.3.0
    • storm-server
    • None

    Description

      Inside this chunk of code

      https://github.com/apache/storm/blob/2.2.x-branch/storm-server/src/main/java/org/apache/storm/daemon/supervisor/BasicContainer.java#L758

      if (systemFreeMemoryMb < mediumMemoryThresholdMb) {
                          if (memoryLimitExceededStart < 0) {
                              memoryLimitExceededStart = Time.currentTimeMillis();
                          } else {
                              long timeInViolation = Time.currentTimeMillis() - memoryLimitExceededStart;
                              if (timeInViolation > mediumMemoryGracePeriodMs) {
                                  LOG.warn(
                                      "{} is using {} MB > memory limit {} MB for {} seconds",
                                      typeOfCheck,
                                      usageMb,
                                      memoryLimitMb,
                                      timeInViolation / 1000);
                                  return true;
                              }
                          }
                      } 
      

      At very beginning, memoryLimitExceededStart in BasicContainer is initialized as 0. :
      https://github.com/apache/storm/blob/2.2.x-branch/storm-server/src/main/java/org/apache/storm/daemon/supervisor/BasicContainer.java#L80

      protected volatile long memoryLimitExceededStart;
      

      So once it hits this scenario, the grace period doesn't really take any effect because the timeInViolation will be very large (equals to currentTime)

      The logs from a test:

      2020-06-08 20:39:18.277 o.a.s.d.s.BasicContainer SLOT_6707 [WARN] WORKER 9c16e81e-4936-4029-bcda-ceb5b74b8f42 is using 167 MB > memory limit 158 MB for 1591648758 seconds
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            ethanli Ethan Li
            ethanli Ethan Li
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 20m
                20m

                Slack

                  Issue deployment