Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-11107

Avoid memory stateBackend to create arbitrary folders under HA path when no checkpoint path configured

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      Currently, memory state-backend would create a folder named with random UUID under HA directory if no checkpoint path ever configured. (the code logic locates within StateBackendLoader#fromApplicationOrConfigOrDefault) However, the default memory state-backend would not only be created on JM side, but also on each task manager's side, which means many folders with random UUID would be created under HA directory. It would result in exception like:

      The directory item limit of /tmp/flink/ha is exceeded: limit=1048576 items=1048576

       If this happens, no new jobs could be submitted only if we clean up those directories manually.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            yunta Yun Tang
            yunta Yun Tang
            Votes:
            2 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 10m
                10m

                Slack

                  Issue deployment