Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-17865

Increase default size of `state.backend.fs.memory-threshold`

    XMLWordPrintableJSON

    Details

    • Release Note:
      Hide
      After FLINK-17865 the default value of `state.backend.fs.memory-threshold` has been increased from 1K to 20K to prevent too many small files created on remote FS for small states. NOTICE: jobs with large parallelism on source or stateful operators may have "JM OOM" or "RPC message exceeding maximum frame size" problem with this change, and please manually set the configuration back to 1K if you encounter such issues.
      Show
      After FLINK-17865 the default value of `state.backend.fs.memory-threshold` has been increased from 1K to 20K to prevent too many small files created on remote FS for small states. NOTICE: jobs with large parallelism on source or stateful operators may have "JM OOM" or "RPC message exceeding maximum frame size" problem with this change, and please manually set the configuration back to 1K if you encounter such issues.

      Description

      As discussed in ML thread, we decided to increase the default value of `state.backend.fs.memory-threshold` from 1K to 20K to prevent too many small files created on remote FS.

      We need to add an explicit release note to notify our users about this change, especially for those having jobs with large parallelism on source or stateful operators.

        Attachments

          Activity

            People

            • Assignee:
              yunta Yun Tang
              Reporter:
              liyu Yu Li
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: