Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-7289

Memory allocation of RocksDB can be problematic in container environments

    XMLWordPrintableJSON

    Details

      Description

      Flink's RocksDB based state backend allocates native memory. The amount of allocated memory by RocksDB is not under the control of Flink or the JVM and can (theoretically) grow without limits.
      In container environments, this can be problematic because the process can exceed the memory budget of the container, and the process will get killed. Currently, there is no other option than trusting RocksDB to be well behaved and to follow its memory configurations. However, limiting RocksDB's memory usage is not as easy as setting a single limit parameter. The memory limit is determined by an interplay of several configuration parameters, which is almost impossible to get right for users. Even worse, multiple RocksDB instances can run inside the same process and make reasoning about the configuration also dependent on the Flink job.

      Some information about the memory management in RocksDB can be found here:
      https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB
      https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide

      We should try to figure out ways to help users in one or more of the following ways:

      • Some way to autotune or calculate the RocksDB configuration.
      • Conservative default values.
      • Additional documentation.

        Attachments

        1. completeRocksdbConfig.txt
          9 kB
          Akshay Iyangar

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                srichter Stefan Richter
              • Votes:
                7 Vote for this issue
                Watchers:
                39 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1.5h
                  1.5h