Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-15068

Disable RocksDB's local LOG by default

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Hide
      Logging in RocksDB (e.g., logging related to flush, compaction, memtable creation, etc.) has been disabled by default to prevent disk space from being filled up unexpectedly. Users that need to enable logging should implement their own RocksDBOptionsFactory that creates DBOptions instances with InfoLogLevel set to INFO_LEVEL.
      Show
      Logging in RocksDB (e.g., logging related to flush, compaction, memtable creation, etc.) has been disabled by default to prevent disk space from being filled up unexpectedly. Users that need to enable logging should implement their own RocksDBOptionsFactory that creates DBOptions instances with InfoLogLevel set to INFO_LEVEL.

    Description

      With Flink's default settings for RocksDB, it will write a log file (not the WAL, but pure logging statements) into the data folder. Besides periodic statistics, it will log compaction attempts, new memtable creations, flushes, etc.

      A few things to note about this practice:

      1. this LOG file is growing over time with no limit
      2. the default logging level is INFO
      3. the statistics in there may help looking into performance and/or disk space problems (but maybe you should be looking and monitoring metrics instead)
      4. this file is not useful for debugging errors since it will be deleted along with the local dir when the TM goes down

      With a custom OptionsFactory, the user can change the behaviour like the following:

          @Override
          public DBOptions createDBOptions(DBOptions currentOptions) {
              currentOptions = super.createDBOptions(currentOptions);
      
              currentOptions.setKeepLogFileNum(10);
              currentOptions.setInfoLogLevel(InfoLogLevel.WARN_LEVEL);
              currentOptions.setStatsDumpPeriodSec(0);
              currentOptions.setMaxLogFileSize(1024 * 1024); // 1 MB each
      
              return currentOptions;
          }

      However, the rotating logger does currently not work (it will not delete old log files - see https://github.com/dataArtisans/frocksdb/pull/12). Also, the user should not have to write his own OptionsFactory to get a sensible default.

      To prevent this file from filling up the disk, I propose to change Flink's default RocksDB settings so that the LOG file is effectively disabled (nothing is written to it by default).

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            nkruber Nico Kruber
            nkruber Nico Kruber
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 20m
                20m

                Slack

                  Issue deployment