Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-7258

IllegalArgumentException in Netty bootstrap with large memory state segment size

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 1.3.1
    • 1.3.2, 1.4.0
    • Runtime / Network
    • None

    Description

      In NettyBootstrap we configure the low and high watermarks in the following order:

      bootstrap.childOption(ChannelOption.WRITE_BUFFER_LOW_WATER_MARK, config.getMemorySegmentSize() + 1);
      bootstrap.childOption(ChannelOption.WRITE_BUFFER_HIGH_WATER_MARK, 2 * config.getMemorySegmentSize());
      

      When the memory segment size is higher than the default high water mark, this throws an `IllegalArgumentException` when a client tries to connect. Hence, this unfortunately only happens during runtime when a intermediate result is requested. This doesn't fail the job, but logs a warning and ignores the failed configuration attempt, potentially resulting in degraded performance because of a lower than expected watermark.

      A simple fix is to first configure the high water mark and only then configure the low watermark.

      Attachments

        Issue Links

          Activity

            People

              uce Ufuk Celebi
              uce Ufuk Celebi
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: