Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-7258

IllegalArgumentException in Netty bootstrap with large memory state segment size

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.3.1
    • Fix Version/s: 1.4.0, 1.3.2
    • Component/s: Network
    • Labels:
      None

      Description

      In NettyBootstrap we configure the low and high watermarks in the following order:

      bootstrap.childOption(ChannelOption.WRITE_BUFFER_LOW_WATER_MARK, config.getMemorySegmentSize() + 1);
      bootstrap.childOption(ChannelOption.WRITE_BUFFER_HIGH_WATER_MARK, 2 * config.getMemorySegmentSize());
      

      When the memory segment size is higher than the default high water mark, this throws an `IllegalArgumentException` when a client tries to connect. Hence, this unfortunately only happens during runtime when a intermediate result is requested. This doesn't fail the job, but logs a warning and ignores the failed configuration attempt, potentially resulting in degraded performance because of a lower than expected watermark.

      A simple fix is to first configure the high water mark and only then configure the low watermark.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                uce Ufuk Celebi
                Reporter:
                uce Ufuk Celebi
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: