Flume
  1. Flume
  2. FLUME-2238

Provide option to configure worker threads in NettyAvroRpcClient

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.4.0
    • Fix Version/s: v1.5.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      From dev ML, Credits to Cameron Gandevia

      Embedded flume agents were using around 50 netty IO worker threads.

      The issue was tracked down to the NettyAvroRpcClient which instantiates
      the NioClientSocketChannelFactory with the default workerCount (2 * the
      number of available processors). We would like finer control over the
      number of IO threads we dedicate to flume

      1. FLUME-2238.patch
        17 kB
        Cameron Gandevia

        Activity

        Hide
        Ashish Paliwal added a comment -

        Would be good to add a new variable like maxIoWorkers, which if specified initializes the appropriate NioClientSocketChannelFactory constructor.

        Show
        Ashish Paliwal added a comment - Would be good to add a new variable like maxIoWorkers, which if specified initializes the appropriate NioClientSocketChannelFactory constructor.
        Hide
        Cameron Gandevia added a comment -

        Will submit a review to the review board once it becomes available again.

        Show
        Cameron Gandevia added a comment - Will submit a review to the review board once it becomes available again.
        Show
        Cameron Gandevia added a comment - https://reviews.apache.org/r/15683/
        Hide
        Cameron Gandevia added a comment -

        Code review fixes

        Show
        Cameron Gandevia added a comment - Code review fixes
        Hide
        Ashish Paliwal added a comment -

        Can you please attach the updated patch to JIRA. We usually follow the following convention FLUME-<JIRA NUM>-<num>.patch, where is num starts from 0 and is updated for each iteration if needed. For current case, post the review comments it shall be like FLUME-2238-1.patch

        Show
        Ashish Paliwal added a comment - Can you please attach the updated patch to JIRA. We usually follow the following convention FLUME-<JIRA NUM>-<num>.patch, where is num starts from 0 and is updated for each iteration if needed. For current case, post the review comments it shall be like FLUME-2238 -1.patch
        Hide
        Hari Shreedharan added a comment -

        +1. I am running the tests now, will commit once that is done

        Show
        Hari Shreedharan added a comment - +1. I am running the tests now, will commit once that is done
        Hide
        Hari Shreedharan added a comment -

        Committed, rev: 209169b. Thanks Cameron!

        Show
        Hari Shreedharan added a comment - Committed, rev: 209169b. Thanks Cameron!
        Hide
        Hudson added a comment -

        FAILURE: Integrated in flume-trunk #524 (See https://builds.apache.org/job/flume-trunk/524/)
        FLUME-2238. Provide option to configure worker threads in NettyAvroRpcClient (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=753e4137918b5bdf559dd50a21db2a832aa1dce3)

        • flume-ng-doc/sphinx/FlumeUserGuide.rst
        • flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java
        • flume-ng-sdk/src/main/java/org/apache/flume/api/NettyAvroRpcClient.java
        • flume-ng-sdk/src/main/java/org/apache/flume/api/RpcClientConfigurationConstants.java
        Show
        Hudson added a comment - FAILURE: Integrated in flume-trunk #524 (See https://builds.apache.org/job/flume-trunk/524/ ) FLUME-2238 . Provide option to configure worker threads in NettyAvroRpcClient (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=753e4137918b5bdf559dd50a21db2a832aa1dce3 ) flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java flume-ng-sdk/src/main/java/org/apache/flume/api/NettyAvroRpcClient.java flume-ng-sdk/src/main/java/org/apache/flume/api/RpcClientConfigurationConstants.java

          People

          • Assignee:
            Cameron Gandevia
            Reporter:
            Ashish Paliwal
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development