Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-9820

Change PutKudu Property "Kudu Client Worker Count" Default Value

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 1.15.3
    • 1.17.0, 1.16.1
    • Core Framework
    • None

    Description

      The PutKudu processor property "Kudu Client Worker Count" has a suboptimal value. Please don't use the current "number of CPUs multiplied by 2" behaviour as it leads to a massive amount of workers in our case with physical servers. We have a 8-node cluster where each server has 64 CPUs. We have about 30 PutKudu processors configured -> a lot of worker threads per default just for kudu.

      We have changed the number of worker threads in our case to the number of concurrent tasks. I don't know, maybe it would be great to set it a bit higher than that, but to be honest, I don't exactly understand the impact. It looks still fast with the current config.

      To sum it up, please set a low default value (eg. 4 or 8) for the property "Kudu Client Worker Count" and not a pseudo dynamic one for the PutKudu processor.

      Btw. are there any suggestions how big the number should be?

      Attachments

        Issue Links

          Activity

            People

              exceptionfactory David Handermann
              jzahner Josef Zahner
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m