Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-5261

Use proper parallelism for engine context APIs

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Patch Available
    • Critical
    • Resolution: Unresolved
    • None
    • None
    • performance

    Description

      do a global search of these APIs

      • org.apache.hudi.common.engine.HoodieEngineContext#flatMap
      • org.apache.hudi.common.engine.HoodieEngineContext#map

      and similar ones take in parallelism.

      A lot of occurrences are using number of items as parallelism, which affect performance. Parallelism should be based on num cores available in the cluster and set by user via parallelism configs.

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HUDI-712 Improve exporter performance and memory usage

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          links to

          Web Link GitHub Pull Request #7449

          Activity

            People

              jonvex Jonathan Vexler
              xushiyan Raymond Xu
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: