Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-2402

oozie-setup.sh sharelib create takes a long time on large clusters

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.2.0
    • Fix Version/s: 4.3.0
    • Component/s: tools
    • Labels:
      None

      Description

      When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib.
      Copy the tarball itself takes only around 10 seconds. It seems like performance could be improved by loading files concurrently in many threads.

        Attachments

        1. OOZIE-2402-1.patch
          6 kB
          Illya Yalovyy
        2. OOZIE-2402-2.patch
          19 kB
          Illya Yalovyy
        3. OOZIE-2402-3.patch
          19 kB
          Illya Yalovyy
        4. OOZIE-2402-4.patch
          19 kB
          Illya Yalovyy

          Activity

            People

            • Assignee:
              yalovyyi Illya Yalovyy
              Reporter:
              yalovyyi Illya Yalovyy
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: