Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-3054

Disable erasure coding for sharelib if Oozie runs on Hadoop 3

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.0.0b1
    • Component/s: client
    • Labels:
      None

      Description

      Hadoop 3 will introduce Erasure Coding support for HDFS. This means less disk usage, but increased CPU load for reading/writing.

      The contents of Oozie sharelib are localized every time an asynchronous action is executed. Therefore, to reduce the possible negative performance impact of EC, Oozie sharelib should not utilize it.

      Since we're supporting Hadoop 2 as well, using the EC API will be done with Java reflection. Later, when we drop support for Hadoop 2, this can be rewritten.

        Attachments

        1. OOZIE-3054-003.patch
          6 kB
          Peter Bacsko
        2. OOZIE-3045-002.patch
          7 kB
          Peter Bacsko
        3. OOZIE-3045-001.patch
          6 kB
          Peter Bacsko

          Issue Links

            Activity

              People

              • Assignee:
                pbacsko Peter Bacsko
                Reporter:
                pbacsko Peter Bacsko
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: