Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-3054

Disable erasure coding for sharelib if Oozie runs on Hadoop 3

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 5.0.0b1
    • client
    • None

    Description

      Hadoop 3 will introduce Erasure Coding support for HDFS. This means less disk usage, but increased CPU load for reading/writing.

      The contents of Oozie sharelib are localized every time an asynchronous action is executed. Therefore, to reduce the possible negative performance impact of EC, Oozie sharelib should not utilize it.

      Since we're supporting Hadoop 2 as well, using the EC API will be done with Java reflection. Later, when we drop support for Hadoop 2, this can be rewritten.

      Attachments

        1. OOZIE-3045-001.patch
          6 kB
          Peter Bacsko
        2. OOZIE-3045-002.patch
          7 kB
          Peter Bacsko
        3. OOZIE-3054-003.patch
          6 kB
          Peter Bacsko

        Issue Links

          Activity

            People

              pbacsko Peter Bacsko
              pbacsko Peter Bacsko
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: