Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha4
    • Component/s: None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      CodecRegistry uses ServiceLoader to dynamically load all implementations of RawErasureCoderFactory. In Hadoop 3.0, there are several built-in implementations, and user can also provide self-defined implementations with the corresponding resource files.
      For each codec, user can configure the order of the implementations with the configuration keys:
      `io.erasurecode.codec.rs.rawcoders` for the default RS codec,
      `io.erasurecode.codec.rs-legacy.rawcoders` for the legacy RS codec,
      `io.erasurecode.codec.xor.rawcoders` for the XOR codec.
      User can also configure self-defined codec with the configuration key like:
      `io.erasurecode.codec.self-defined.rawcoders`.
      For each codec, Hadoop will use the implementation according to the order configured. If the former implementation fails, it will fall back to call the latter one. The order is defined by a list of coder names separated by commas. The names for the built-in implementations are:
      `rs_native` and `rs_java` for the default RS codec, of which the former is a native implementation which leverages Intel ISA-L library, which is the default implementation and the latter is the implementation in pure Java,
      `rs-legacy_java` for the legacy RS codec, which is the default implementation in pure Java,
      `xor_native` and `xor_java` for the XOR codec, of which the former is the Intel ISA-L implementation which is the default one and the latter in pure Java.
      Show
      CodecRegistry uses ServiceLoader to dynamically load all implementations of RawErasureCoderFactory. In Hadoop 3.0, there are several built-in implementations, and user can also provide self-defined implementations with the corresponding resource files. For each codec, user can configure the order of the implementations with the configuration keys: `io.erasurecode.codec.rs.rawcoders` for the default RS codec, `io.erasurecode.codec.rs-legacy.rawcoders` for the legacy RS codec, `io.erasurecode.codec.xor.rawcoders` for the XOR codec. User can also configure self-defined codec with the configuration key like: `io.erasurecode.codec.self-defined.rawcoders`. For each codec, Hadoop will use the implementation according to the order configured. If the former implementation fails, it will fall back to call the latter one. The order is defined by a list of coder names separated by commas. The names for the built-in implementations are: `rs_native` and `rs_java` for the default RS codec, of which the former is a native implementation which leverages Intel ISA-L library, which is the default implementation and the latter is the implementation in pure Java, `rs-legacy_java` for the legacy RS codec, which is the default implementation in pure Java, `xor_native` and `xor_java` for the XOR codec, of which the former is the Intel ISA-L implementation which is the default one and the latter in pure Java.

      Description

      This is a follow-on task for HADOOP-13010 as discussed over there. There may be some better approach allowing to customize and configure erasure coders than the current having raw coder factory, as Colin P. McCabe suggested. Will copy the relevant comments here to continue the discussion.

        Attachments

        1. HADOOP-13200.02.patch
          34 kB
          Tim Yao
        2. HADOOP-13200.03.patch
          33 kB
          Tim Yao
        3. HADOOP-13200.04.patch
          37 kB
          Tim Yao
        4. HADOOP-13200.05.patch
          38 kB
          Tim Yao
        5. HADOOP-13200.06.patch
          40 kB
          Tim Yao
        6. HADOOP-13200.07.patch
          42 kB
          Tim Yao
        7. HADOOP-13200.08.patch
          41 kB
          Tim Yao
        8. HADOOP-13200.09.patch
          41 kB
          Tim Yao
        9. HADOOP-13200.10.patch
          41 kB
          Tim Yao
        10. HADOOP-13200.11.patch
          45 kB
          Tim Yao

          Issue Links

            Activity

              People

              • Assignee:
                timmyyao Tim Yao
                Reporter:
                drankye Kai Zheng
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: