Details

    • Sub-task
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • None
    • 3.0.0-alpha4
    • None
    • Incompatible change, Reviewed
    • Hide
      CodecRegistry uses ServiceLoader to dynamically load all implementations of RawErasureCoderFactory. In Hadoop 3.0, there are several built-in implementations, and user can also provide self-defined implementations with the corresponding resource files.
      For each codec, user can configure the order of the implementations with the configuration keys:
      `io.erasurecode.codec.rs.rawcoders` for the default RS codec,
      `io.erasurecode.codec.rs-legacy.rawcoders` for the legacy RS codec,
      `io.erasurecode.codec.xor.rawcoders` for the XOR codec.
      User can also configure self-defined codec with the configuration key like:
      `io.erasurecode.codec.self-defined.rawcoders`.
      For each codec, Hadoop will use the implementation according to the order configured. If the former implementation fails, it will fall back to call the latter one. The order is defined by a list of coder names separated by commas. The names for the built-in implementations are:
      `rs_native` and `rs_java` for the default RS codec, of which the former is a native implementation which leverages Intel ISA-L library, which is the default implementation and the latter is the implementation in pure Java,
      `rs-legacy_java` for the legacy RS codec, which is the default implementation in pure Java,
      `xor_native` and `xor_java` for the XOR codec, of which the former is the Intel ISA-L implementation which is the default one and the latter in pure Java.
      Show
      CodecRegistry uses ServiceLoader to dynamically load all implementations of RawErasureCoderFactory. In Hadoop 3.0, there are several built-in implementations, and user can also provide self-defined implementations with the corresponding resource files. For each codec, user can configure the order of the implementations with the configuration keys: `io.erasurecode.codec.rs.rawcoders` for the default RS codec, `io.erasurecode.codec.rs-legacy.rawcoders` for the legacy RS codec, `io.erasurecode.codec.xor.rawcoders` for the XOR codec. User can also configure self-defined codec with the configuration key like: `io.erasurecode.codec.self-defined.rawcoders`. For each codec, Hadoop will use the implementation according to the order configured. If the former implementation fails, it will fall back to call the latter one. The order is defined by a list of coder names separated by commas. The names for the built-in implementations are: `rs_native` and `rs_java` for the default RS codec, of which the former is a native implementation which leverages Intel ISA-L library, which is the default implementation and the latter is the implementation in pure Java, `rs-legacy_java` for the legacy RS codec, which is the default implementation in pure Java, `xor_native` and `xor_java` for the XOR codec, of which the former is the Intel ISA-L implementation which is the default one and the latter in pure Java.

    Description

      This is a follow-on task for HADOOP-13010 as discussed over there. There may be some better approach allowing to customize and configure erasure coders than the current having raw coder factory, as cmccabe suggested. Will copy the relevant comments here to continue the discussion.

      Attachments

        1. HADOOP-13200.02.patch
          34 kB
          Tim Yao
        2. HADOOP-13200.03.patch
          33 kB
          Tim Yao
        3. HADOOP-13200.04.patch
          37 kB
          Tim Yao
        4. HADOOP-13200.05.patch
          38 kB
          Tim Yao
        5. HADOOP-13200.06.patch
          40 kB
          Tim Yao
        6. HADOOP-13200.07.patch
          42 kB
          Tim Yao
        7. HADOOP-13200.08.patch
          41 kB
          Tim Yao
        8. HADOOP-13200.09.patch
          41 kB
          Tim Yao
        9. HADOOP-13200.10.patch
          41 kB
          Tim Yao
        10. HADOOP-13200.11.patch
          45 kB
          Tim Yao

        Issue Links

          Activity

            People

              timmyyao Tim Yao
              drankye Kai Zheng
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: