Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11842 Common side change for follow-on work for erasure coding phase I
  3. HADOOP-12047

Indicate preference not to affect input buffers during coding in erasure coder

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0-alpha1
    • None
    • None
    • Reviewed

    Description

      It's good to define and ensure input buffers are not affected during coding process in raw erasure coders. Below are copied from discussion with Jing Zhao in HDFS-8481:

      In that case we cannot reuse the source buffers I guess? Then do we need to expose this information in the decoder?

      Good catch Jing! Yes in this case we can't reuse the source buffers here as they need to be passed to caller/applications without being changed. I'm planning to re-implement the Java coders in HADOOP-12041 and related, when done it's possible to ensure the input buffers not to be affected. Benefits of doing this in coder layer: 1) a more clear contract between coder and caller in more general sense for the inputs; 2) concrete coder may have specific tweak to optimize in the aspect, ideally no input data copying at all, worst, make the copy, but all transparent to callers; 3) allow new coders (LRC, HH) to be layered on other primitive coders (RS, XOR) more easily.

      Attachments

        1. HADOOP-12047-HDFS-7285-v1.patch
          23 kB
          Kai Zheng
        2. HADOOP-12047-v2.patch
          23 kB
          Kai Zheng
        3. HADOOP-12047-v3.patch
          24 kB
          Kai Zheng
        4. HADOOP-12047-v4.patch
          24 kB
          Kai Zheng
        5. HADOOP-12047-v5.patch
          24 kB
          Kai Zheng
        6. initial-poc.patch
          15 kB
          Kai Zheng

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            drankye Kai Zheng
            drankye Kai Zheng
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment