Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14352

backport client-side EC support to branch-2

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: erasure-coding, hdfs-client
    • Labels:
      None

      Description

      Currently, Hadoop 2.x clients can't read or write striped files from HDFS. This affects compatibility with 3.x clusters in two ways:

      • The obvious impact is that 2.x clients can't make use of the new erasure coding in feature in Hadoop 3.
      • For some use cases, clients built against Hadoop 3 won't be able to use erasure coding either. This is because if they write a striped file, then clients built against Hadoop 2 won't be able to read it.

      This ticket proposes backporting the client-side components of HDFS-7285 to branch-2 for improved compatibility between 2.x clients and 3.x clusters. I believe this can be done without also backporting the changes made to the NameNodes and the DataNodes. While many lines of code would need to be backported, most of it is new code that can be copy/pasted from trunk, which simplifies the process. The existing code in DFSClient, DFSInputStream, DFSOutputStream, etc. that would need to be modified is still significant, but much smaller.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Steven Rand Steven Rand
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated: