Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.1
    • Component/s: hdfs-client
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      The default checksum algorithm used on HDFS is now CRC32C. Data from previous versions of Hadoop can still be read backwards-compatibly.

      Description

      Once the other subtasks/parts of HDFS-2080 are complete, CRC32C will be a much more efficient checksum algorithm than CRC32. Hence we should change the default checksum to CRC32C.

      However, in order to continue to support append against blocks created with the old checksum, we will need to implement some kind of handshaking in the write pipeline.

      1. hdfs-2130.txt
        33 kB
        Todd Lipcon
      2. hdfs-2130.txt
        32 kB
        Todd Lipcon
      3. hdfs-2130.txt
        32 kB
        Todd Lipcon

        Issue Links

          Activity

          Allen Wittenauer made changes -
          Fix Version/s 0.24.0 [ 12317653 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Todd Lipcon made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Release Note The default checksum algorithm used on HDFS is now CRC32C. Data from previous versions of Hadoop can still be read backwards-compatibly.
          Fix Version/s 0.24.0 [ 12317653 ]
          Fix Version/s 0.23.1 [ 12318885 ]
          Resolution Fixed [ 1 ]
          Todd Lipcon made changes -
          Attachment hdfs-2130.txt [ 12502067 ]
          Todd Lipcon made changes -
          Attachment hdfs-2130.txt [ 12501994 ]
          Todd Lipcon made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 0.23.1 [ 12318885 ]
          Todd Lipcon made changes -
          Attachment hdfs-2130.txt [ 12501737 ]
          Todd Lipcon made changes -
          Assignee Todd Lipcon [ tlipcon ]
          Todd Lipcon made changes -
          Link This issue is blocked by HDFS-2521 [ HDFS-2521 ]
          Todd Lipcon made changes -
          Fix Version/s 0.23.0 [ 12315571 ]
          Description Once the other subtasks/parts of HDFS-2080 are complete, CRC32C will be a much more efficient checksum algorithm than CRC32. Hence we should change the default checksum to CRC32C.

          However, in order to continue to support append against blocks created with the old checksum, we will need to implement some kind of handshaking in the write pipeline.
          Todd Lipcon made changes -
          Component/s hdfs client [ 12312928 ]
          Todd Lipcon made changes -
          Field Original Value New Value
          Fix Version/s 0.23.0 [ 12315571 ]
          Todd Lipcon created issue -

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development