Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2080 Speed up DFS read path by lessening checksum overhead
  3. HDFS-2130

Switch default checksum to CRC32C

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.23.1
    • hdfs-client
    • None
    • Reviewed
    • The default checksum algorithm used on HDFS is now CRC32C. Data from previous versions of Hadoop can still be read backwards-compatibly.

    Description

      Once the other subtasks/parts of HDFS-2080 are complete, CRC32C will be a much more efficient checksum algorithm than CRC32. Hence we should change the default checksum to CRC32C.

      However, in order to continue to support append against blocks created with the old checksum, we will need to implement some kind of handshaking in the write pipeline.

      Attachments

        1. hdfs-2130.txt
          32 kB
          Todd Lipcon
        2. hdfs-2130.txt
          32 kB
          Todd Lipcon
        3. hdfs-2130.txt
          33 kB
          Todd Lipcon

        Issue Links

          is blocked by

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-2521 Remove custom checksum headers from data transfer protocol

          • Major - Major loss of function.
          • Closed

          Activity

            People

              tlipcon Todd Lipcon
              tlipcon Todd Lipcon
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: