Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-928

make checksums optional per FileSystem

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.12.0
    • fs
    • None

    Description

      Checksumming is currently built into the base FileSystem class. It should instead be optional, with each FileSystem implementation electing whether to use the Hadoop-provided checksum system, or to disable it, or to implement its own custom checksum system.

      To implement this, a ChecksumFileSystem implementation can be provided that wraps another FileSystem implementation, implementing checksums as in Hadoop's current mandatory implementation (i.e., as a separate crc file per file that's elided from directory listings). The 'raw' FileSystem methods would be removed. FSDataInputStream and FSDataOutputStream would be made interfaces.

      Attachments

        1. checksum.patch
          129 kB
          Hairong Kuang
        2. checksum1.patch
          166 kB
          Hairong Kuang
        3. checksum2.patch
          164 kB
          Hairong Kuang
        4. checksum3.patch
          165 kB
          Hairong Kuang

        Issue Links

          Activity

            People

              hairong Hairong Kuang
              cutting Doug Cutting
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: