Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-928

make checksums optional per FileSystem

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: fs
    • Labels:
      None

      Description

      Checksumming is currently built into the base FileSystem class. It should instead be optional, with each FileSystem implementation electing whether to use the Hadoop-provided checksum system, or to disable it, or to implement its own custom checksum system.

      To implement this, a ChecksumFileSystem implementation can be provided that wraps another FileSystem implementation, implementing checksums as in Hadoop's current mandatory implementation (i.e., as a separate crc file per file that's elided from directory listings). The 'raw' FileSystem methods would be removed. FSDataInputStream and FSDataOutputStream would be made interfaces.

        Attachments

        1. checksum3.patch
          165 kB
          Hairong Kuang
        2. checksum2.patch
          164 kB
          Hairong Kuang
        3. checksum1.patch
          166 kB
          Hairong Kuang
        4. checksum.patch
          129 kB
          Hairong Kuang

          Issue Links

            Activity

              People

              • Assignee:
                hairong Hairong Kuang
                Reporter:
                cutting Doug Cutting
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: