Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16150

checksumFS doesn't wrap concat(): concatenated files don't have checksums

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.2.0
    • Fix Version/s: 3.3.0
    • Component/s: fs
    • Labels:
      None

      Description

      Followon from HADOOP-16107. FilterFS passes through the concat operation, and checksum FS doesn't override that call -so files created through concat do not have checksums.

      If people are using a checksummed fs directly with the expectations that they will, that expectation is not being met.

      What to do?

      • fail always?
      • fail if checksums are enabled?
      • try and implement the concat operation from raw local up at the checksum level

      append() just gives up always; doing the same for concat would be the simplest. Again, brings us back to "need a way to see if an FS supports a feature before invocation", here checksum fs would reject append and concat

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                stevel@apache.org Steve Loughran
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: