Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5853

ChecksumFileSystem.getContentSummary() including contents for crc files

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • None
    • None
    • None
    • None

    Description

      Trying to track down some differences in Hive statistics between hadoop-1/hadoop-2. It looks like although ChecksumFileSystem.listStatus() filters out CRC files, getContentSummary() falls back to using the FilterFileSystem.getContentSummary() implementation, which calls fs.getContentSummary(). The underlying fs may not have the same filters as the ChecksumFileSystem and so the CRC files can get included in the content summary.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-10425 Incompatible behavior of LocalFileSystem:getContentSummary

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            People

              Unassigned Unassigned
              jdere Jason Dere
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: