Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10797

Disk usage summary of snapshots causes renamed blocks to get counted twice

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.8.0
    • 2.8.0, 3.0.0-alpha2
    • snapshots
    • None
    • Reviewed
    • Hide
      Disk usage summaries previously incorrectly counted files twice if they had been renamed (including files moved to Trash) since being snapshotted. Summaries now include current data plus snapshotted data that is no longer under the directory either due to deletion or being moved outside of the directory.
      Show
      Disk usage summaries previously incorrectly counted files twice if they had been renamed (including files moved to Trash) since being snapshotted. Summaries now include current data plus snapshotted data that is no longer under the directory either due to deletion or being moved outside of the directory.

    Description

      DirectoryWithSnapshotFeature.computeContentSummary4Snapshot calculates how much disk usage is used by a snapshot by tallying up the files in the snapshot that have since been deleted (that way it won't overlap with regular files whose disk usage is computed separately). However that is determined from a diff that shows moved (to Trash or otherwise) or renamed files as a deletion and a creation operation that may overlap with the list of blocks. Only the deletion operation is taken into consideration, and this causes those blocks to get represented twice in the disk usage tallying.

      Attachments

        1. HDFS-10797.001.patch
          6 kB
          Sean Mackrory
        2. HDFS-10797.002.patch
          7 kB
          Sean Mackrory
        3. HDFS-10797.003.patch
          7 kB
          Sean Mackrory
        4. HDFS-10797.004.patch
          11 kB
          Sean Mackrory
        5. HDFS-10797.005.patch
          21 kB
          Sean Mackrory
        6. HDFS-10797.006.patch
          25 kB
          Sean Mackrory
        7. HDFS-10797.007.patch
          24 kB
          Sean Mackrory
        8. HDFS-10797.008.patch
          24 kB
          Sean Mackrory
        9. HDFS-10797.009.patch
          25 kB
          Sean Mackrory
        10. HDFS-10797.010.patch
          25 kB
          Sean Mackrory
        11. HDFS-10797.010.patch
          25 kB
          Sean Mackrory

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. HDFS-11661 GetContentSummary uses excessive amounts of memory

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          is broken by

          Bug - A problem which impairs or prevents the functions of the product. HDFS-11515 -du throws ConcurrentModificationException

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-11705 BUG: Inconsistent storagespace for directory

          • Major - Major loss of function.
          • Open

          Activity

            People

              mackrorysd Sean Mackrory
              mackrorysd Sean Mackrory
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: