Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10797

Disk usage summary of snapshots causes renamed blocks to get counted twice

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: snapshots
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Disk usage summaries previously incorrectly counted files twice if they had been renamed (including files moved to Trash) since being snapshotted. Summaries now include current data plus snapshotted data that is no longer under the directory either due to deletion or being moved outside of the directory.
      Show
      Disk usage summaries previously incorrectly counted files twice if they had been renamed (including files moved to Trash) since being snapshotted. Summaries now include current data plus snapshotted data that is no longer under the directory either due to deletion or being moved outside of the directory.

      Description

      DirectoryWithSnapshotFeature.computeContentSummary4Snapshot calculates how much disk usage is used by a snapshot by tallying up the files in the snapshot that have since been deleted (that way it won't overlap with regular files whose disk usage is computed separately). However that is determined from a diff that shows moved (to Trash or otherwise) or renamed files as a deletion and a creation operation that may overlap with the list of blocks. Only the deletion operation is taken into consideration, and this causes those blocks to get represented twice in the disk usage tallying.

        Attachments

        1. HDFS-10797.010.patch
          25 kB
          Sean Mackrory
        2. HDFS-10797.010.patch
          25 kB
          Sean Mackrory
        3. HDFS-10797.009.patch
          25 kB
          Sean Mackrory
        4. HDFS-10797.008.patch
          24 kB
          Sean Mackrory
        5. HDFS-10797.007.patch
          24 kB
          Sean Mackrory
        6. HDFS-10797.006.patch
          25 kB
          Sean Mackrory
        7. HDFS-10797.005.patch
          21 kB
          Sean Mackrory
        8. HDFS-10797.004.patch
          11 kB
          Sean Mackrory
        9. HDFS-10797.003.patch
          7 kB
          Sean Mackrory
        10. HDFS-10797.002.patch
          7 kB
          Sean Mackrory
        11. HDFS-10797.001.patch
          6 kB
          Sean Mackrory

          Issue Links

            Activity

              People

              • Assignee:
                mackrorysd Sean Mackrory
                Reporter:
                mackrorysd Sean Mackrory
              • Votes:
                0 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: