Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6293

Issues with OIV processing PB-based fsimages

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 2.4.0
    • 2.5.0
    • None
    • None
    • Reviewed
    • Hide
      Set "dfs.namenode.legacy-oiv-image.dir" to an appropriate directory to make standby name node or secondary name node save its file system state in the old fsimage format during checkpointing. This image can be used for offline analysis using the OfflineImageViewer. Use the "hdfs oiv_legacy" command to process the old fsimage format.
      Show
      Set "dfs.namenode.legacy-oiv-image.dir" to an appropriate directory to make standby name node or secondary name node save its file system state in the old fsimage format during checkpointing. This image can be used for offline analysis using the OfflineImageViewer. Use the "hdfs oiv_legacy" command to process the old fsimage format.

    Description

      There are issues with OIV when processing fsimages in protobuf.

      Due to the internal layout changes introduced by the protobuf-based fsimage, OIV consumes excessive amount of memory. We have tested with a fsimage with about 140M files/directories. The peak heap usage when processing this image in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of heap (max new size was 1GB). It should be possible to process any image with the default heap size of 1.5GB.

      Another issue is the complete change of format/content in OIV's XML output. I also noticed that the secret manager section has no tokens while there were unexpired tokens in the original image (pre-2.4.0). I did not check whether they were also missing in the new pb fsimage.

      Attachments

        1. HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch
          157 kB
          Kihwal Lee
        2. HDFS-6293_sbn_ckpt_retention.patch
          66 kB
          Kihwal Lee
        3. HDFS-6293.000.patch
          19 kB
          Haohui Mai
        4. HDFS-6293.001.patch
          20 kB
          Haohui Mai
        5. HDFS-6293.002-save-deprecated-fsimage.patch
          61 kB
          Haohui Mai
        6. HDFS-6293.branch-2.patch
          163 kB
          Kihwal Lee
        7. HDFS-6293.trunk.patch
          163 kB
          Kihwal Lee
        8. HDFS-6293.trunk.patch
          163 kB
          Kihwal Lee
        9. HDFS-6293.v2.branch-2.patch
          162 kB
          Kihwal Lee
        10. HDFS-6293.v2.trunk.patch
          163 kB
          Kihwal Lee
        11. Heap Histogram.html
          137 kB
          Kihwal Lee

        Issue Links

          breaks

          Test - A new unit, integration or system test. HDFS-6419 TestBookKeeperHACheckpoints#TestSBNCheckpoints fails on trunk

          • Major - Major loss of function.
          • Closed
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. HDFS-6914 Resolve huge memory consumption Issue with OIV processing PB-based fsimages

          • Major - Major loss of function.
          • Patch Available
          is related to

          Sub-task - The sub-task of the issue HDFS-6673 Add delimited format support to PB OIV tool

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HDFS-11709 StandbyCheckpointer should handle an non-existing legacyOivImageDir gracefully

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-6400 Cannot execute "hdfs oiv_legacy"

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-6310 PBImageXmlWriter should output information about Delegation Tokens

          • Major - Major loss of function.
          • Patch Available

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-6519 Document oiv_legacy command

          • Major - Major loss of function.
          • Closed

          Activity

            People

              kihwal Kihwal Lee
              kihwal Kihwal Lee
              Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: