Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14148

HDFS OIV ReverseXML SnapshotSection parser throws exception when there are more than one snapshottable directory

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.3.0, 3.2.1, 3.1.3
    • Component/s: hdfs
    • Labels:
      None

      Description

      The current HDFS OIV tool doesn't seem to support snapshot well when reversing XML back to binary.

      $ hdfs oiv -i fsimage_0000000000000026542.xml -o reverse.bin -p ReverseXML
      OfflineImageReconstructor failed: Found unknown XML keys in <dir>: dir
      java.io.IOException: Found unknown XML keys in <dir>: dir
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
      
      $ grep -n "<dir>" fsimage_0000000000000026542.xml
      228:<INodeReferenceSection></INodeReferenceSection><SnapshotSection><snapshotCounter>2</snapshotCounter><numSnapshots>2</numSnapshots><snapshottableDir><dir>20492</dir><dir>20495</dir></snapshottableDir>
      

      This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6

      This problem can be easily reproduced when there are at least TWO snapshot-enabled directories. Apply HDFS-14148.test.001.patch and run TestOfflineImageViewer#testReverseXmlRoundTrip() to see my point.

        Attachments

        1. fsimage_0000000000000000024
          1 kB
          Siyao Meng
        2. fsimage_0000000000000000441
          4 kB
          Siyao Meng
        3. HDFS-14148.test.001.patch
          1 kB
          Siyao Meng

          Issue Links

            Activity

              People

              • Assignee:
                smeng Siyao Meng
                Reporter:
                smeng Siyao Meng
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: