[HDFS-14148] HDFS OIV ReverseXML SnapshotSection parser throws exception when there are more than one snapshottable directory

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.3.0, 3.2.1, 3.1.3
• Component/s: hdfs
• Target Version/s: 3.3.0, 3.2.1, 3.1.3
• Assignee: Siyao Meng
• Reporter: Siyao Meng
• Created: 13/Dec/18 22:38
• Updated: 02/Oct/19 17:15
• Resolved: 13/Aug/19 00:26

Description

The current HDFS OIV tool doesn't seem to support snapshot well when reversing XML back to binary.

$ hdfs oiv -i fsimage_0000000000000026542.xml -o reverse.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in <dir>: dir
java.io.IOException: Found unknown XML keys in <dir>: dir
	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)

$ grep -n "<dir>" fsimage_0000000000000026542.xml
228:<INodeReferenceSection></INodeReferenceSection><SnapshotSection><snapshotCounter>2</snapshotCounter><numSnapshots>2</numSnapshots><snapshottableDir><dir>20492</dir><dir>20495</dir></snapshottableDir>

This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6

This problem can be easily reproduced when there are at least TWO snapshot-enabled directories. Apply HDFS-14148.test.001.patch and run TestOfflineImageViewer#testReverseXmlRoundTrip() to see my point.

Issue Links

• is broken by: HDFS-9835 OIV: add ReverseXML processor which reconstructs an fsimage from an XML file
• links to: GitHub Pull Request #1274
• links to: GitHub Pull Request #1276