[HDFS-4850] fix OfflineImageViewer to work on fsimages with empty files or snapshots

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.1.0-beta
• Component/s: tools
• Labels: snapshot
• Target Version/s: 2.1.0-beta
• Hadoop Flags: Reviewed
• Assignee: Jing Zhao
• Reporter: Stephen Chu
• Created: 25/May/13 00:18
• Updated: 18/Feb/14 22:20
• Resolved: 05/Jun/13 23:56

Description

I deployed hadoop-trunk HDFS and created /user/schu/. I then forced a checkpoint, fetched the fsimage, and ran the default OfflineImageViewer successfully on the fsimage.

schu-mbp:~ schu$ hdfs oiv -i fsimage_0000000000000000004 -o oiv_out_1
schu-mbp:~ schu$ cat oiv_out_1
drwxr-xr-x  -     schu supergroup          0 2013-05-24 16:59 /
drwxr-xr-x  -     schu supergroup          0 2013-05-24 16:59 /user
drwxr-xr-x  -     schu supergroup          0 2013-05-24 16:59 /user/schu
schu-mbp:~ schu$ 

I then touched an empty file /user/schu/testFile1

schu-mbp:~ schu$ hadoop fs -lsr /
lsr: DEPRECATED: Please use 'ls -R' instead.
drwxr-xr-x   - schu supergroup          0 2013-05-24 16:59 /user
drwxr-xr-x   - schu supergroup          0 2013-05-24 17:00 /user/schu
-rw-r--r--   1 schu supergroup          0 2013-05-24 17:00 /user/schu/testFile1

and forced another checkpoint, fetched the fsimage, and reran the OfflineImageViewer. I encountered a NegativeArraySizeException:

schu-mbp:~ schu$ hdfs oiv -i fsimage_0000000000000000008 -o oiv_out_2
Input ended unexpectedly.
2013-05-24 17:01:13,622 ERROR [main] offlineImageViewer.OfflineImageViewer (OfflineImageViewer.java:go(140)) - image loading failed at offset 402
Exception in thread "main" java.lang.NegativeArraySizeException
	at org.apache.hadoop.io.Text.readString(Text.java:458)
	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processPermission(ImageLoaderCurrent.java:370)
	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processINode(ImageLoaderCurrent.java:671)
	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processChildren(ImageLoaderCurrent.java:557)
	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processDirectoryWithSnapshot(ImageLoaderCurrent.java:464)
	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processDirectoryWithSnapshot(ImageLoaderCurrent.java:470)
	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processDirectoryWithSnapshot(ImageLoaderCurrent.java:470)
	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processLocalNameINodesWithSnapshot(ImageLoaderCurrent.java:444)
	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processINodes(ImageLoaderCurrent.java:398)
	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.loadImage(ImageLoaderCurrent.java:199)
	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewer.go(OfflineImageViewer.java:136)
	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewer.main(OfflineImageViewer.java:260)

This is reproducible. I've reproduced this scenario after formatting HDFS and restarting and touching an empty file /testFile1.

Attached are the data dirs, the fsimage before creating the empty file (fsimage_0000000000000000004) and the fsimage afterwards (fsimage_0000000000000000004) and their outputs, oiv_out_1 and oiv_out_2 respectively.

The oiv_out_2 does not include the empty /user/schu/testFile1.

I don't run into this problem using hadoop-2.0.4-alpha.

Issue Links

• breaks: HDFS-5961 OIV cannot load fsimages containing a symbolic link