Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4850

fix OfflineImageViewer to work on fsimages with empty files or snapshots

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.1.0-beta
    • Component/s: tools
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      I deployed hadoop-trunk HDFS and created /user/schu/. I then forced a checkpoint, fetched the fsimage, and ran the default OfflineImageViewer successfully on the fsimage.

      schu-mbp:~ schu$ hdfs oiv -i fsimage_0000000000000000004 -o oiv_out_1
      schu-mbp:~ schu$ cat oiv_out_1
      drwxr-xr-x  -     schu supergroup          0 2013-05-24 16:59 /
      drwxr-xr-x  -     schu supergroup          0 2013-05-24 16:59 /user
      drwxr-xr-x  -     schu supergroup          0 2013-05-24 16:59 /user/schu
      schu-mbp:~ schu$ 
      

      I then touched an empty file /user/schu/testFile1

      schu-mbp:~ schu$ hadoop fs -lsr /
      lsr: DEPRECATED: Please use 'ls -R' instead.
      drwxr-xr-x   - schu supergroup          0 2013-05-24 16:59 /user
      drwxr-xr-x   - schu supergroup          0 2013-05-24 17:00 /user/schu
      -rw-r--r--   1 schu supergroup          0 2013-05-24 17:00 /user/schu/testFile1
      

      and forced another checkpoint, fetched the fsimage, and reran the OfflineImageViewer. I encountered a NegativeArraySizeException:

      schu-mbp:~ schu$ hdfs oiv -i fsimage_0000000000000000008 -o oiv_out_2
      Input ended unexpectedly.
      2013-05-24 17:01:13,622 ERROR [main] offlineImageViewer.OfflineImageViewer (OfflineImageViewer.java:go(140)) - image loading failed at offset 402
      Exception in thread "main" java.lang.NegativeArraySizeException
      	at org.apache.hadoop.io.Text.readString(Text.java:458)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processPermission(ImageLoaderCurrent.java:370)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processINode(ImageLoaderCurrent.java:671)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processChildren(ImageLoaderCurrent.java:557)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processDirectoryWithSnapshot(ImageLoaderCurrent.java:464)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processDirectoryWithSnapshot(ImageLoaderCurrent.java:470)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processDirectoryWithSnapshot(ImageLoaderCurrent.java:470)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processLocalNameINodesWithSnapshot(ImageLoaderCurrent.java:444)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processINodes(ImageLoaderCurrent.java:398)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.loadImage(ImageLoaderCurrent.java:199)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewer.go(OfflineImageViewer.java:136)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewer.main(OfflineImageViewer.java:260)
      

      This is reproducible. I've reproduced this scenario after formatting HDFS and restarting and touching an empty file /testFile1.

      Attached are the data dirs, the fsimage before creating the empty file (fsimage_0000000000000000004) and the fsimage afterwards (fsimage_0000000000000000004) and their outputs, oiv_out_1 and oiv_out_2 respectively.

      The oiv_out_2 does not include the empty /user/schu/testFile1.

      I don't run into this problem using hadoop-2.0.4-alpha.

        Attachments

        1. datadirs.tar.gz
          3 kB
          Stephen Chu
        2. fsimage_0000000000000000004
          0.4 kB
          Stephen Chu
        3. fsimage_0000000000000000008
          0.4 kB
          Stephen Chu
        4. HDFS-4850.000.patch
          0.8 kB
          Jing Zhao
        5. HDFS-4850.001.patch
          18 kB
          Jing Zhao
        6. oiv_out_1
          0.2 kB
          Stephen Chu
        7. oiv_out_2
          0.2 kB
          Stephen Chu

          Issue Links

            Activity

              People

              • Assignee:
                jingzhao Jing Zhao
                Reporter:
                schu Stephen Chu
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: