Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6102

Lower the default maximum items per directory to fix PB fsimage loading

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.4.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change

      Description

      Found by Stephen Chu during testing. We were creating a bunch of directories in a single directory to blow up the fsimage size, and it ends up we hit this error when trying to load a very large fsimage:

      2014-03-13 13:57:03,901 INFO org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode: Loading 24523605 INodes.
      2014-03-13 13:57:59,038 ERROR org.apache.hadoop.hdfs.server.namenode.FSImage: Failed to load image from FSImageFile(file=/dfs/nn/current/fsimage_0000000000024532742, cpktTxId=0000000000024532742)
      com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large.  May be malicious.  Use CodedInputStream.setSizeLimit() to increase the size limit.
              at com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
              at com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
              at com.google.protobuf.CodedInputStream.readRawByte(CodedInputStream.java:769)
              at com.google.protobuf.CodedInputStream.readRawVarint64(CodedInputStream.java:462)
              at com.google.protobuf.CodedInputStream.readUInt64(CodedInputStream.java:188)
              at org.apache.hadoop.hdfs.server.namenode.FsImageProto$INodeDirectorySection$DirEntry.<init>(FsImageProto.java:9839)
              at org.apache.hadoop.hdfs.server.namenode.FsImageProto$INodeDirectorySection$DirEntry.<init>(FsImageProto.java:9770)
              at org.apache.hadoop.hdfs.server.namenode.FsImageProto$INodeDirectorySection$DirEntry$1.parsePartialFrom(FsImageProto.java:9901)
              at org.apache.hadoop.hdfs.server.namenode.FsImageProto$INodeDirectorySection$DirEntry$1.parsePartialFrom(FsImageProto.java:9896)
              at 52)
      ...
      

      Some further research reveals there's a 64MB max size per PB message, which seems to be what we're hitting here.

        Attachments

        1. hdfs-6102-1.patch
          4 kB
          Andrew Wang
        2. hdfs-6102-2.patch
          7 kB
          Andrew Wang

          Issue Links

            Activity

              People

              • Assignee:
                andrew.wang Andrew Wang
                Reporter:
                andrew.wang Andrew Wang
              • Votes:
                0 Vote for this issue
                Watchers:
                17 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: