Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-17452

Failed taking snapshot - region Manifest proto-message too large

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 1.4.0, 2.0.0
    • Component/s: snapshots
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      With MOB, it is possible that there are lots of files under mobdir. When taking snapshot, the region manifest could be very large. Similar to HBASE-15430, we are seeing use cases running into the following exception:

      Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit.
              at com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
              at com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
              at com.google.protobuf.CodedInputStream.readRawBytes(CodedInputStream.java:811)
              at com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:329)
              at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$StoreFile.<init>(SnapshotProtos.java:1313)
              at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$StoreFile.<init>(SnapshotProtos.java:1263)
              at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$StoreFile$1.parsePartialFrom(SnapshotProtos.java:1364)
              at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$StoreFile$1.parsePartialFrom(SnapshotProtos.java:1359)
              at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
              at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$FamilyFiles.<init>(SnapshotProtos.java:2161)
              at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$FamilyFiles.<init>(SnapshotProtos.java:2103)
              at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$FamilyFiles$1.parsePartialFrom(SnapshotProtos.java:2197)
              at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$FamilyFiles$1.parsePartialFrom(SnapshotProtos.java:2192)
              at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
              at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.<init>(SnapshotProtos.java:1165)
              at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.<init>(SnapshotProtos.java:1094)
              at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1201)
              at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1196)
              at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
              at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
              at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
              at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
              at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.parseFrom(SnapshotProtos.java:3111)
              at org.apache.hadoop.hbase.snapshot.SnapshotManifestV2$2.call(SnapshotManifestV2.java:139)
              at org.apache.hadoop.hbase.snapshot.SnapshotManifestV2$2.call(SnapshotManifestV2.java:134)
              at java.util.concurrent.FutureTask.run(FutureTask.java:262)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
              at java.util.concurrent.FutureTask.run(FutureTask.java:262)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              ... 1 more
      

        Attachments

        1. 17452.branch-1.txt
          11 kB
          Ted Yu
        2. HBASE-17452-master-001.patch
          10 kB
          Hua Xiang

          Activity

            People

            • Assignee:
              huaxiang Hua Xiang
              Reporter:
              huaxiang Hua Xiang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: