Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-15430

Failed taking snapshot - Manifest proto-message too large

    XMLWordPrintableJSON

    Details

    • Release Note:
      Failed taking snapshot - Manifest proto-message too large. add property ("snapshot.manifest.size.limit") to change max size of proto-message
    • Tags:
      snapshot, protocol buffer
    • Flags:
      Patch

      Description

      the size of a protobuf message is 64MB (default). but the size of snapshot meta is over 64MB.

      Caused by: com.google.protobuf.InvalidProtocolBufferException via Failed taking snapshot

      { ss=snapshot_xxx table=xxx type=FLUSH }

      due to exception:Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit.:com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit.
      at org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83)
      at org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:307)
      at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:341)
      ... 10 more
      Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit.
      at com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
      at com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
      at com.google.protobuf.CodedInputStream.readRawBytes(CodedInputStream.java:811)
      at com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:329)
      at org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo.<init>(HBaseProtos.java:3767)
      at org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo.<init>(HBaseProtos.java:3699)
      at org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo$1.parsePartialFrom(HBaseProtos.java:3815)
      at org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo$1.parsePartialFrom(HBaseProtos.java:3810)
      at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
      at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.<init>(SnapshotProtos.java:1152)
      at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.<init>(SnapshotProtos.java:1094)
      at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1201)
      at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1196)
      at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
      at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.<init>(SnapshotProtos.java:3858)
      at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.<init>(SnapshotProtos.java:3792)
      at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3894)
      at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3889)
      at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
      at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
      at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
      at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
      at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.parseFrom(SnapshotProtos.java:4094)
      at org.apache.hadoop.hbase.snapshot.SnapshotManifest.readDataManifest(SnapshotManifest.java:433)
      at org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:273)
      at org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
      at org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifySnapshot(MasterSnapshotVerifier.java:106

        Attachments

        1. hbase-15430.patch
          2 kB
          JunHo Cho
        2. hbase-15430-v1.patch
          8 kB
          JunHo Cho
        3. hbase-15430-v2.patch
          8 kB
          JunHo Cho
        4. hbase-15430-v3.branch.0.98.patch
          8 kB
          JunHo Cho
        5. HBASE-15430-v4.patch
          8 kB
          Matteo Bertozzi
        6. HBASE-15430-v5.patch
          10 kB
          JunHo Cho
        7. HBASE-15430-v6.patch
          10 kB
          Matteo Bertozzi

          Activity

            People

            • Assignee:
              miloveme JunHo Cho
              Reporter:
              miloveme JunHo Cho
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: