Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-9373

[replication] data loss because replication doesn't expect partial reads

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.95.2
    • Fix Version/s: 0.98.0, 0.96.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When I see this in the logs it often means we got a partial read and then we have the wrong offset when reading the rest of the file

      2013-08-28 23:16:07,182 ERROR [ReplicationExecutor-0.replicationSource,1-jdec2hbase0403-5,60020,1377730319617] org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader: Invalid PB while reading WAL, probably an unexpected EOF, ignoring
      com.google.protobuf.InvalidProtocolBufferException: Protocol message tag had invalid wire type.
              at com.google.protobuf.InvalidProtocolBufferException.invalidWireType(InvalidProtocolBufferException.java:99)
              at com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:498)
              at com.google.protobuf.GeneratedMessage.parseUnknownField(GeneratedMessage.java:193)
              at org.apache.hadoop.hbase.protobuf.generated.WALProtos$WALKey.<init>(WALProtos.java:686)
              at org.apache.hadoop.hbase.protobuf.generated.WALProtos$WALKey.<init>(WALProtos.java:644)
              at org.apache.hadoop.hbase.protobuf.generated.WALProtos$WALKey$1.parsePartialFrom(WALProtos.java:771)
              at org.apache.hadoop.hbase.protobuf.generated.WALProtos$WALKey$1.parsePartialFrom(WALProtos.java:766)
              at org.apache.hadoop.hbase.protobuf.generated.WALProtos$WALKey$Builder.mergeFrom(WALProtos.java:1444)
              at org.apache.hadoop.hbase.protobuf.generated.WALProtos$WALKey$Builder.mergeFrom(WALProtos.java:1218)
              at com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:220)
              at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:912)
              at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:267)
              at com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:290)
              at com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:926)
              at com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:296)
              at com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:918)
              at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:197)
              at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:98)
              at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:89)
              at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:390)
              at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:298)
      

        Attachments

        1. 9373.txt
          4 kB
          Michael Stack
        2. 9373-v2.txt
          4 kB
          Jean-Daniel Cryans
        3. 9373-v3.txt
          7 kB
          Jean-Daniel Cryans
        4. 9373-v4.patch
          7 kB
          Jean-Daniel Cryans

          Activity

            People

            • Assignee:
              jdcryans Jean-Daniel Cryans
              Reporter:
              jdcryans Jean-Daniel Cryans
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: