HBase
  1. HBase
  2. HBASE-8615

HLog Compression may fail due to Hadoop fs input stream returning partial bytes

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.98.0, 0.95.2
    • Component/s: Replication
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In a recent test run, I noticed the following in test output:

      2013-05-24 22:01:02,424 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] fs.HFileSystem$ReorderWALBlocks(327): /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428 is an HLog file, so reordering blocks, last hostname will be:kiyo.gq1.ygridcore.net
      2013-05-24 22:01:02,429 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 132235, fileLength: 132243, trailerPresent: true
      2013-05-24 22:01:02,438 ERROR [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] wal.ProtobufLogReader(236): Error  while reading 691 WAL KVs; started reading at 53272 and read up to 65538
      2013-05-24 22:01:02,438 WARN  [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(324): 2 Got:
      java.io.IOException: Error  while reading 691 WAL KVs; started reading at 53272 and read up to 65538
              at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:237)
              at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:96)
              at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:89)
              at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:404)
              at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:320)
      Caused by: java.lang.IndexOutOfBoundsException: index (30062) must be less than size (1)
              at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
              at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
              at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.get(LRUDictionary.java:124)
              at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.access$000(LRUDictionary.java:71)
              at org.apache.hadoop.hbase.regionserver.wal.LRUDictionary.getEntry(LRUDictionary.java:42)
              at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.readIntoArray(WALCellCodec.java:210)
              at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.parseCell(WALCellCodec.java:184)
              at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:46)
              at org.apache.hadoop.hbase.regionserver.wal.WALEdit.readFromCells(WALEdit.java:213)
              at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:217)
              ... 4 more
      2013-05-24 22:01:02,439 DEBUG [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2] regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 times 10
      

      Will attach test output.

      1. org.apache.hadoop.hbase.replication.TestReplicationQueueFailoverCompressed-output.txt
        744 kB
        Ted Yu
      2. HBASE-8615-test.patch
        3 kB
        Jean-Daniel Cryans
      3. 172.21.3.117%2C60020%2C1375222888304.1375222894855.zip
        8.20 MB
        Jean-Daniel Cryans
      4. 8615-v2.txt
        5 kB
        Ted Yu
      5. 8615-v3.txt
        4 kB
        Ted Yu
      6. 8615-v4.txt
        6 kB
        Ted Yu
      7. 8615-v5.txt
        6 kB
        Ted Yu

        Issue Links

          Activity

          Ted Yu created issue -
          Ted Yu made changes -
          Field Original Value New Value
          Attachment org.apache.hadoop.hbase.replication.TestReplicationQueueFailoverCompressed-output.txt [ 12584767 ]
          Ted Yu made changes -
          Component/s Replication [ 12313650 ]
          Jean-Daniel Cryans made changes -
          Assignee Jean-Daniel Cryans [ jdcryans ]
          Jean-Daniel Cryans made changes -
          Summary TestReplicationQueueFailoverCompressed#queueFailover fails on hadoop 2.0 due to IndexOutOfBoundsException HLog Compression fails in mysterious ways (working title)
          Fix Version/s 0.98.0 [ 12323143 ]
          Fix Version/s 0.95.2 [ 12320040 ]
          Priority Major [ 3 ] Critical [ 2 ]
          Jean-Daniel Cryans made changes -
          Attachment HBASE-8615-test.patch [ 12595524 ]
          Attachment 172.21.3.117%2C60020%2C1375222888304.1375222894855.zip [ 12595525 ]
          Jean-Daniel Cryans made changes -
          Link This issue is depended upon by HBASE-9061 [ HBASE-9061 ]
          Jean-Daniel Cryans made changes -
          Fix Version/s 0.96.0 [ 12324822 ]
          Fix Version/s 0.95.2 [ 12320040 ]
          Ted Yu made changes -
          Attachment 8615-v2.txt [ 12596744 ]
          Ted Yu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Jean-Daniel Cryans [ jdcryans ] Ted Yu [ yuzhihong@gmail.com ]
          Ted Yu made changes -
          Attachment 8615-v3.txt [ 12596761 ]
          Ted Yu made changes -
          Attachment 8615-v4.txt [ 12596963 ]
          Ted Yu made changes -
          Attachment 8615-v5.txt [ 12596980 ]
          Ted Yu made changes -
          Summary HLog Compression fails in mysterious ways (working title) HLog Compression may fail due to Hadoop fs input stream returning partial bytes
          Ted Yu made changes -
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 0.95.2 [ 12320040 ]
          Fix Version/s 0.96.0 [ 12324822 ]
          stack made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          stack made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Ted Yu
              Reporter:
              Ted Yu
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development