Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-25507

Leak of ESTABLISHED sockets when compaction encountered "java.io.IOException: Invalid HFile block magic"

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0-alpha-1, 2.4.1
    • 3.0.0-alpha-1, 2.3.5, 2.4.2
    • Compaction
    • None
    • Reviewed

    Description

      Recently, we found socket leaks on our production cluster. The leaked sockets are in ESTABLISHED state. We found this happened on RS who owned a particular region from our analysis of metrics monitor and logs. RS without this region works normally. 

      On the RS who owns the particular region, we found Exceptions as follows,

      java.io.IOException: java.io.IOException: Could not seek StoreFileScanner[org.apache.hadoop.hbase.ioo
      .HalfStoreFileReader$1@5388be2f, cur=null] to key org.apache.hadoop.hbase.CellUtil$FirstOnRowDeleteFF
      amilyCell@25aa56fd
              at org.apache.hadoop.hbase.regionserver.StoreScanner.parallelSeek(StoreScanner.java:1128)
              at org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:437)
              at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:329)
              at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:302)
              at org.apache.hadoop.hbase.regionserver.compactions.Compactor.createScanner(Compactor.java:88
      06)
              at org.apache.hadoop.hbase.regionserver.compactions.StripeCompactor$StripeInternalScannerFacc
      tory.createScanner(StripeCompactor.java:82)
              at org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:316)
              at org.apache.hadoop.hbase.regionserver.compactions.StripeCompactor.compact(StripeCompactor..
      java:120)
              at org.apache.hadoop.hbase.regionserver.compactions.StripeCompactionPolicy$SplitStripeCompacc
      tionRequest.execute(StripeCompactionPolicy.java:662)
              at org.apache.hadoop.hbase.regionserver.StripeStoreEngine$StripeCompaction.compact(StripeStoo
      reEngine.java:114)
              at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1461)
              at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2121)
              at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(Comm
      pactSplitThread.java:519)
              at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitt
      Thread.java:555)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: Could not seek StoreFileScanner[org.apache.hadoop.hbase.io.HalfStoree
      FileReader$1@5388be2f, cur=null] to key org.apache.hadoop.hbase.CellUtil$FirstOnRowDeleteFamilyCell@@
      25aa56fd
              at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:229)
              at org.apache.hadoop.hbase.regionserver.handler.ParallelSeekHandler.process(ParallelSeekHandd
      ler.java:56)
              at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
              ... 3 more
      Caused by: java.io.IOException: Invalid HFile block magic: \x00\x00\x00\x00\x00\x00\x00\x00
              at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:159)
              at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:171)
              at org.apache.hadoop.hbase.io.hfile.HFileBlock.createFromBuff(HFileBlock.java:333)
              at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlockk
      .java:1753)
              at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:155
      52)
              at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:539)
              at org.apache.hadoop.hbase.io.hfile.HFileScannerImpl.readAndUpdateNewBlock(HFileScannerImpl..
      java:737)
              at org.apache.hadoop.hbase.io.hfile.HFileScannerImpl.seekTo(HFileScannerImpl.java:726)
              at org.apache.hadoop.hbase.io.HalfStoreFileReader$1.seekTo(HalfStoreFileReader.java:161)
              at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.javaa
      :315)
              at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:211)
              ... 5 more
      

      The count of established sockets is always increasing, see picture,

       

       

       

       

       

       

       

      Attachments

        1. problem-region-move-logs.png
          1002 kB
          Xiaolin Ha
        2. increasing-of-established-sockets-image.png
          238 kB
          Xiaolin Ha
        3. errorlogs.png
          432 kB
          Xiaolin Ha

        Issue Links

          Activity

            People

              Xiaolin Ha Xiaolin Ha
              Xiaolin Ha Xiaolin Ha
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: