HBase
  1. HBase
  2. HBASE-9818

NPE in HFileBlock#AbstractFSReader#readAtOffset

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      HFileBlock#istream seems to be null. I was wondering should we hide FSDataInputStreamWrapper#useHBaseChecksum.

      By the way, this happened when online schema change is enabled (encoding)

      2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] regionserver.HRegionServer:
      java.lang.NullPointerException
              at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
              at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
              at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
              at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
              at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
              at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
              at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
              at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
              at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
              at org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
              at org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
              at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
              at org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
              at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
              at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
              at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
              at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
              at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
              at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
              at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
              at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
              at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
              at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
              at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
              at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
              at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
              at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
              at java.lang.Thread.run(Thread.java:724)
      2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] regionserver.HRegionServer:
      org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected nextCallSeq: 53438 But the nextCallSeq got from client: 53437; request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: false next_call_seq: 53437
              at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
              at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
              at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
              at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
              at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
              at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
              at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
              at java.lang.Thread.run(Thread.java:724)
      
      1. 9818-combined.txt
        6 kB
        Ted Yu
      2. 9818-combined-2.txt
        6 kB
        Ted Yu
      3. 9818-trial.txt
        2 kB
        Ted Yu
      4. 9818-trial2.txt
        2 kB
        Ted Yu
      5. 9818-v2.txt
        4 kB
        Ted Yu
      6. 9818-v3.txt
        5 kB
        Ted Yu
      7. 9818-v4.txt
        4 kB
        Ted Yu
      8. 9818-v5.txt
        5 kB
        Ted Yu
      9. trunk-9818_v1.1.patch
        1.0 kB
        Jimmy Xiang
      10. trunk-9818.patch
        0.9 kB
        Jimmy Xiang

        Activity

        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12613446/9818-combined-2.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7827//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7827//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12613446/9818-combined-2.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 1 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7827//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7827//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7827//console This message is automatically generated.
        Hide
        Jimmy Xiang added a comment -

        Ted Yu, with HBASE-9926, I haven't seen such NPE in my test any more. Can we close this issue? Thanks for digging into it.

        Show
        Jimmy Xiang added a comment - Ted Yu , with HBASE-9926 , I haven't seen such NPE in my test any more. Can we close this issue? Thanks for digging into it.
        Hide
        Jimmy Xiang added a comment -

        That's fine with me.

        Show
        Jimmy Xiang added a comment - That's fine with me.
        Hide
        stack added a comment -

        Maybe we should close this issue then and do the other fixup in a new one dedicated to HFileBlock cleanup?

        Show
        stack added a comment - Maybe we should close this issue then and do the other fixup in a new one dedicated to HFileBlock cleanup?
        Hide
        Jimmy Xiang added a comment -

        Yes, I fixed the main issue in HBASE-9926. The stream is null because the file is closed. There are also some minor issue/enhancement with HFileBlock (more related to this call, not scanner) that can be fixed with this jira.

        Show
        Jimmy Xiang added a comment - Yes, I fixed the main issue in HBASE-9926 . The stream is null because the file is closed. There are also some minor issue/enhancement with HFileBlock (more related to this call, not scanner) that can be fixed with this jira.
        Hide
        stack added a comment -

        Jimmy Xiang Did you open another issue to fix this issue? If so, what was it?

        Show
        stack added a comment - Jimmy Xiang Did you open another issue to fix this issue? If so, what was it?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12613014/9818-combined.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7807//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7807//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12613014/9818-combined.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7807//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7807//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        Patch combining patch v5 and Jimmy's patch.

        Show
        Ted Yu added a comment - Patch combining patch v5 and Jimmy's patch.
        Hide
        Ted Yu added a comment -

        Trial2 didn't use volatile field.
        When wrapper is closed, stack trace is captured in an exception.

        However, 200 iterations of tests didn't trigger NPE.

        Show
        Ted Yu added a comment - Trial2 didn't use volatile field. When wrapper is closed, stack trace is captured in an exception. However, 200 iterations of tests didn't trigger NPE.
        Hide
        Jimmy Xiang added a comment -

        Ted Yu, what do you think about my patch 1.1?

        Show
        Jimmy Xiang added a comment - Ted Yu , what do you think about my patch 1.1?
        Hide
        Ted Yu added a comment -

        Another round of 200 iterations with 9818-trial.txt passed.
        Will continue digging.

        Show
        Ted Yu added a comment - Another round of 200 iterations with 9818-trial.txt passed. Will continue digging.
        Hide
        Jimmy Xiang added a comment -

        Assigned back to Ted since he is still looking into it.

        Show
        Jimmy Xiang added a comment - Assigned back to Ted since he is still looking into it.
        Hide
        Jimmy Xiang added a comment -

        Attached v1.1 that fixed some unit tests failures.

        Show
        Jimmy Xiang added a comment - Attached v1.1 that fixed some unit tests failures.
        Hide
        Jimmy Xiang added a comment -

        Attached "trunk-9818.patch" which makes sure all scanners are closed if a store file is closed.

        Show
        Jimmy Xiang added a comment - Attached "trunk-9818.patch" which makes sure all scanners are closed if a store file is closed.
        Hide
        Ted Yu added a comment -

        TestHFileBlock#testConcurrentReading shows how multiple FSReader's use the same FSDataInputStreamWrapper

        Another solution I can think of is to introduce ref counting for the streams FSDataInputStreamWrapper wraps. The FSDataInputStreamWrapper#close() would decrement the count and would close the underlying stream if count reaches 0.

        Show
        Ted Yu added a comment - TestHFileBlock#testConcurrentReading shows how multiple FSReader's use the same FSDataInputStreamWrapper Another solution I can think of is to introduce ref counting for the streams FSDataInputStreamWrapper wraps. The FSDataInputStreamWrapper#close() would decrement the count and would close the underlying stream if count reaches 0.
        Hide
        Ted Yu added a comment -

        Unfortunately the change in the trial patch seemed to affect timing.
        After 200 iterations, there was still no failure:

        $ grep 'rver.TestHRegion' 9818.out | wc
            200     400   11400
        
        Show
        Ted Yu added a comment - Unfortunately the change in the trial patch seemed to affect timing. After 200 iterations, there was still no failure: $ grep 'rver.TestHRegion' 9818.out | wc 200 400 11400
        Hide
        Ted Yu added a comment -

        I tried to use 9818-trial.txt for detecting where the close() came from.

        First attempt ended in with device full.
        Rerunning the two tests now.

        Show
        Ted Yu added a comment - I tried to use 9818-trial.txt for detecting where the close() came from. First attempt ended in with device full. Rerunning the two tests now.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12612473/9818-v5.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        -1 core tests. The patch failed these unit tests:

        -1 core zombie tests. There are 1 zombie test(s): at org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:488)

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7764//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7764//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12612473/9818-v5.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 1 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. -1 core tests . The patch failed these unit tests: -1 core zombie tests . There are 1 zombie test(s): at org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:488) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7764//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7764//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7764//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12612439/9818-v5.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        -1 core tests. The patch failed these unit tests:

        -1 core zombie tests. There are 1 zombie test(s): at org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:488)

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7760//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7760//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12612439/9818-v5.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 1 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. -1 core tests . The patch failed these unit tests: -1 core zombie tests . There are 1 zombie test(s): at org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:488) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7760//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7760//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7760//console This message is automatically generated.
        Hide
        Jimmy Xiang added a comment -

        I am ok with the change. But I don't think it fixes the issue. If the stream is really closed, instead of NPE, we will get some IOException instead.

        Show
        Jimmy Xiang added a comment - I am ok with the change. But I don't think it fixes the issue. If the stream is really closed, instead of NPE, we will get some IOException instead.
        Hide
        Sergey Shelukhin added a comment -
        +    return useHBaseChecksum ?
        +        new Pair<FSDataInputStream, Boolean>(this.streamNoFsChecksum, useHBaseChecksum) : 
        +          new Pair<FSDataInputStream, Boolean>(this.stream, useHBaseChecksum);
        

        the boolean should be saved to local variable before checking, the value could change and result in no checksum (or both) being used.
        Otherwise +1 from me... hopefully someone else can also take a look.
        I am not 100% sure if this is complete fix or just narrows the window quite a bit

        Show
        Sergey Shelukhin added a comment - + return useHBaseChecksum ? + new Pair<FSDataInputStream, Boolean >( this .streamNoFsChecksum, useHBaseChecksum) : + new Pair<FSDataInputStream, Boolean >( this .stream, useHBaseChecksum); the boolean should be saved to local variable before checking, the value could change and result in no checksum (or both) being used. Otherwise +1 from me... hopefully someone else can also take a look. I am not 100% sure if this is complete fix or just narrows the window quite a bit
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12612405/9818-v5.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        -1 core tests. The patch failed these unit tests:

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7756//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7756//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12612405/9818-v5.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 1 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. -1 core tests . The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7756//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7756//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        Patch v5 fixes javadoc warning

        Show
        Ted Yu added a comment - Patch v5 fixes javadoc warning
        Hide
        Ted Yu added a comment -

        200 iterations of TestHRegion and TestAtomicOperation passed.

        Show
        Ted Yu added a comment - 200 iterations of TestHRegion and TestAtomicOperation passed.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12612315/9818-v4.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7743//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7743//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12612315/9818-v4.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 1 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7743//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7743//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7743//console This message is automatically generated.
        Hide
        Jimmy Xiang added a comment -

        I was wondering how v4 solves the issue. With the patch, if the stream is closed somewhere, instead of a NPE, we may get an IOException saying the stream is closed. If the stream is not closed somewhere, why is it null? Never initialized?

        Show
        Jimmy Xiang added a comment - I was wondering how v4 solves the issue. With the patch, if the stream is closed somewhere, instead of a NPE, we may get an IOException saying the stream is closed. If the stream is not closed somewhere, why is it null? Never initialized?
        Hide
        Ted Yu added a comment -

        Patch v4 unifies FSDataInputStreamWrapper#getStream() and FSDataInputStreamWrapper#shouldUseHBaseChecksum()

        The tests, on Linux, have reached iteration #82.

        Show
        Ted Yu added a comment - Patch v4 unifies FSDataInputStreamWrapper#getStream() and FSDataInputStreamWrapper#shouldUseHBaseChecksum() The tests, on Linux, have reached iteration #82.
        Hide
        Sergey Shelukhin added a comment -

        actually, I looked at it, encapsulating is not such a good idea (returning boolean and stream together), it will not remove the race with close unless a common lock is added. So root cause might be elsewhere...

        Show
        Sergey Shelukhin added a comment - actually, I looked at it, encapsulating is not such a good idea (returning boolean and stream together), it will not remove the race with close unless a common lock is added. So root cause might be elsewhere...
        Hide
        Sergey Shelukhin added a comment -

        logging lgtm. What does it say when it fails

        Show
        Sergey Shelukhin added a comment - logging lgtm. What does it say when it fails
        Hide
        Sergey Shelukhin added a comment -

        is retrying intentional? probably we should find root cause and not just retry.

        Show
        Sergey Shelukhin added a comment - is retrying intentional? probably we should find root cause and not just retry.
        Hide
        Ted Yu added a comment -

        Patch v3 allows TestHRegion and TestAtomicOperation to reach iteration #46.
        Please comment.

        Show
        Ted Yu added a comment - Patch v3 allows TestHRegion and TestAtomicOperation to reach iteration #46. Please comment.
        Hide
        Ted Yu added a comment -

        From https://builds.apache.org/job/PreCommit-HBASE-Build/7739/testReport/junit/org.apache.hadoop.hbase.regionserver.wal/TestHLog/testAppendClose/ :

        Stacktrace
        
        java.lang.OutOfMemoryError: Java heap space
        	at org.apache.hadoop.hdfs.util.LightWeightGSet.<init>(LightWeightGSet.java:81)
        	at org.apache.hadoop.hdfs.server.namenode.BlocksMap.<init>(BlocksMap.java:320)
        	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:223)
        	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:299)
        	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:569)
        	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1479)
        	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:278)
        	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSClusterForTestHLog(HBaseTestingUtility.java:563)
        	at org.apache.hadoop.hbase.regionserver.wal.TestHLog.testAppendClose(TestHLog.java:434)
        

        Looks like an environment issue.
        'stream closed' exception, added in patch, didn't show up.

        Show
        Ted Yu added a comment - From https://builds.apache.org/job/PreCommit-HBASE-Build/7739/testReport/junit/org.apache.hadoop.hbase.regionserver.wal/TestHLog/testAppendClose/ : Stacktrace java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.hdfs.util.LightWeightGSet.<init>(LightWeightGSet.java:81) at org.apache.hadoop.hdfs.server.namenode.BlocksMap.<init>(BlocksMap.java:320) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:223) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:299) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:569) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1479) at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:278) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSClusterForTestHLog(HBaseTestingUtility.java:563) at org.apache.hadoop.hbase.regionserver.wal.TestHLog.testAppendClose(TestHLog.java:434) Looks like an environment issue. 'stream closed' exception, added in patch, didn't show up.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12612207/9818-v2.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.regionserver.wal.TestHLog

        -1 core zombie tests. There are 1 zombie test(s): at org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:486)

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7739//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7739//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12612207/9818-v2.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.wal.TestHLog -1 core zombie tests . There are 1 zombie test(s): at org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:486) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7739//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7739//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7739//console This message is automatically generated.
        Hide
        Sergey Shelukhin added a comment -

        But yeah maybe wrapper method needs to be encapsulated with getting stream together. Maybe there's no real sync issue, and it just needs to not throw, and then on next read it will rebuild the heap. Although it does seem pretty suspect, it has a stream and someone closes it in parallel.

        Show
        Sergey Shelukhin added a comment - But yeah maybe wrapper method needs to be encapsulated with getting stream together. Maybe there's no real sync issue, and it just needs to not throw, and then on next read it will rebuild the heap. Although it does seem pretty suspect, it has a stream and someone closes it in parallel.
        Hide
        Sergey Shelukhin added a comment -

        From logs and code it seems like compactions are closing the stream via wrapper (via long chain of classes)
        Beforehand, they are supposed to notify all scanners to rebuild heap, which is a synchronized method (on StoreScanner) and only then close (next/etc are also synchronized so it should all be properly sequenced). But somehow it's not happening I suspect.
        Also in some cases stacks are during some initialization, not next(), so I haven't looked/not sure how that was supposed to be synched.
        I didn't have a lot of time, so just looked at the code. Looped testWritesWhileGetting 100 times and it never failed locally, which is sad.
        Is there any chance/tool to find out when approx these failures started

        Show
        Sergey Shelukhin added a comment - From logs and code it seems like compactions are closing the stream via wrapper (via long chain of classes) Beforehand, they are supposed to notify all scanners to rebuild heap, which is a synchronized method (on StoreScanner) and only then close (next/etc are also synchronized so it should all be properly sequenced). But somehow it's not happening I suspect. Also in some cases stacks are during some initialization, not next(), so I haven't looked/not sure how that was supposed to be synched. I didn't have a lot of time, so just looked at the code. Looped testWritesWhileGetting 100 times and it never failed locally, which is sad. Is there any chance/tool to find out when approx these failures started
        Hide
        Ted Yu added a comment -

        I am looping TestHRegion and TestAtomicOperation 200 times, respectively.
        Previously TestAtomicOperation failed at iteration #7. Now the tests reach iteration #17 and are running.

        Show
        Ted Yu added a comment - I am looping TestHRegion and TestAtomicOperation 200 times, respectively. Previously TestAtomicOperation failed at iteration #7. Now the tests reach iteration #17 and are running.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12612102/9818-v1.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7734//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7734//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12612102/9818-v1.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7734//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7734//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        Encountered NPE in the else branch (with patch v1):

        2013-11-05 04:23:17,524 DEBUG [Thread-776] compactions.ExploringCompactionPolicy(122): Exploring compaction algorithm has selected 3 files of size 3318 starting at candidate #24 after considering 1884 permutations with 1884 in ratio
        2013-11-05 04:23:17,524 DEBUG [Thread-776] regionserver.HStore(1359): fa933591407f5591c2410908ea5065f9 - colfamily11: Initiating minor compaction
        2013-11-05 04:23:17,524 INFO  [Thread-776] regionserver.HRegion(1288): Starting compaction on colfamily11 in region testRowMutationMultiThreads,,1383625393150.fa933591407f5591c2410908ea5065f9.
        2013-11-05 04:23:17,524 INFO  [Thread-776] regionserver.HStore(1001): Starting compaction of 3 file(s) in colfamily11 of testRowMutationMultiThreads,,1383625393150.fa933591407f5591c2410908ea5065f9. into tmpdir=/grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/data/default/testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/.tmp, totalSize=3.2 K
        2013-11-05 04:23:17,525 DEBUG [Thread-776] compactions.Compactor(147): Compacting file:/grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/data/default/testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/colfamily11/94f3260116304004853721be928701f9, keycount=1, bloomtype=ROW, size=1.1 K, encoding=NONE, seqNum=1331
        2013-11-05 04:23:17,525 DEBUG [Thread-776] compactions.Compactor(147): Compacting file:/grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/data/default/testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/colfamily11/795449ca77094e769ac390749c89f92b, keycount=1, bloomtype=ROW, size=1.1 K, encoding=NONE, seqNum=1342
        2013-11-05 04:23:17,525 DEBUG [Thread-776] compactions.Compactor(147): Compacting file:/grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/data/default/testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/colfamily11/9e2cc96151754cedbc0d461653cbea63, keycount=1, bloomtype=ROW, size=1.1 K, encoding=NONE, seqNum=1353
        2013-11-05 04:23:17,531 DEBUG [Thread-776] regionserver.HRegionFileSystem(338): Committing store file /grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/data/default/testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/.tmp/3ac822791e534965b35b0b3d5022881a as /grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/data/default/testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/colfamily11/3ac822791e534965b35b0b3d5022881a
        2013-11-05 04:23:17,532 DEBUG [Thread-776] regionserver.HStore(1430): Removing store files after compaction...
        Exception in thread "Thread-777" java.lang.NullPointerException
                at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1230)
                at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1493)
                at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1324)
                at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:360)
        2013-11-05 04:23:17,533 DEBUG [Thread-776] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:file:/grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/data/default/testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/colfamily11/94f3260116304004853721be928701f9, to file:/grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/archive/data/default/testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/colfamily11/94f3260116304004853721be928701f9
                at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:776)
                at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:234)
                at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:149)
                at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:172)
                at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1688)
                at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:3427)
                at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1746)
                at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1738)
        2013-11-05 04:23:17,534 DEBUG [Thread-776] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:file:/grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/data/default/testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/colfamily11/795449ca77094e769ac390749c89f92b, to file:/grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/archive/data/default/testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/colfamily11/795449ca77094e769ac390749c89f92b
                at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1715)
                at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4364)
                at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4339)
                at org.apache.hadoop.hbase.regionserver.TestAtomicOperation$2.run(TestAtomicOperation.java:360)
        
        Show
        Ted Yu added a comment - Encountered NPE in the else branch (with patch v1): 2013-11-05 04:23:17,524 DEBUG [ Thread -776] compactions.ExploringCompactionPolicy(122): Exploring compaction algorithm has selected 3 files of size 3318 starting at candidate #24 after considering 1884 permutations with 1884 in ratio 2013-11-05 04:23:17,524 DEBUG [ Thread -776] regionserver.HStore(1359): fa933591407f5591c2410908ea5065f9 - colfamily11: Initiating minor compaction 2013-11-05 04:23:17,524 INFO [ Thread -776] regionserver.HRegion(1288): Starting compaction on colfamily11 in region testRowMutationMultiThreads,,1383625393150.fa933591407f5591c2410908ea5065f9. 2013-11-05 04:23:17,524 INFO [ Thread -776] regionserver.HStore(1001): Starting compaction of 3 file(s) in colfamily11 of testRowMutationMultiThreads,,1383625393150.fa933591407f5591c2410908ea5065f9. into tmpdir=/grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/data/ default /testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/.tmp, totalSize=3.2 K 2013-11-05 04:23:17,525 DEBUG [ Thread -776] compactions.Compactor(147): Compacting file:/grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/data/ default /testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/colfamily11/94f3260116304004853721be928701f9, keycount=1, bloomtype=ROW, size=1.1 K, encoding=NONE, seqNum=1331 2013-11-05 04:23:17,525 DEBUG [ Thread -776] compactions.Compactor(147): Compacting file:/grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/data/ default /testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/colfamily11/795449ca77094e769ac390749c89f92b, keycount=1, bloomtype=ROW, size=1.1 K, encoding=NONE, seqNum=1342 2013-11-05 04:23:17,525 DEBUG [ Thread -776] compactions.Compactor(147): Compacting file:/grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/data/ default /testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/colfamily11/9e2cc96151754cedbc0d461653cbea63, keycount=1, bloomtype=ROW, size=1.1 K, encoding=NONE, seqNum=1353 2013-11-05 04:23:17,531 DEBUG [ Thread -776] regionserver.HRegionFileSystem(338): Committing store file /grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/data/ default /testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/.tmp/3ac822791e534965b35b0b3d5022881a as /grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/data/ default /testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/colfamily11/3ac822791e534965b35b0b3d5022881a 2013-11-05 04:23:17,532 DEBUG [ Thread -776] regionserver.HStore(1430): Removing store files after compaction... Exception in thread " Thread -777" java.lang.NullPointerException at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1230) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1493) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1324) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:360) 2013-11-05 04:23:17,533 DEBUG [ Thread -776] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:file:/grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/data/ default /testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/colfamily11/94f3260116304004853721be928701f9, to file:/grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/archive/data/ default /testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/colfamily11/94f3260116304004853721be928701f9 at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:776) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:234) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:149) at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:172) at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1688) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:3427) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1746) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1738) 2013-11-05 04:23:17,534 DEBUG [ Thread -776] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:file:/grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/data/ default /testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/colfamily11/795449ca77094e769ac390749c89f92b, to file:/grid/0/dev/ty/trunk/hbase-server/target/test-data/1a023b65-addc-4c5d-acaf-e346220aff31/archive/data/ default /testRowMutationMultiThreads/fa933591407f5591c2410908ea5065f9/colfamily11/795449ca77094e769ac390749c89f92b at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1715) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4364) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4339) at org.apache.hadoop.hbase.regionserver.TestAtomicOperation$2.run(TestAtomicOperation.java:360)
        Hide
        Ted Yu added a comment -

        Haven't figured out how istream got closed and reset.

        Patch v1 would provide some clue. The added parameter is for figuring out which stream of the wrapper gets used - see HFileBlock, line 1322:

              FSDataInputStream is = streamWrapper.getStream(doVerificationThruHBaseChecksum);
        
        Show
        Ted Yu added a comment - Haven't figured out how istream got closed and reset. Patch v1 would provide some clue. The added parameter is for figuring out which stream of the wrapper gets used - see HFileBlock, line 1322: FSDataInputStream is = streamWrapper.getStream(doVerificationThruHBaseChecksum);
        Hide
        Sergey Shelukhin added a comment -

        Seems to coincide with compaction file replacement... probably the close is indeed called and the stream is closed in the wrapper.
        I wonder why it didn't happen before

        Show
        Sergey Shelukhin added a comment - Seems to coincide with compaction file replacement... probably the close is indeed called and the stream is closed in the wrapper. I wonder why it didn't happen before
        Hide
        Ted Yu added a comment -

        From https://builds.apache.org/job/PreCommit-HBASE-Build/7730//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/ :

        2013-11-05 00:29:54,062 DEBUG [Thread-106] regionserver.HStore(1430): Removing store files after compaction...
        Exception in thread "Thread-108" java.lang.NullPointerException	at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
        	at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1487)
        	at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
        	at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:360)
        	at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:776)
        	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:234)
        	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:149)
        	at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:172)
        	at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1688)
        	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:3543)
        	at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1749)
        	at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1741)
        	at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1718)
        	at org.apache.hadoop.hbase.regionserver.TestAtomicOperation$3.run(TestAtomicOperation.java:449)
        
        Show
        Ted Yu added a comment - From https://builds.apache.org/job/PreCommit-HBASE-Build/7730//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/ : 2013-11-05 00:29:54,062 DEBUG [ Thread -106] regionserver.HStore(1430): Removing store files after compaction... Exception in thread " Thread -108" java.lang.NullPointerException at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1487) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:360) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:776) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:234) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:149) at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:172) at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1688) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:3543) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1749) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1741) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1718) at org.apache.hadoop.hbase.regionserver.TestAtomicOperation$3.run(TestAtomicOperation.java:449)
        Show
        ramkrishna.s.vasudevan added a comment - I got this in https://builds.apache.org/job/PreCommit-HBASE-Build/7722//testReport/org.apache.hadoop.hbase.regionserver/TestHRegion/testWritesWhileGetting/ . But not able to reproduce next time.
        Hide
        Sergey Shelukhin added a comment -

        unassign for now... I will still look if theres more info

        Show
        Sergey Shelukhin added a comment - unassign for now... I will still look if theres more info
        Hide
        Sergey Shelukhin added a comment -

        Hmm... I looked at the code a little bit and so far I am in "this shouldn't happen" stage.
        Are you sure "close" is not called in parallel? Could be from region close due to schema change. Can you provide surrounding logs.

        "prepare" on the wrapper is called in ctor so I am assuming it couldn't be prepare-read race.
        Otherwise (outside prepare) the boolean is never set to a value until the corresponding stream is not null, all things are volatile, and streams are never null-ed until close.

        Show
        Sergey Shelukhin added a comment - Hmm... I looked at the code a little bit and so far I am in "this shouldn't happen" stage. Are you sure "close" is not called in parallel? Could be from region close due to schema change. Can you provide surrounding logs. "prepare" on the wrapper is called in ctor so I am assuming it couldn't be prepare-read race. Otherwise (outside prepare) the boolean is never set to a value until the corresponding stream is not null, all things are volatile, and streams are never null-ed until close.
        Hide
        Jimmy Xiang added a comment -

        Trunk.

        Show
        Jimmy Xiang added a comment - Trunk.
        Hide
        ramkrishna.s.vasudevan added a comment -

        Which version does this happen Jimmy?

        Show
        ramkrishna.s.vasudevan added a comment - Which version does this happen Jimmy?
        Hide
        Jimmy Xiang added a comment -

        These happened at least three times in one run. It happened before too.

        Show
        Jimmy Xiang added a comment - These happened at least three times in one run. It happened before too.

          People

          • Assignee:
            Ted Yu
            Reporter:
            Jimmy Xiang
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:

              Development