HBase
  1. HBase
  2. HBASE-9239

TestStoreFileBlockCacheSummary#testBlockCacheSummary occasionally fails

    Details

    • Type: Test Test
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.98.0, 0.96.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      From https://builds.apache.org/job/HBase-0.95/455/testReport/org.apache.hadoop.hbase.regionserver/TestStoreFileBlockCacheSummary/testBlockCacheSummary/ :

      java.lang.AssertionError: blockCache summary has entries expected:<2> but was:<3>
      	at org.junit.Assert.fail(Assert.java:88)
      	at org.junit.Assert.failNotEquals(Assert.java:743)
      	at org.junit.Assert.assertEquals(Assert.java:118)
      	at org.junit.Assert.assertEquals(Assert.java:555)
      	at org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary.testBlockCacheSummary(TestStoreFileBlockCacheSummary.java:112)
      

      This was due to an additional entry for namespace table:

      2013-08-16 00:24:25,337 INFO  [pool-1-thread-1] regionserver.TestStoreFileBlockCacheSummary(110): blockCacheSummary: [BlockCacheSummaryEntry [table=namespace, columnFamily=info, blocks=1, heapSize=512], BlockCacheSummaryEntry [table=testTable, columnFamily=testFamily, blocks=1, heapSize=664], BlockCacheSummaryEntry [table=testTable2, columnFamily=testFamily, blocks=1, heapSize=664]]
      

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        19m 40s 1 Ted Yu 16/Aug/13 02:19
        Patch Available Patch Available Resolved Resolved
        7d 23h 59m 1 Ted Yu 24/Aug/13 02:18
        Ted Yu made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in hbase-0.95 #460 (See https://builds.apache.org/job/hbase-0.95/460/)
        HBASE-9239 TestStoreFileBlockCacheSummary#testBlockCacheSummary occasionally fails (tedyu: rev 1514878)

        • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java
        Show
        Hudson added a comment - SUCCESS: Integrated in hbase-0.95 #460 (See https://builds.apache.org/job/hbase-0.95/460/ ) HBASE-9239 TestStoreFileBlockCacheSummary#testBlockCacheSummary occasionally fails (tedyu: rev 1514878) /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in hbase-0.95-on-hadoop2 #247 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/247/)
        HBASE-9239 TestStoreFileBlockCacheSummary#testBlockCacheSummary occasionally fails (tedyu: rev 1514878)

        • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java
        Show
        Hudson added a comment - FAILURE: Integrated in hbase-0.95-on-hadoop2 #247 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/247/ ) HBASE-9239 TestStoreFileBlockCacheSummary#testBlockCacheSummary occasionally fails (tedyu: rev 1514878) /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #681 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/681/)
        HBASE-9239 TestStoreFileBlockCacheSummary#testBlockCacheSummary occasionally fails (Ted Yu) (tedyu: rev 1514770)

        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #681 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/681/ ) HBASE-9239 TestStoreFileBlockCacheSummary#testBlockCacheSummary occasionally fails (Ted Yu) (tedyu: rev 1514770) /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK #4403 (See https://builds.apache.org/job/HBase-TRUNK/4403/)
        HBASE-9239 TestStoreFileBlockCacheSummary#testBlockCacheSummary occasionally fails (Ted Yu) (tedyu: rev 1514770)

        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #4403 (See https://builds.apache.org/job/HBase-TRUNK/4403/ ) HBASE-9239 TestStoreFileBlockCacheSummary#testBlockCacheSummary occasionally fails (Ted Yu) (tedyu: rev 1514770) /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java
        Ted Yu made changes -
        Hadoop Flags Reviewed [ 10343 ]
        Ted Yu made changes -
        Fix Version/s 0.96.0 [ 12324822 ]
        Hide
        Ted Yu added a comment -

        TestBlockCacheColumnFamilySummary passed in trunk builds.

        Integrated to 0.95 branch as well since the cited test failure happened in 0.95 build.

        Show
        Ted Yu added a comment - TestBlockCacheColumnFamilySummary passed in trunk builds. Integrated to 0.95 branch as well since the cited test failure happened in 0.95 build.
        Hide
        Ted Yu added a comment -

        Integrated to trunk.

        Thanks for the reviews.

        Show
        Ted Yu added a comment - Integrated to trunk. Thanks for the reviews.
        Hide
        Anoop Sam John added a comment -

        Just seeing the patch I also got the same doubt as Ram mentioned.

        The contract of this interface is to return the List in sorted order by Table name, then ColumnFamily.

        Said above , +1 then

        Show
        Anoop Sam John added a comment - Just seeing the patch I also got the same doubt as Ram mentioned. The contract of this interface is to return the List in sorted order by Table name, then ColumnFamily. Said above , +1 then
        Hide
        ramkrishna.s.vasudevan added a comment -

        Okie.. So +1 then. Thanks Ted.

        Show
        ramkrishna.s.vasudevan added a comment - Okie.. So +1 then. Thanks Ted.
        Hide
        Ted Yu added a comment -

        I think implementation for namespace is complete.
        So no new entry would be added, I assume.

        Show
        Ted Yu added a comment - I think implementation for namespace is complete. So no new entry would be added, I assume.
        Hide
        ramkrishna.s.vasudevan added a comment -

        Yes Ted. I read that javadoc. But anyother namespace entry could be added? If so again this index may change. So i suggested that way.

        Show
        ramkrishna.s.vasudevan added a comment - Yes Ted. I read that javadoc. But anyother namespace entry could be added? If so again this index may change. So i suggested that way.
        Hide
        Ted Yu added a comment -

        I thought about extracting entries for the test tables.
        However I think that is not needed because of the following (javadoc of getBlockCacheColumnFamilySummaries()):

        The contract of this interface is to return the List in sorted order by Table name, then ColumnFamily.

        Meaning the test tables would always follow namespace table in the list.

        Show
        Ted Yu added a comment - I thought about extracting entries for the test tables. However I think that is not needed because of the following (javadoc of getBlockCacheColumnFamilySummaries()): The contract of this interface is to return the List in sorted order by Table name, then ColumnFamily. Meaning the test tables would always follow namespace table in the list.
        Hide
        ramkrishna.s.vasudevan added a comment -

        @Ted
        Patch is good.
        I have a suggestion, from the logs could see that the NameSpaceJanitor issues a scan on the Meta and that includes Namespace into the blockcache summary.
        From code am not getting where the scan is happening through NameSpaceJanitor.
        May be instead working with the index, we can just check if the testTable and testTable2 are available in the list that we are checking.
        If the catalog janitor adds any other namespace entries this may fail? So we will check if the list contains those two tables then it may be easier.

        Show
        ramkrishna.s.vasudevan added a comment - @Ted Patch is good. I have a suggestion, from the logs could see that the NameSpaceJanitor issues a scan on the Meta and that includes Namespace into the blockcache summary. From code am not getting where the scan is happening through NameSpaceJanitor. May be instead working with the index, we can just check if the testTable and testTable2 are available in the list that we are checking. If the catalog janitor adds any other namespace entries this may fail? So we will check if the list contains those two tables then it may be easier.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12598339/9239-v1.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6772//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6772//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6772//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6772//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6772//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6772//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6772//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6772//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6772//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6772//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12598339/9239-v1.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6772//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6772//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6772//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6772//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6772//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6772//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6772//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6772//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6772//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6772//console This message is automatically generated.
        Ted Yu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Fix Version/s 0.98.0 [ 12323143 ]
        Ted Yu made changes -
        Field Original Value New Value
        Attachment 9239-v1.txt [ 12598339 ]
        Hide
        Ted Yu added a comment -

        Patch v1 allows more than 2 BlockCacheColumnFamilySummary entries to be retrieved.

        Subsequent assertions work on the last two entries.

        Test passed locally.

        Show
        Ted Yu added a comment - Patch v1 allows more than 2 BlockCacheColumnFamilySummary entries to be retrieved. Subsequent assertions work on the last two entries. Test passed locally.
        Ted Yu created issue -

          People

          • Assignee:
            Ted Yu
            Reporter:
            Ted Yu
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development