HBase
  1. HBase
  2. HBASE-4542

add filter info to slow query logging

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.89.20100924
    • Fix Version/s: 0.94.0, 0.95.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Slow query log doesn't report filters in effect.

      For example:

      (operationTooSlow): \
      {"processingtimems":3468,"client":"10.138.43.206:40035","timeRange": [0,9223372036854775807],\
      "starttimems":1317772005821,"responsesize":42411, \
      "class":"HRegionServer","table":"myTable","families":{"CF1":"ALL"]},\
      "row":"6c3b8efa132f0219b7621ed1e5c8c70b","queuetimems":0,\
      "method":"get","totalColumns":1,"maxVersions":1,"storeLimit":-1}
      

      the above would suggest that all columns of myTable:CF1 are being requested for the given row. But in reality there could be filters in effect (such as ColumnPrefixFilter, ColumnRangeFilter, TimestampsFilter() etc.). We should enhance the slow query log to capture & report this information.

        Activity

        Lars Hofhansl made changes -
        Fix Version/s 0.94.0 [ 12316419 ]
        stack made changes -
        Fix Version/s 0.95.0 [ 12324094 ]
        Fix Version/s 0.94.0 [ 12316419 ]
        Fix Version/s 0.96.0 [ 12320040 ]
        Lars Hofhansl made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94 #44 (See https://builds.apache.org/job/HBase-0.94/44/)
        HBASE-4542 add filter info to slow query logging (Madhuwanti Vaidya) (Revision 1303503)

        Result = SUCCESS
        larsh :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Get.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Scan.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/ColumnPrefixFilter.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/ColumnRangeFilter.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/CompareFilter.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/DependentColumnFilter.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/FilterBase.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/InclusiveStopFilter.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/MultipleColumnPrefixFilter.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/PageFilter.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/SkipFilter.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/TimestampsFilter.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/WhileMatchFilter.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestOperation.java
        Show
        Hudson added a comment - Integrated in HBase-0.94 #44 (See https://builds.apache.org/job/HBase-0.94/44/ ) HBASE-4542 add filter info to slow query logging (Madhuwanti Vaidya) (Revision 1303503) Result = SUCCESS larsh : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Get.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Scan.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/ColumnPrefixFilter.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/ColumnRangeFilter.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/CompareFilter.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/DependentColumnFilter.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/FilterBase.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/InclusiveStopFilter.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/MultipleColumnPrefixFilter.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/PageFilter.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/SkipFilter.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/TimestampsFilter.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/WhileMatchFilter.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestOperation.java
        Hide
        Lars Hofhansl added a comment -

        Never mind, extracted the patch with svn diff -r r1298923:r1299019

        Show
        Lars Hofhansl added a comment - Never mind, extracted the patch with svn diff -r r1298923:r1299019
        Lars Hofhansl made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Resolution Fixed [ 1 ]
        Hide
        Lars Hofhansl added a comment -

        Comitted to 0.94 aswell.

        Show
        Lars Hofhansl added a comment - Comitted to 0.94 aswell.
        Hide
        Lars Hofhansl added a comment -

        Could you confirm that "Add-filter-info-to-slow-query-logging-2012-03-06_14_28_13.patch"
        is the patch to apply?
        Otherwise I'll extract the patch from svn. Thanks.

        Show
        Lars Hofhansl added a comment - Could you confirm that "Add-filter-info-to-slow-query-logging-2012-03-06_14_28_13.patch" is the patch to apply? Otherwise I'll extract the patch from svn. Thanks.
        Lars Hofhansl made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Hide
        Lars Hofhansl added a comment -

        Reopening so I won't forget about the 0.94 part.

        Show
        Lars Hofhansl added a comment - Reopening so I won't forget about the 0.94 part.
        Hide
        Lars Hofhansl added a comment -

        Please... Do not mark an issue as fixed if it is targeted to multiple versions but not all versions are committed. Pretty please?

        Show
        Lars Hofhansl added a comment - Please... Do not mark an issue as fixed if it is targeted to multiple versions but not all versions are committed. Pretty please?
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2676 (See https://builds.apache.org/job/HBase-TRUNK/2676/)
        [jira] HBASE-4542 Add filter info to slow query logging

        Author: Zhiqiu Kong

        Summary:
        The original 89-fb diff is: https://reviews.facebook.net/D1263

        Slow opertaion log does not provide enough information when a filter is
        present. The followings are done to add the filter info:

        1) Added toString() method for filters inheriting FilterBase, this
        affect 22 filters and their subclasses. The info added includes the
        filter's name and its members. For example, for TimestampsFilter, we'll
        output its class name as well as the defined timestamps.

        2) Added a field 'filter' in Get::toMap() and Scan::toMap() to enable
        the logging of filter info.

        Task ID: #750975

        Blame Rev:

        Test Plan:
        1. Run and passed unit-tests to make sure it does not break things

        2. Run kannan's script to trigger the slow operation logging, checked
        for each filter to make sure the filter info was logged. To be more
        detailed, the output log are as following (only 'filter' filed is put
        here for ease of reading):

        "filter":"TimestampsFilter (3/3): [2, 3, 5]"
        "filter":"TimestampsFilter (5/6): [2, 3, 5, 7, 11]"
        "filter":"ColumnPrefixFilter col2"
        "filter":"ColumnRangeFilter [col2a, col2b]"
        "filter":"ColumnCountGetFilter 8"
        "filter":"ColumnPaginationFilter (4, 4)"
        "filter":"InclusiveStopFilter row"
        "filter":"PrefixFilter row"
        "filter":"PageFilter 1"
        "filter":"SkipFilter TimestampsFilter (1/1): [1000]"
        "filter":"WhileMatchFilter TimestampsFilter (3/3): [2, 3, 5]"
        "filter":"KeyOnlyFilter"
        "filter":"FirstKeyOnlyFilter"
        "filter":"MultipleColumnPrefixFilter (3/3): [a, b, c]"
        "filter":"DependentColumnFilter (family, qualifier, true, LESS, value)"
        "filter":"FamilyFilter (LESS, value)"
        "filter":"QualifierFilter (LESS, value)"
        "filter":"RowFilter (LESS, value)"
        "filter":"ValueFilter (LESS, value)"
        "filter":"KeyOnlyFilter"
        "filter":"FirstKeyOnlyFilter"
        "filter":"SingleColumnValueFilter (family, qualifier, EQUAL, value)"
        "filter":"SingleColumnValueExcludeFilter (family, qualifier, EQUAL,
        value)"
        "filter":"FilterList AND (2/2): [KeyOnlyFilter, FirstKeyOnlyFilter]"
        Please check ~zhiqiu/Codes/scripts/testFilter.rb for the testing script.

        3. Added unit test cases to TestOperation to verify the filters'
        toString() method works well.

        Reviewed By: mbautin

        Reviewers: Kannan, madhuvaidya, mbautin, JIRA

        CC: Kannan, madhuvaidya, mbautin, zhiqiu, stack

        Differential Revision: https://reviews.facebook.net/D1539 (Revision 1299019)

        Result = FAILURE
        mbautin :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Get.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/ColumnPrefixFilter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/ColumnRangeFilter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/CompareFilter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/DependentColumnFilter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/FilterBase.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/InclusiveStopFilter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/MultipleColumnPrefixFilter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/PageFilter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/SkipFilter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/TimestampsFilter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/WhileMatchFilter.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestOperation.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2676 (See https://builds.apache.org/job/HBase-TRUNK/2676/ ) [jira] HBASE-4542 Add filter info to slow query logging Author: Zhiqiu Kong Summary: The original 89-fb diff is: https://reviews.facebook.net/D1263 Slow opertaion log does not provide enough information when a filter is present. The followings are done to add the filter info: 1) Added toString() method for filters inheriting FilterBase, this affect 22 filters and their subclasses. The info added includes the filter's name and its members. For example, for TimestampsFilter, we'll output its class name as well as the defined timestamps. 2) Added a field 'filter' in Get::toMap() and Scan::toMap() to enable the logging of filter info. Task ID: #750975 Blame Rev: Test Plan: 1. Run and passed unit-tests to make sure it does not break things 2. Run kannan's script to trigger the slow operation logging, checked for each filter to make sure the filter info was logged. To be more detailed, the output log are as following (only 'filter' filed is put here for ease of reading): "filter":"TimestampsFilter (3/3): [2, 3, 5] " "filter":"TimestampsFilter (5/6): [2, 3, 5, 7, 11] " "filter":"ColumnPrefixFilter col2" "filter":"ColumnRangeFilter [col2a, col2b] " "filter":"ColumnCountGetFilter 8" "filter":"ColumnPaginationFilter (4, 4)" "filter":"InclusiveStopFilter row" "filter":"PrefixFilter row" "filter":"PageFilter 1" "filter":"SkipFilter TimestampsFilter (1/1): [1000] " "filter":"WhileMatchFilter TimestampsFilter (3/3): [2, 3, 5] " "filter":"KeyOnlyFilter" "filter":"FirstKeyOnlyFilter" "filter":"MultipleColumnPrefixFilter (3/3): [a, b, c] " "filter":"DependentColumnFilter (family, qualifier, true, LESS, value)" "filter":"FamilyFilter (LESS, value)" "filter":"QualifierFilter (LESS, value)" "filter":"RowFilter (LESS, value)" "filter":"ValueFilter (LESS, value)" "filter":"KeyOnlyFilter" "filter":"FirstKeyOnlyFilter" "filter":"SingleColumnValueFilter (family, qualifier, EQUAL, value)" "filter":"SingleColumnValueExcludeFilter (family, qualifier, EQUAL, value)" "filter":"FilterList AND (2/2): [KeyOnlyFilter, FirstKeyOnlyFilter] " Please check ~zhiqiu/Codes/scripts/testFilter.rb for the testing script. 3. Added unit test cases to TestOperation to verify the filters' toString() method works well. Reviewed By: mbautin Reviewers: Kannan, madhuvaidya, mbautin, JIRA CC: Kannan, madhuvaidya, mbautin, zhiqiu, stack Differential Revision: https://reviews.facebook.net/D1539 (Revision 1299019) Result = FAILURE mbautin : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Get.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/ColumnPrefixFilter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/ColumnRangeFilter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/CompareFilter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/DependentColumnFilter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/FilterBase.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/InclusiveStopFilter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/MultipleColumnPrefixFilter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/PageFilter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/SkipFilter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/TimestampsFilter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/WhileMatchFilter.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestOperation.java
        Hide
        stack added a comment -

        And Lars, I think it should be safe since its on toStringery that is going on in this patch... no changes in Interfaces.

        Show
        stack added a comment - And Lars, I think it should be safe since its on toStringery that is going on in this patch... no changes in Interfaces.
        Hide
        stack added a comment -

        +1 for 0.94. This is good stuff.

        Show
        stack added a comment - +1 for 0.94. This is good stuff.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-security #133 (See https://builds.apache.org/job/HBase-TRUNK-security/133/)
        [jira] HBASE-4542 Add filter info to slow query logging

        Author: Zhiqiu Kong

        Summary:
        The original 89-fb diff is: https://reviews.facebook.net/D1263

        Slow opertaion log does not provide enough information when a filter is
        present. The followings are done to add the filter info:

        1) Added toString() method for filters inheriting FilterBase, this
        affect 22 filters and their subclasses. The info added includes the
        filter's name and its members. For example, for TimestampsFilter, we'll
        output its class name as well as the defined timestamps.

        2) Added a field 'filter' in Get::toMap() and Scan::toMap() to enable
        the logging of filter info.

        Task ID: #750975

        Blame Rev:

        Test Plan:
        1. Run and passed unit-tests to make sure it does not break things

        2. Run kannan's script to trigger the slow operation logging, checked
        for each filter to make sure the filter info was logged. To be more
        detailed, the output log are as following (only 'filter' filed is put
        here for ease of reading):

        "filter":"TimestampsFilter (3/3): [2, 3, 5]"
        "filter":"TimestampsFilter (5/6): [2, 3, 5, 7, 11]"
        "filter":"ColumnPrefixFilter col2"
        "filter":"ColumnRangeFilter [col2a, col2b]"
        "filter":"ColumnCountGetFilter 8"
        "filter":"ColumnPaginationFilter (4, 4)"
        "filter":"InclusiveStopFilter row"
        "filter":"PrefixFilter row"
        "filter":"PageFilter 1"
        "filter":"SkipFilter TimestampsFilter (1/1): [1000]"
        "filter":"WhileMatchFilter TimestampsFilter (3/3): [2, 3, 5]"
        "filter":"KeyOnlyFilter"
        "filter":"FirstKeyOnlyFilter"
        "filter":"MultipleColumnPrefixFilter (3/3): [a, b, c]"
        "filter":"DependentColumnFilter (family, qualifier, true, LESS, value)"
        "filter":"FamilyFilter (LESS, value)"
        "filter":"QualifierFilter (LESS, value)"
        "filter":"RowFilter (LESS, value)"
        "filter":"ValueFilter (LESS, value)"
        "filter":"KeyOnlyFilter"
        "filter":"FirstKeyOnlyFilter"
        "filter":"SingleColumnValueFilter (family, qualifier, EQUAL, value)"
        "filter":"SingleColumnValueExcludeFilter (family, qualifier, EQUAL,
        value)"
        "filter":"FilterList AND (2/2): [KeyOnlyFilter, FirstKeyOnlyFilter]"
        Please check ~zhiqiu/Codes/scripts/testFilter.rb for the testing script.

        3. Added unit test cases to TestOperation to verify the filters'
        toString() method works well.

        Reviewed By: mbautin

        Reviewers: Kannan, madhuvaidya, mbautin, JIRA

        CC: Kannan, madhuvaidya, mbautin, zhiqiu, stack

        Differential Revision: https://reviews.facebook.net/D1539 (Revision 1299019)

        Result = FAILURE
        mbautin :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Get.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/ColumnPrefixFilter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/ColumnRangeFilter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/CompareFilter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/DependentColumnFilter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/FilterBase.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/InclusiveStopFilter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/MultipleColumnPrefixFilter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/PageFilter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/SkipFilter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/TimestampsFilter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/WhileMatchFilter.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestOperation.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-security #133 (See https://builds.apache.org/job/HBase-TRUNK-security/133/ ) [jira] HBASE-4542 Add filter info to slow query logging Author: Zhiqiu Kong Summary: The original 89-fb diff is: https://reviews.facebook.net/D1263 Slow opertaion log does not provide enough information when a filter is present. The followings are done to add the filter info: 1) Added toString() method for filters inheriting FilterBase, this affect 22 filters and their subclasses. The info added includes the filter's name and its members. For example, for TimestampsFilter, we'll output its class name as well as the defined timestamps. 2) Added a field 'filter' in Get::toMap() and Scan::toMap() to enable the logging of filter info. Task ID: #750975 Blame Rev: Test Plan: 1. Run and passed unit-tests to make sure it does not break things 2. Run kannan's script to trigger the slow operation logging, checked for each filter to make sure the filter info was logged. To be more detailed, the output log are as following (only 'filter' filed is put here for ease of reading): "filter":"TimestampsFilter (3/3): [2, 3, 5] " "filter":"TimestampsFilter (5/6): [2, 3, 5, 7, 11] " "filter":"ColumnPrefixFilter col2" "filter":"ColumnRangeFilter [col2a, col2b] " "filter":"ColumnCountGetFilter 8" "filter":"ColumnPaginationFilter (4, 4)" "filter":"InclusiveStopFilter row" "filter":"PrefixFilter row" "filter":"PageFilter 1" "filter":"SkipFilter TimestampsFilter (1/1): [1000] " "filter":"WhileMatchFilter TimestampsFilter (3/3): [2, 3, 5] " "filter":"KeyOnlyFilter" "filter":"FirstKeyOnlyFilter" "filter":"MultipleColumnPrefixFilter (3/3): [a, b, c] " "filter":"DependentColumnFilter (family, qualifier, true, LESS, value)" "filter":"FamilyFilter (LESS, value)" "filter":"QualifierFilter (LESS, value)" "filter":"RowFilter (LESS, value)" "filter":"ValueFilter (LESS, value)" "filter":"KeyOnlyFilter" "filter":"FirstKeyOnlyFilter" "filter":"SingleColumnValueFilter (family, qualifier, EQUAL, value)" "filter":"SingleColumnValueExcludeFilter (family, qualifier, EQUAL, value)" "filter":"FilterList AND (2/2): [KeyOnlyFilter, FirstKeyOnlyFilter] " Please check ~zhiqiu/Codes/scripts/testFilter.rb for the testing script. 3. Added unit test cases to TestOperation to verify the filters' toString() method works well. Reviewed By: mbautin Reviewers: Kannan, madhuvaidya, mbautin, JIRA CC: Kannan, madhuvaidya, mbautin, zhiqiu, stack Differential Revision: https://reviews.facebook.net/D1539 (Revision 1299019) Result = FAILURE mbautin : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Get.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/ColumnPrefixFilter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/ColumnRangeFilter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/CompareFilter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/DependentColumnFilter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/FilterBase.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/InclusiveStopFilter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/MultipleColumnPrefixFilter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/PageFilter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/SkipFilter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/TimestampsFilter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/WhileMatchFilter.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestOperation.java
        Hide
        Lars Hofhansl added a comment -

        What's the thought here w.r.t. 0.94? I could see this either way.

        Show
        Lars Hofhansl added a comment - What's the thought here w.r.t. 0.94? I could see this either way.
        Hide
        Lars Hofhansl added a comment -

        Are you going to commit this to 0.94 as well? (trunk is 0.96 now)

        Show
        Lars Hofhansl added a comment - Are you going to commit this to 0.94 as well? (trunk is 0.96 now)
        Lars Hofhansl made changes -
        Fix Version/s 0.96.0 [ 12320040 ]
        Mikhail Bautin made changes -
        Fix Version/s 0.94.0 [ 12316419 ]
        Mikhail Bautin made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Phabricator added a comment -

        mbautin has committed the revision "[jira] HBASE-4542 Add filter info to slow query logging".

        REVISION DETAIL
        https://reviews.facebook.net/D1539

        COMMIT
        https://reviews.facebook.net/rHBASE1299019

        Show
        Phabricator added a comment - mbautin has committed the revision " [jira] HBASE-4542 Add filter info to slow query logging". REVISION DETAIL https://reviews.facebook.net/D1539 COMMIT https://reviews.facebook.net/rHBASE1299019
        Hide
        stack added a comment -

        +1 on commit (Thanks for taking care of the this Mikhail)

        Show
        stack added a comment - +1 on commit (Thanks for taking care of the this Mikhail)
        Hide
        Phabricator added a comment -

        mbautin has accepted the revision "[jira] HBASE-4542 Add filter info to slow query logging".

        REVISION DETAIL
        https://reviews.facebook.net/D1539

        BRANCH
        filterLog

        Show
        Phabricator added a comment - mbautin has accepted the revision " [jira] HBASE-4542 Add filter info to slow query logging". REVISION DETAIL https://reviews.facebook.net/D1539 BRANCH filterLog
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12517322/Add-filter-info-to-slow-query-logging-2012-03-06_14_28_13.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 4 new or modified tests.

        -1 javadoc. The javadoc tool appears to have generated -129 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 154 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
        org.apache.hadoop.hbase.mapred.TestTableMapReduce
        org.apache.hadoop.hbase.mapreduce.TestImportTsv

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1121//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1121//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1121//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517322/Add-filter-info-to-slow-query-logging-2012-03-06_14_28_13.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -129 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 154 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestImportTsv Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1121//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1121//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1121//console This message is automatically generated.
        Hide
        Mikhail Bautin added a comment -

        All unit tests passed locally. Waiting for Hadoop QA before committing.

        Show
        Mikhail Bautin added a comment - All unit tests passed locally. Waiting for Hadoop QA before committing.
        Mikhail Bautin made changes -
        Hide
        Mikhail Bautin added a comment -

        Rebasing patch on trunk

        Show
        Mikhail Bautin added a comment - Rebasing patch on trunk
        Hide
        Phabricator added a comment -

        zhiqiu has commented on the revision "[jira] HBASE-4542 Add filter info to slow query logging".

        @mbautin Thanks a lot!

        REVISION DETAIL
        https://reviews.facebook.net/D1539

        Show
        Phabricator added a comment - zhiqiu has commented on the revision " [jira] HBASE-4542 Add filter info to slow query logging". @mbautin Thanks a lot! REVISION DETAIL https://reviews.facebook.net/D1539
        Hide
        Phabricator added a comment -

        mbautin has commented on the revision "[jira] HBASE-4542 Add filter info to slow query logging".

        It looks like this never got committed into trunk. I will rebase the patch on trunk changes.

        REVISION DETAIL
        https://reviews.facebook.net/D1539

        Show
        Phabricator added a comment - mbautin has commented on the revision " [jira] HBASE-4542 Add filter info to slow query logging". It looks like this never got committed into trunk. I will rebase the patch on trunk changes. REVISION DETAIL https://reviews.facebook.net/D1539
        Hide
        Phabricator added a comment -

        mbautin has committed the revision "[jira] HBASE-4542 [89-fb] Add filter info to slow query logging".

        REVISION DETAIL
        https://reviews.facebook.net/D1263

        COMMIT
        https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1239780

        Show
        Phabricator added a comment - mbautin has committed the revision " [jira] HBASE-4542 [89-fb] Add filter info to slow query logging". REVISION DETAIL https://reviews.facebook.net/D1263 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1239780
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12512929/0001-jira-HBASE-4542-Add-filter-info-to-slow-query-loggin.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 5 new or modified tests.

        -1 javadoc. The javadoc tool appears to have generated -140 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 157 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
        org.apache.hadoop.hbase.mapred.TestTableMapReduce
        org.apache.hadoop.hbase.io.hfile.TestHFileBlock
        org.apache.hadoop.hbase.mapreduce.TestImportTsv

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/894//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/894//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/894//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512929/0001-jira-HBASE-4542-Add-filter-info-to-slow-query-loggin.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -140 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 157 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.io.hfile.TestHFileBlock org.apache.hadoop.hbase.mapreduce.TestImportTsv Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/894//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/894//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/894//console This message is automatically generated.
        Hide
        Phabricator added a comment -

        zhiqiu has commented on the revision "[jira] HBASE-4542 Add filter info to slow query logging".

        @stack I just attached a patch to JIRA (https://issues.apache.org/jira/browse/HBASE-4542). Thanks!

        REVISION DETAIL
        https://reviews.facebook.net/D1539

        Show
        Phabricator added a comment - zhiqiu has commented on the revision " [jira] HBASE-4542 Add filter info to slow query logging". @stack I just attached a patch to JIRA ( https://issues.apache.org/jira/browse/HBASE-4542 ). Thanks! REVISION DETAIL https://reviews.facebook.net/D1539
        Zhiqiu Kong made changes -
        Hide
        Phabricator added a comment -

        stack has commented on the revision "[jira] HBASE-4542 Add filter info to slow query logging".

        Above seems fine to me. If its ready to go, stick it up in Apache JIRA so we can commit. Thanks.

        REVISION DETAIL
        https://reviews.facebook.net/D1539

        Show
        Phabricator added a comment - stack has commented on the revision " [jira] HBASE-4542 Add filter info to slow query logging". Above seems fine to me. If its ready to go, stick it up in Apache JIRA so we can commit. Thanks. REVISION DETAIL https://reviews.facebook.net/D1539
        Hide
        Phabricator added a comment -

        zhiqiu has commented on the revision "[jira] HBASE-4542 Add filter info to slow query logging".

        INLINE COMMENTS
        src/main/java/org/apache/hadoop/hbase/filter/FilterBase.java:146 Right. But maybe it will be less flexible? Say if we want to change super.getString() later to add sth else other than the class name (for subclasses which do not override it), then we need to change all subclasses which override this method back to call this.getClass().getSimpleName() directly again.

        And, calling super.toString() => this.getClass().getSimpleName() will introduce an extra function call thus extra overhead (though very little).
        src/main/java/org/apache/hadoop/hbase/filter/FilterList.java:317 My thought was to keep the log readable for long FilterList. Thus any FilterList w/ more than MAX_LOG_FILTERS will be truncated. We should have a threshold like this, right? But to hardcode it like this might not be a good idea.
        src/main/java/org/apache/hadoop/hbase/filter/MultipleColumnPrefixFilter.java:143 Just like above, say we have 10000 prefixes defined. It might be good to truncate it to keep the log readable.

        REVISION DETAIL
        https://reviews.facebook.net/D1539

        Show
        Phabricator added a comment - zhiqiu has commented on the revision " [jira] HBASE-4542 Add filter info to slow query logging". INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/filter/FilterBase.java:146 Right. But maybe it will be less flexible? Say if we want to change super.getString() later to add sth else other than the class name (for subclasses which do not override it), then we need to change all subclasses which override this method back to call this.getClass().getSimpleName() directly again. And, calling super.toString() => this.getClass().getSimpleName() will introduce an extra function call thus extra overhead (though very little). src/main/java/org/apache/hadoop/hbase/filter/FilterList.java:317 My thought was to keep the log readable for long FilterList. Thus any FilterList w/ more than MAX_LOG_FILTERS will be truncated. We should have a threshold like this, right? But to hardcode it like this might not be a good idea. src/main/java/org/apache/hadoop/hbase/filter/MultipleColumnPrefixFilter.java:143 Just like above, say we have 10000 prefixes defined. It might be good to truncate it to keep the log readable. REVISION DETAIL https://reviews.facebook.net/D1539
        Hide
        Phabricator added a comment -

        stack has commented on the revision "[jira] HBASE-4542 Add filter info to slow query logging".

        +1

        Some small comments.

        INLINE COMMENTS
        src/main/java/org/apache/hadoop/hbase/filter/FilterBase.java:146 Should the subclasses call this and then decorate what this returns with the filter subclass specifics? Otherwise you have this repeating of this.getClass().getSimpleName() in each subclass toString. No biggie, just a thought.
        src/main/java/org/apache/hadoop/hbase/filter/FilterList.java:317 Why we do this? The toString is not being allowed actual filter list size?
        src/main/java/org/apache/hadoop/hbase/filter/MultipleColumnPrefixFilter.java:143 Yeah, I don't get this constraining upper bound.

        REVISION DETAIL
        https://reviews.facebook.net/D1539

        Show
        Phabricator added a comment - stack has commented on the revision " [jira] HBASE-4542 Add filter info to slow query logging". +1 Some small comments. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/filter/FilterBase.java:146 Should the subclasses call this and then decorate what this returns with the filter subclass specifics? Otherwise you have this repeating of this.getClass().getSimpleName() in each subclass toString. No biggie, just a thought. src/main/java/org/apache/hadoop/hbase/filter/FilterList.java:317 Why we do this? The toString is not being allowed actual filter list size? src/main/java/org/apache/hadoop/hbase/filter/MultipleColumnPrefixFilter.java:143 Yeah, I don't get this constraining upper bound. REVISION DETAIL https://reviews.facebook.net/D1539
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12512660/D1539.1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/882//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512660/D1539.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/882//console This message is automatically generated.
        Hide
        Phabricator added a comment -

        zhiqiu has commented on the revision "[jira] HBASE-4542 Add filter info to slow query logging".

        @mbautin Thanks! Have updated JIRA status.

        REVISION DETAIL
        https://reviews.facebook.net/D1539

        Show
        Phabricator added a comment - zhiqiu has commented on the revision " [jira] HBASE-4542 Add filter info to slow query logging". @mbautin Thanks! Have updated JIRA status. REVISION DETAIL https://reviews.facebook.net/D1539
        Zhiqiu Kong made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Affects Version/s 0.89.20100924 [ 12315366 ]
        Hide
        Phabricator added a comment -

        mbautin has commented on the revision "[jira] HBASE-4542 Add filter info to slow query logging".

        @zhiqiu: don't worry about filling those fields in the "Submit Patch" form. We can correct those fields later if necessary. The purpose of setting JIRA status to "Patch Available" is to make the Apache Jenkins continuous integration server pick up the patch and run a subset of the test suite ("small" and "Medium" tests) against it.

        In some cases, the format of the patch automatically generated by Phabricator and uploaded to JIRA is such that it cannot be applied by the standard patch command for some reason. In those cases, Hadoop QA (Jenkins server) will post an update to the JIRA saying that it could not apply the patch. If that happens, you can generate a patch manually using the following command:

        git format-patch --no-prefix HEAD^..HEAD

        Then, upload this patch to the JIRA (https://issues.apache.org/jira/browse/HBASE-4542) using the "+" button at the top of the "Attachments" section of the page.

        REVISION DETAIL
        https://reviews.facebook.net/D1539

        Show
        Phabricator added a comment - mbautin has commented on the revision " [jira] HBASE-4542 Add filter info to slow query logging". @zhiqiu: don't worry about filling those fields in the "Submit Patch" form. We can correct those fields later if necessary. The purpose of setting JIRA status to "Patch Available" is to make the Apache Jenkins continuous integration server pick up the patch and run a subset of the test suite ("small" and "Medium" tests) against it. In some cases, the format of the patch automatically generated by Phabricator and uploaded to JIRA is such that it cannot be applied by the standard patch command for some reason. In those cases, Hadoop QA (Jenkins server) will post an update to the JIRA saying that it could not apply the patch. If that happens, you can generate a patch manually using the following command: git format-patch --no-prefix HEAD^..HEAD Then, upload this patch to the JIRA ( https://issues.apache.org/jira/browse/HBASE-4542 ) using the "+" button at the top of the "Attachments" section of the page. REVISION DETAIL https://reviews.facebook.net/D1539
        Hide
        Phabricator added a comment -

        zhiqiu has commented on the revision "[jira] HBASE-4542 Add filter info to slow query logging".

        @mbautin Sure! Should I fill in "Affects Version/s", "Release Notes" or other fields in the Submit Patch form?

        REVISION DETAIL
        https://reviews.facebook.net/D1539

        Show
        Phabricator added a comment - zhiqiu has commented on the revision " [jira] HBASE-4542 Add filter info to slow query logging". @mbautin Sure! Should I fill in "Affects Version/s", "Release Notes" or other fields in the Submit Patch form? REVISION DETAIL https://reviews.facebook.net/D1539
        Hide
        Phabricator added a comment -

        mbautin has commented on the revision "[jira] HBASE-4542 [89-fb] Add filter info to slow query logging".

        Hi Zhiqiu, is this a diff against Apache HBase trunk? If so, please remove the [89-fb] tag from the title. Thanks!

        REVISION DETAIL
        https://reviews.facebook.net/D1539

        Show
        Phabricator added a comment - mbautin has commented on the revision " [jira] HBASE-4542 [89-fb] Add filter info to slow query logging". Hi Zhiqiu, is this a diff against Apache HBase trunk? If so, please remove the [89-fb] tag from the title. Thanks! REVISION DETAIL https://reviews.facebook.net/D1539
        Hide
        Phabricator added a comment -

        mbautin has commented on the revision "[jira] HBASE-4542 Add filter info to slow query logging".

        @zhiqiu: thanks for fixing the title. Also, can you add your original 89-fb diff URL in this diff's description and set the JIRA state to "Patch Available"? This is done using the "Submit Patch" button on the JIRA page (https://issues.apache.org/jira/browse/HBASE-4542). There seem to be temporary issues with the Apache JIRA website at the moment but they will probably be resolved in an hour or two.

        REVISION DETAIL
        https://reviews.facebook.net/D1539

        Show
        Phabricator added a comment - mbautin has commented on the revision " [jira] HBASE-4542 Add filter info to slow query logging". @zhiqiu: thanks for fixing the title. Also, can you add your original 89-fb diff URL in this diff's description and set the JIRA state to "Patch Available"? This is done using the "Submit Patch" button on the JIRA page ( https://issues.apache.org/jira/browse/HBASE-4542 ). There seem to be temporary issues with the Apache JIRA website at the moment but they will probably be resolved in an hour or two. REVISION DETAIL https://reviews.facebook.net/D1539
        Hide
        Phabricator added a comment -

        zhiqiu has commented on the revision "[jira] HBASE-4542 Add filter info to slow query logging".

        Thanks for pointing this out, Mikhail! Fixed.

        REVISION DETAIL
        https://reviews.facebook.net/D1539

        Show
        Phabricator added a comment - zhiqiu has commented on the revision " [jira] HBASE-4542 Add filter info to slow query logging". Thanks for pointing this out, Mikhail! Fixed. REVISION DETAIL https://reviews.facebook.net/D1539
        Phabricator made changes -
        Attachment D1539.1.patch [ 12512660 ]
        Hide
        Phabricator added a comment -

        zhiqiu requested code review of "[jira] HBASE-4542 [89-fb] Add filter info to slow query logging".
        Reviewers: Kannan, madhuvaidya, mbautin, JIRA

        Slow opertaion log does not provide enough information when a filter is
        present. The followings are done to add the filter info:

        1) Added toString() method for filters inheriting FilterBase, this
        affect 22 filters and their subclasses. The info added includes the
        filter's name and its members. For example, for TimestampsFilter, we'll
        output its class name as well as the defined timestamps.

        2) Added a field 'filter' in Get::toMap() and Scan::toMap() to enable
        the logging of filter info.

        Task ID: #750975

        Blame Rev:

        TEST PLAN
        1. Run and passed unit-tests to make sure it does not break things

        2. Run kannan's script to trigger the slow operation logging, checked
        for each filter to make sure the filter info was logged. To be more
        detailed, the output log are as following (only 'filter' filed is put
        here for ease of reading):

        "filter":"TimestampsFilter (3/3): [2, 3, 5]"
        "filter":"TimestampsFilter (5/6): [2, 3, 5, 7, 11]"
        "filter":"ColumnPrefixFilter col2"
        "filter":"ColumnRangeFilter [col2a, col2b]"
        "filter":"ColumnCountGetFilter 8"
        "filter":"ColumnPaginationFilter (4, 4)"
        "filter":"InclusiveStopFilter row"
        "filter":"PrefixFilter row"
        "filter":"PageFilter 1"
        "filter":"SkipFilter TimestampsFilter (1/1): [1000]"
        "filter":"WhileMatchFilter TimestampsFilter (3/3): [2, 3, 5]"
        "filter":"KeyOnlyFilter"
        "filter":"FirstKeyOnlyFilter"
        "filter":"MultipleColumnPrefixFilter (3/3): [a, b, c]"
        "filter":"DependentColumnFilter (family, qualifier, true, LESS, value)"
        "filter":"FamilyFilter (LESS, value)"
        "filter":"QualifierFilter (LESS, value)"
        "filter":"RowFilter (LESS, value)"
        "filter":"ValueFilter (LESS, value)"
        "filter":"KeyOnlyFilter"
        "filter":"FirstKeyOnlyFilter"
        "filter":"SingleColumnValueFilter (family, qualifier, EQUAL, value)"
        "filter":"SingleColumnValueExcludeFilter (family, qualifier, EQUAL,
        value)"
        "filter":"FilterList AND (2/2): [KeyOnlyFilter, FirstKeyOnlyFilter]"
        Please check ~zhiqiu/Codes/scripts/testFilter.rb for the testing script.

        3. Added unit test cases to TestOperation to verify the filters'
        toString() method works well.

        Revert Plan:

        Tags:

        REVISION DETAIL
        https://reviews.facebook.net/D1539

        AFFECTED FILES
        src/main/java/org/apache/hadoop/hbase/client/Get.java
        src/main/java/org/apache/hadoop/hbase/client/Scan.java
        src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java
        src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java
        src/main/java/org/apache/hadoop/hbase/filter/ColumnPrefixFilter.java
        src/main/java/org/apache/hadoop/hbase/filter/ColumnRangeFilter.java
        src/main/java/org/apache/hadoop/hbase/filter/CompareFilter.java
        src/main/java/org/apache/hadoop/hbase/filter/DependentColumnFilter.java
        src/main/java/org/apache/hadoop/hbase/filter/FilterBase.java
        src/main/java/org/apache/hadoop/hbase/filter/FilterList.java
        src/main/java/org/apache/hadoop/hbase/filter/InclusiveStopFilter.java
        src/main/java/org/apache/hadoop/hbase/filter/MultipleColumnPrefixFilter.java
        src/main/java/org/apache/hadoop/hbase/filter/PageFilter.java
        src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java
        src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java
        src/main/java/org/apache/hadoop/hbase/filter/SkipFilter.java
        src/main/java/org/apache/hadoop/hbase/filter/TimestampsFilter.java
        src/main/java/org/apache/hadoop/hbase/filter/WhileMatchFilter.java
        src/test/java/org/apache/hadoop/hbase/client/TestOperation.java

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/3231/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - zhiqiu requested code review of " [jira] HBASE-4542 [89-fb] Add filter info to slow query logging". Reviewers: Kannan, madhuvaidya, mbautin, JIRA Slow opertaion log does not provide enough information when a filter is present. The followings are done to add the filter info: 1) Added toString() method for filters inheriting FilterBase, this affect 22 filters and their subclasses. The info added includes the filter's name and its members. For example, for TimestampsFilter, we'll output its class name as well as the defined timestamps. 2) Added a field 'filter' in Get::toMap() and Scan::toMap() to enable the logging of filter info. Task ID: #750975 Blame Rev: TEST PLAN 1. Run and passed unit-tests to make sure it does not break things 2. Run kannan's script to trigger the slow operation logging, checked for each filter to make sure the filter info was logged. To be more detailed, the output log are as following (only 'filter' filed is put here for ease of reading): "filter":"TimestampsFilter (3/3): [2, 3, 5] " "filter":"TimestampsFilter (5/6): [2, 3, 5, 7, 11] " "filter":"ColumnPrefixFilter col2" "filter":"ColumnRangeFilter [col2a, col2b] " "filter":"ColumnCountGetFilter 8" "filter":"ColumnPaginationFilter (4, 4)" "filter":"InclusiveStopFilter row" "filter":"PrefixFilter row" "filter":"PageFilter 1" "filter":"SkipFilter TimestampsFilter (1/1): [1000] " "filter":"WhileMatchFilter TimestampsFilter (3/3): [2, 3, 5] " "filter":"KeyOnlyFilter" "filter":"FirstKeyOnlyFilter" "filter":"MultipleColumnPrefixFilter (3/3): [a, b, c] " "filter":"DependentColumnFilter (family, qualifier, true, LESS, value)" "filter":"FamilyFilter (LESS, value)" "filter":"QualifierFilter (LESS, value)" "filter":"RowFilter (LESS, value)" "filter":"ValueFilter (LESS, value)" "filter":"KeyOnlyFilter" "filter":"FirstKeyOnlyFilter" "filter":"SingleColumnValueFilter (family, qualifier, EQUAL, value)" "filter":"SingleColumnValueExcludeFilter (family, qualifier, EQUAL, value)" "filter":"FilterList AND (2/2): [KeyOnlyFilter, FirstKeyOnlyFilter] " Please check ~zhiqiu/Codes/scripts/testFilter.rb for the testing script. 3. Added unit test cases to TestOperation to verify the filters' toString() method works well. Revert Plan: Tags: REVISION DETAIL https://reviews.facebook.net/D1539 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/client/Get.java src/main/java/org/apache/hadoop/hbase/client/Scan.java src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java src/main/java/org/apache/hadoop/hbase/filter/ColumnPrefixFilter.java src/main/java/org/apache/hadoop/hbase/filter/ColumnRangeFilter.java src/main/java/org/apache/hadoop/hbase/filter/CompareFilter.java src/main/java/org/apache/hadoop/hbase/filter/DependentColumnFilter.java src/main/java/org/apache/hadoop/hbase/filter/FilterBase.java src/main/java/org/apache/hadoop/hbase/filter/FilterList.java src/main/java/org/apache/hadoop/hbase/filter/InclusiveStopFilter.java src/main/java/org/apache/hadoop/hbase/filter/MultipleColumnPrefixFilter.java src/main/java/org/apache/hadoop/hbase/filter/PageFilter.java src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java src/main/java/org/apache/hadoop/hbase/filter/SkipFilter.java src/main/java/org/apache/hadoop/hbase/filter/TimestampsFilter.java src/main/java/org/apache/hadoop/hbase/filter/WhileMatchFilter.java src/test/java/org/apache/hadoop/hbase/client/TestOperation.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/3231/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
        Hide
        Phabricator added a comment -

        mbautin has accepted the revision "[jira] [HBase-4542] [89-fb] Add filter info to slow query logging".

        @zhiqiu: looks good!

        REVISION DETAIL
        https://reviews.facebook.net/D1263

        Show
        Phabricator added a comment - mbautin has accepted the revision " [jira] [HBase-4542] [89-fb] Add filter info to slow query logging". @zhiqiu: looks good! REVISION DETAIL https://reviews.facebook.net/D1263
        Phabricator made changes -
        Attachment D1263.2.patch [ 12511018 ]
        Hide
        Phabricator added a comment -

        zhiqiu requested code review of "[jira] [HBase-4542] [89-fb] Add filter info to slow query logging".
        Reviewers: Kannan, madhuvaidya, mbautin, JIRA

        Slow opertaion log does not provide enough information when a filter is
        present. The followings are done to add the filter info:

        1) Added toString() method for filters inheriting FilterBase, this
        affect 22 filters and their subclasses. The info added includes the
        filter's name and its members. For example, for TimestampsFilter, we'll
        output its class name as well as the defined timestamps.

        2) Added a field 'filter' in Get::toMap() and
        Scan::toMap() to enable the logging of filter info.

        TEST PLAN
        1. Run and passed unit-tests to make sure it does not break things

        2. Run kannan's script to trigger the slow operation logging, checked
        for each filter to make sure the filter info was logged. To be more
        detailed, the output log are as following (only 'filter' filed is put
        here for ease of reading):

        • "filter":"TimestampsFilter (3/3): [2, 3, 5]"
        • "filter":"TimestampsFilter (5/6): [2, 3, 5, 7, 11]"
        • "filter":"ColumnPrefixFilter col2"
        • "filter":"ColumnRangeFilter [col2a, col2b]"
        • "filter":"ColumnCountGetFilter 8"
        • "filter":"ColumnPaginationFilter (4, 4)"
        • "filter":"InclusiveStopFilter row"
        • "filter":"PrefixFilter row"
        • "filter":"PageFilter 1"
        • "filter":"SkipFilter TimestampsFilter (1/1): [1000]"
        • "filter":"WhileMatchFilter TimestampsFilter (3/3): [2, 3, 5]"
        • "filter":"KeyOnlyFilter"
        • "filter":"FirstKeyOnlyFilter"
        • "filter":"MultipleColumnPrefixFilter (3/3): [a, b, c]"
        • "filter":"DependentColumnFilter (family, qualifier, true, LESS, value)"
        • "filter":"FamilyFilter (LESS, value)"
        • "filter":"QualifierFilter (LESS, value)"
        • "filter":"RowFilter (LESS, value)"
        • "filter":"ValueFilter (LESS, value)"
        • "filter":"KeyOnlyFilter"
        • "filter":"FirstKeyOnlyFilter"
        • "filter":"SingleColumnValueFilter (family, qualifier, EQUAL, value)"
        • "filter":"SingleColumnValueExcludeFilter (family, qualifier, EQUAL, value)"
        • "filter":"FilterList AND (2/2): [KeyOnlyFilter, FirstKeyOnlyFilter]"

        Please check ~zhiqiu/Codes/scripts/testFilter.rb for the testing script.

        3. Added unit test cases to TestOperation to verify the filters'
        toString() method works well.

        Revert Plan:

        Tags:

        REVISION DETAIL
        https://reviews.facebook.net/D1263

        AFFECTED FILES
        src/main/java/org/apache/hadoop/hbase/client/Get.java
        src/main/java/org/apache/hadoop/hbase/client/Scan.java
        src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java
        src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java
        src/main/java/org/apache/hadoop/hbase/filter/ColumnPrefixFilter.java
        src/main/java/org/apache/hadoop/hbase/filter/ColumnRangeFilter.java
        src/main/java/org/apache/hadoop/hbase/filter/CompareFilter.java
        src/main/java/org/apache/hadoop/hbase/filter/DependentColumnFilter.java
        src/main/java/org/apache/hadoop/hbase/filter/FilterBase.java
        src/main/java/org/apache/hadoop/hbase/filter/FilterList.java
        src/main/java/org/apache/hadoop/hbase/filter/InclusiveStopFilter.java
        src/main/java/org/apache/hadoop/hbase/filter/MultipleColumnPrefixFilter.java
        src/main/java/org/apache/hadoop/hbase/filter/PageFilter.java
        src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java
        src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java
        src/main/java/org/apache/hadoop/hbase/filter/SkipFilter.java
        src/main/java/org/apache/hadoop/hbase/filter/TimestampsFilter.java
        src/main/java/org/apache/hadoop/hbase/filter/WhileMatchFilter.java
        src/test/java/org/apache/hadoop/hbase/client/TestOperation.java

        Show
        Phabricator added a comment - zhiqiu requested code review of " [jira] [HBase-4542] [89-fb] Add filter info to slow query logging". Reviewers: Kannan, madhuvaidya, mbautin, JIRA Slow opertaion log does not provide enough information when a filter is present. The followings are done to add the filter info: 1) Added toString() method for filters inheriting FilterBase, this affect 22 filters and their subclasses. The info added includes the filter's name and its members. For example, for TimestampsFilter, we'll output its class name as well as the defined timestamps. 2) Added a field 'filter' in Get::toMap() and Scan::toMap() to enable the logging of filter info. TEST PLAN 1. Run and passed unit-tests to make sure it does not break things 2. Run kannan's script to trigger the slow operation logging, checked for each filter to make sure the filter info was logged. To be more detailed, the output log are as following (only 'filter' filed is put here for ease of reading): "filter":"TimestampsFilter (3/3): [2, 3, 5] " "filter":"TimestampsFilter (5/6): [2, 3, 5, 7, 11] " "filter":"ColumnPrefixFilter col2" "filter":"ColumnRangeFilter [col2a, col2b] " "filter":"ColumnCountGetFilter 8" "filter":"ColumnPaginationFilter (4, 4)" "filter":"InclusiveStopFilter row" "filter":"PrefixFilter row" "filter":"PageFilter 1" "filter":"SkipFilter TimestampsFilter (1/1): [1000] " "filter":"WhileMatchFilter TimestampsFilter (3/3): [2, 3, 5] " "filter":"KeyOnlyFilter" "filter":"FirstKeyOnlyFilter" "filter":"MultipleColumnPrefixFilter (3/3): [a, b, c] " "filter":"DependentColumnFilter (family, qualifier, true, LESS, value)" "filter":"FamilyFilter (LESS, value)" "filter":"QualifierFilter (LESS, value)" "filter":"RowFilter (LESS, value)" "filter":"ValueFilter (LESS, value)" "filter":"KeyOnlyFilter" "filter":"FirstKeyOnlyFilter" "filter":"SingleColumnValueFilter (family, qualifier, EQUAL, value)" "filter":"SingleColumnValueExcludeFilter (family, qualifier, EQUAL, value)" "filter":"FilterList AND (2/2): [KeyOnlyFilter, FirstKeyOnlyFilter] " Please check ~zhiqiu/Codes/scripts/testFilter.rb for the testing script. 3. Added unit test cases to TestOperation to verify the filters' toString() method works well. Revert Plan: Tags: REVISION DETAIL https://reviews.facebook.net/D1263 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/client/Get.java src/main/java/org/apache/hadoop/hbase/client/Scan.java src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java src/main/java/org/apache/hadoop/hbase/filter/ColumnPrefixFilter.java src/main/java/org/apache/hadoop/hbase/filter/ColumnRangeFilter.java src/main/java/org/apache/hadoop/hbase/filter/CompareFilter.java src/main/java/org/apache/hadoop/hbase/filter/DependentColumnFilter.java src/main/java/org/apache/hadoop/hbase/filter/FilterBase.java src/main/java/org/apache/hadoop/hbase/filter/FilterList.java src/main/java/org/apache/hadoop/hbase/filter/InclusiveStopFilter.java src/main/java/org/apache/hadoop/hbase/filter/MultipleColumnPrefixFilter.java src/main/java/org/apache/hadoop/hbase/filter/PageFilter.java src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java src/main/java/org/apache/hadoop/hbase/filter/SkipFilter.java src/main/java/org/apache/hadoop/hbase/filter/TimestampsFilter.java src/main/java/org/apache/hadoop/hbase/filter/WhileMatchFilter.java src/test/java/org/apache/hadoop/hbase/client/TestOperation.java
        Kannan Muthukkaruppan made changes -
        Description Slow query log doesn't report filter in effect.

        For example:
        {code}
        (operationTooSlow): \
        {"processingtimems":3468,"client":"10.138.43.206:40035","timeRange": [0,9223372036854775807],\
        "starttimems":1317772005821,"responsesize":42411, \
        "class":"HRegionServer","table":"myTable","families":{"CF1":"ALL"]},\
        "row":"6c3b8efa132f0219b7621ed1e5c8c70b","queuetimems":0,\
        "method":"get","totalColumns":1,"maxVersions":1,"storeLimit":-1}
        {code}

        the above would suggest that all columns of myTable:CF1 are being requested for the given row. But in reality there could be filters in effect (such as ColumnPrefixFilter, ColumnRangeFilter, TimestampsFilter() etc.). We should enhance the slow query log to capture & report this information.

        Slow query log doesn't report filters in effect.

        For example:
        {code}
        (operationTooSlow): \
        {"processingtimems":3468,"client":"10.138.43.206:40035","timeRange": [0,9223372036854775807],\
        "starttimems":1317772005821,"responsesize":42411, \
        "class":"HRegionServer","table":"myTable","families":{"CF1":"ALL"]},\
        "row":"6c3b8efa132f0219b7621ed1e5c8c70b","queuetimems":0,\
        "method":"get","totalColumns":1,"maxVersions":1,"storeLimit":-1}
        {code}

        the above would suggest that all columns of myTable:CF1 are being requested for the given row. But in reality there could be filters in effect (such as ColumnPrefixFilter, ColumnRangeFilter, TimestampsFilter() etc.). We should enhance the slow query log to capture & report this information.

        Kannan Muthukkaruppan made changes -
        Field Original Value New Value
        Assignee Madhuwanti Vaidya [ madhuvaidya ]
        Kannan Muthukkaruppan created issue -

          People

          • Assignee:
            Madhuwanti Vaidya
            Reporter:
            Kannan Muthukkaruppan
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development