HBase
  1. HBase
  2. HBASE-9800

Impl CoprocessorRowcounter to run in command-line

    Details

    • Type: New Feature New Feature
    • Status: Patch Available
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.94.3
    • Fix Version/s: None
    • Component/s: Coprocessors
    • Labels:
      None

      Description

      We want to count the rows of table daily but the default rowcounter using mapreduce is inefficient. Impl rowcounter using coprocessor makes it better.
      Furthermore, It must provide the mechanism to choose the way to output result.

      1. HBASE-9800-0.94-v2.patch
        7 kB
        chendihao
      2. HBASE-9800-0.94-v1.patch
        7 kB
        chendihao

        Activity

        Hide
        Ted Yu added a comment -

        What's the difference from AggregationClient / AggregateProtocol ?
        See AggregationClient#rowCount()

        Show
        Ted Yu added a comment - What's the difference from AggregationClient / AggregateProtocol ? See AggregationClient#rowCount()
        Hide
        chendihao added a comment -

        Yes, I know AggregationClient.rowCount(). We write a main function to call it from time to time. Because it's not quite real-time, we have a plan to count the rows daily at midnight. It might be helpful for us to know the rough statistics about tables while not affecting the online services.

        Show
        chendihao added a comment - Yes, I know AggregationClient.rowCount(). We write a main function to call it from time to time. Because it's not quite real-time, we have a plan to count the rows daily at midnight. It might be helpful for us to know the rough statistics about tables while not affecting the online services.
        Hide
        chendihao added a comment -

        patch for 0.94

        Show
        chendihao added a comment - patch for 0.94
        Hide
        chendihao added a comment -

        I will put it in coprocessor/example package. Any suggestions? Ted Yu

        Show
        chendihao added a comment - I will put it in coprocessor/example package. Any suggestions? Ted Yu
        Hide
        Ted Yu added a comment -

        Sounds good.

        Show
        Ted Yu added a comment - Sounds good.
        Hide
        chendihao added a comment -

        Ted Yu stack It's a nice feature for us to count the rows. But the issue has been opened for a long time, could you help to review the code or just close it?

        Show
        chendihao added a comment - Ted Yu stack It's a nice feature for us to count the rows. But the issue has been opened for a long time, could you help to review the code or just close it?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12609088/HBASE-9800-0.94-v1.patch
        against trunk revision .
        ATTACHMENT ID: 12609088

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 lineLengths. The patch introduces the following lines longer than 100:
        + String sinkClassName = conf.get("coprocessor.rowcounter.sink", "org.apache.hadoop.hbase.coprocessor.example.CoprocessorRowcounter$StdOutSink");
        + bufferedWriter = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(("rowcounter_result"))));
        + put.add(Bytes.toBytes(columnFamily), Bytes.toBytes(columnQualifier), Bytes.toBytes(String.valueOf(rowCount)));

        +1 site. The mvn site goal succeeds with this patch.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface

        -1 core zombie tests. There are 1 zombie test(s):

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8957//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8957//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8957//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8957//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8957//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8957//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8957//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8957//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8957//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8957//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8957//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12609088/HBASE-9800-0.94-v1.patch against trunk revision . ATTACHMENT ID: 12609088 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: + String sinkClassName = conf.get("coprocessor.rowcounter.sink", "org.apache.hadoop.hbase.coprocessor.example.CoprocessorRowcounter$StdOutSink"); + bufferedWriter = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(("rowcounter_result")))); + put.add(Bytes.toBytes(columnFamily), Bytes.toBytes(columnQualifier), Bytes.toBytes(String.valueOf(rowCount))); +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface -1 core zombie tests . There are 1 zombie test(s): Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8957//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8957//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8957//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8957//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8957//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8957//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8957//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8957//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8957//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8957//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8957//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        Year is not needed in license header.

        +  public static class LocalFileSink implements Sink{
        

        Where is LocalFileSink used ?

        +          Put put = new Put(Bytes.toBytes(tableName+"_"+date));
        

        Why is date needed in row key ?

        Can you attach patch for trunk ?

        Show
        Ted Yu added a comment - Year is not needed in license header. + public static class LocalFileSink implements Sink{ Where is LocalFileSink used ? + Put put = new Put(Bytes.toBytes(tableName+ "_" +date)); Why is date needed in row key ? Can you attach patch for trunk ?
        Hide
        chendihao added a comment -

        Thank Ted Yu for reviewing.

        1. I will update the license header.
        2. LocalFileSink can be used when we set "coprocessor.rowcounter.sink" as "org.apache.hadoop.hbase.coprocessor.example.CoprocessorRowcounter$LocalFileSink" in the configuration file.
        3. We periodically count the rows, so we need to indicate the date of the data. It's not so general but meets our requirements.
        4. The patch for trunk will be uploaded later if we need it.

        Show
        chendihao added a comment - Thank Ted Yu for reviewing. 1. I will update the license header. 2. LocalFileSink can be used when we set "coprocessor.rowcounter.sink" as "org.apache.hadoop.hbase.coprocessor.example.CoprocessorRowcounter$LocalFileSink" in the configuration file. 3. We periodically count the rows, so we need to indicate the date of the data. It's not so general but meets our requirements. 4. The patch for trunk will be uploaded later if we need it.
        Hide
        chendihao added a comment -

        patch for 0.94 v2

        Show
        chendihao added a comment - patch for 0.94 v2
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12634418/HBASE-9800-0.94-v2.patch
        against trunk revision .
        ATTACHMENT ID: 12634418

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

        -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8967//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8967//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8967//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8967//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8967//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8967//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8967//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8967//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8967//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8967//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8967//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12634418/HBASE-9800-0.94-v2.patch against trunk revision . ATTACHMENT ID: 12634418 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. -1 javadoc . The javadoc tool appears to have generated 2 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8967//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8967//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8967//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8967//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8967//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8967//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8967//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8967//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8967//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8967//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8967//console This message is automatically generated.

          People

          • Assignee:
            chendihao
            Reporter:
            chendihao
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development