Pig
  1. Pig
  2. PIG-2615

HBase region observer coprocessor not firing from Pig 'store'

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.9.2
    • Fix Version/s: None
    • Component/s: data
    • Labels:
    • Environment:

      Pig 0.9.2, HBase 0.92, Hadoop 1.0.0

      Description

      When using pig to load data into HBase the region observer coprocessor (specifically prePut() as I have not tested all the other possible region observer combinations) with the 'store' command the coprocessor is not getting invoked.

      I have tracked this down and come up with a simple patch. A record of the investigation is on both the Pig and HBase user forums.

        Activity

        Hide
        Nick Pinckernell added a comment -

        I've checked pig 0.10.0 and the issue still remains. Also, my above comment may be a little off as I don't recall exactly what HBase did not like about the 'ts' that was being generated by pig. My test was specifically with a RegionObserver prePut coprocessor, I was not testing any of the other hooks.

        Show
        Nick Pinckernell added a comment - I've checked pig 0.10.0 and the issue still remains. Also, my above comment may be a little off as I don't recall exactly what HBase did not like about the 'ts' that was being generated by pig. My test was specifically with a RegionObserver prePut coprocessor, I was not testing any of the other hooks.
        Hide
        Nick Pinckernell added a comment -

        Here is the conversation that was had at the HBase forum:
        http://mail-archives.apache.org/mod_mbox/hbase-user/201203.mbox/%3CCABsY1jQ8+OiLh7SYkXZ8iO=NosY8khz7iYs+6W4u6SxCpJ5s9g@mail.gmail.com%3E

        For pig 0.9.2 the issue looked like when pig was invoking the put.add() method in the HBaseStorage class that the ts that was being passed was incorrect and the coprocessor would not fire for those rows with that version of the 'ts' could not be found. I have not checked pig 0.10.0 yet.

        Show
        Nick Pinckernell added a comment - Here is the conversation that was had at the HBase forum: http://mail-archives.apache.org/mod_mbox/hbase-user/201203.mbox/%3CCABsY1jQ8+OiLh7SYkXZ8iO=NosY8khz7iYs+6W4u6SxCpJ5s9g@mail.gmail.com%3E For pig 0.9.2 the issue looked like when pig was invoking the put.add() method in the HBaseStorage class that the ts that was being passed was incorrect and the coprocessor would not fire for those rows with that version of the 'ts' could not be found. I have not checked pig 0.10.0 yet.
        Hide
        Bill Graham added a comment -

        Changing from Patch Available to Open.

        Why does providing a ts or not effect the firing of coprocessors?

        Show
        Bill Graham added a comment - Changing from Patch Available to Open. Why does providing a ts or not effect the firing of coprocessors?
        Hide
        Nick Pinckernell added a comment -

        Dmitriy, the latest from SVN seems to suggest that HBaseStorage is still using hbase-0.90.0. I am unable to add any coprocessor unit tests until HBaseStorage officially supports HBase 0.92. After a quick search the only JIRA I found referencing hbase-0.92 was here: https://issues.apache.org/jira/browse/PIG-2347

        Show
        Nick Pinckernell added a comment - Dmitriy, the latest from SVN seems to suggest that HBaseStorage is still using hbase-0.90.0. I am unable to add any coprocessor unit tests until HBaseStorage officially supports HBase 0.92. After a quick search the only JIRA I found referencing hbase-0.92 was here: https://issues.apache.org/jira/browse/PIG-2347
        Hide
        Nick Pinckernell added a comment -

        I'll get that test case added in the next few days. I did think about that case of custom timestamps and think it would be useful. If there are any feature requests to that effect or I need it for my purposes then I may code up a feature patch.

        Show
        Nick Pinckernell added a comment - I'll get that test case added in the next few days. I did think about that case of custom timestamps and think it would be useful. If there are any feature requests to that effect or I need it for my purposes then I may code up a feature patch.
        Hide
        Dmitriy V. Ryaboy added a comment -

        Oh wow that was a small patch .
        Nick, can you add a testcase?

        What do you think of uses that require users to manually control timestamps? Will doing so completely prevent coprocessors from firing? This isn't possible right now anyway, but I can see wanting to add this in the near future.

        Show
        Dmitriy V. Ryaboy added a comment - Oh wow that was a small patch . Nick, can you add a testcase? What do you think of uses that require users to manually control timestamps? Will doing so completely prevent coprocessors from firing? This isn't possible right now anyway, but I can see wanting to add this in the near future.
        Hide
        Dmitriy V. Ryaboy added a comment -

        Assigning to Nick.
        Sorry this went unnoticed. Will review.

        Show
        Dmitriy V. Ryaboy added a comment - Assigning to Nick. Sorry this went unnoticed. Will review.
        Hide
        Nick Pinckernell added a comment -

        Patch

        Show
        Nick Pinckernell added a comment - Patch

          People

          • Assignee:
            Nick Pinckernell
            Reporter:
            Nick Pinckernell
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development