Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2085

HBaseStorage fails with multiple STORE statements

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Scripts with multiple STORE statements using HBaseStorage fail when run against a cluster (they succeed in local mode). Below is an example script:

      raw = LOAD 'hbase_split_load_bug.txt' AS
            (f1: chararray, f2:chararray);
      
      SPLIT raw INTO apples IF (f2 == 'apple'), oranges IF (f2 == 'orange');
      
      STORE apples INTO 'hbase://test_table'
         USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('info:apple');
      
      STORE oranges INTO 'hbase://test_table'
         USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('info:orange');
      

      The server throws the following exception after apples is successfully stored:

      Backend error message
      ---------------------
      java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@6273305c closed
              at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:566)
              at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1113)
              at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1233)
              at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:819)
              at org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.close(TableOutputFormat.java:106)
              at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReducePOStoreImpl.tearDown(MapReducePOStoreImpl.java:96)
              at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.tearDown(POStore.java:122)
              at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.cleanup(PigMapBase.java:128)
              at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
              at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
              at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
              at org.apache.hadoop.mapred.Child.main(Child.java:170)
      

        Attachments

        1. PIG-2085_example_input.txt
          0.1 kB
          Bill Graham
        2. PIG-2085_schema.hbase
          0.1 kB
          Bill Graham
        3. PIG-2085_example_script.pig
          0.5 kB
          Bill Graham

          Activity

            People

            • Assignee:
              billgraham Bill Graham
              Reporter:
              billgraham Bill Graham
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: