Hive
  1. Hive
  2. HIVE-4995

select * may incorrectly return empty fields with hbase-handler

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.11.0
    • Fix Version/s: 0.12.0
    • Component/s: HBase Handler
    • Labels:
      None

      Description

      HIVE-3725 added capability to pull hbase columns with prefixes. However the way the current logic to add columns stands in HiveHBaseTableInput format, it might cause some columns to incorrectly display empty fields.

      Consider the following query:

      CREATE EXTERNAL TABLE test_table(key string, value1 map<string,string>, value2 string)
      ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe' 
      STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' 
      WITH SERDEPROPERTIES 
      ("hbase.columns.mapping" = ":key,cf-a:prefix.*,cf-a:another_col") 
      TBLPROPERTIES ("hbase.table.name" = "test_table");
      

      Given the existing logic in HiveHBaseTableInputFormat:

      for (int i = 0; i < columnsMapping.size(); i++) 
      {
              ColumnMapping colMap = columnsMapping.get(i);
              if (colMap.hbaseRowKey) {
                continue;
              }
      
              if (colMap.qualifierName == null) {
                scan.addFamily(colMap.familyNameBytes);
              } else {
                scan.addColumn(colMap.familyNameBytes, colMap.qualifierNameBytes);
              }
      }
      

      So for the above query, the 'addFamily' will be called first followed by 'addColumn' for the column family "cf-a". This will wipe away whatever we had set with the 'addFamily' call in the previous step resulting in an empty column when queried.

      1. HIVE-4995.1.patch.txt
        19 kB
        Swarnim Kulkarni
      2. HIVE-4995.1.patch.txt
        19 kB
        Swarnim Kulkarni

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        13m 14s 1 Swarnim Kulkarni 04/Aug/13 16:35
        Patch Available Patch Available Resolved Resolved
        1d 23h 1m 1 Brock Noland 06/Aug/13 15:36
        Resolved Resolved Closed Closed
        70d 8h 52m 1 Ashutosh Chauhan 16/Oct/13 00:29
        Ashutosh Chauhan made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Ashutosh Chauhan added a comment -

        This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

        Show
        Ashutosh Chauhan added a comment - This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hive-trunk-h0.21 #2249 (See https://builds.apache.org/job/Hive-trunk-h0.21/2249/)
        HIVE-4995: select * may incorrectly return empty fields with hbase-handler (Swarnim Kulkarni via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1510973)

        • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
        • /hive/trunk/hbase-handler/src/test/queries/positive/hbase_binary_map_queries_prefix.q
        • /hive/trunk/hbase-handler/src/test/results/positive/hbase_binary_map_queries_prefix.q.out
        Show
        Hudson added a comment - FAILURE: Integrated in Hive-trunk-h0.21 #2249 (See https://builds.apache.org/job/Hive-trunk-h0.21/2249/ ) HIVE-4995 : select * may incorrectly return empty fields with hbase-handler (Swarnim Kulkarni via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1510973 ) /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java /hive/trunk/hbase-handler/src/test/queries/positive/hbase_binary_map_queries_prefix.q /hive/trunk/hbase-handler/src/test/results/positive/hbase_binary_map_queries_prefix.q.out
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #119 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/119/)
        HIVE-4995: select * may incorrectly return empty fields with hbase-handler (Swarnim Kulkarni via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1510973)

        • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
        • /hive/trunk/hbase-handler/src/test/queries/positive/hbase_binary_map_queries_prefix.q
        • /hive/trunk/hbase-handler/src/test/results/positive/hbase_binary_map_queries_prefix.q.out
        Show
        Hudson added a comment - SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #119 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/119/ ) HIVE-4995 : select * may incorrectly return empty fields with hbase-handler (Swarnim Kulkarni via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1510973 ) /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java /hive/trunk/hbase-handler/src/test/queries/positive/hbase_binary_map_queries_prefix.q /hive/trunk/hbase-handler/src/test/results/positive/hbase_binary_map_queries_prefix.q.out
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hive-trunk-hadoop2-ptest #48 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/48/)
        HIVE-4995: select * may incorrectly return empty fields with hbase-handler (Swarnim Kulkarni via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1510973)

        • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
        • /hive/trunk/hbase-handler/src/test/queries/positive/hbase_binary_map_queries_prefix.q
        • /hive/trunk/hbase-handler/src/test/results/positive/hbase_binary_map_queries_prefix.q.out
        Show
        Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2-ptest #48 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/48/ ) HIVE-4995 : select * may incorrectly return empty fields with hbase-handler (Swarnim Kulkarni via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1510973 ) /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java /hive/trunk/hbase-handler/src/test/queries/positive/hbase_binary_map_queries_prefix.q /hive/trunk/hbase-handler/src/test/results/positive/hbase_binary_map_queries_prefix.q.out
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hive-trunk-hadoop2 #332 (See https://builds.apache.org/job/Hive-trunk-hadoop2/332/)
        HIVE-4995: select * may incorrectly return empty fields with hbase-handler (Swarnim Kulkarni via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1510973)

        • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
        • /hive/trunk/hbase-handler/src/test/queries/positive/hbase_binary_map_queries_prefix.q
        • /hive/trunk/hbase-handler/src/test/results/positive/hbase_binary_map_queries_prefix.q.out
        Show
        Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2 #332 (See https://builds.apache.org/job/Hive-trunk-hadoop2/332/ ) HIVE-4995 : select * may incorrectly return empty fields with hbase-handler (Swarnim Kulkarni via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1510973 ) /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java /hive/trunk/hbase-handler/src/test/queries/positive/hbase_binary_map_queries_prefix.q /hive/trunk/hbase-handler/src/test/results/positive/hbase_binary_map_queries_prefix.q.out
        Brock Noland made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.12.0 [ 12324312 ]
        Resolution Fixed [ 1 ]
        Hide
        Brock Noland added a comment -

        Committed to trunk! Thank you for your contribution!

        Show
        Brock Noland added a comment - Committed to trunk! Thank you for your contribution!
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12596171/HIVE-4995.1.patch.txt

        ERROR: -1 due to 1 failed/errored test(s), 2760 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1
        

        Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/310/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/310/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests failed with: TestsFailedException: 1 tests failed
        

        This message is automatically generated.

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12596171/HIVE-4995.1.patch.txt ERROR: -1 due to 1 failed/errored test(s), 2760 tests executed Failed tests: org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1 Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/310/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/310/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 1 tests failed This message is automatically generated.
        Hide
        Brock Noland added a comment -

        +1

        Show
        Brock Noland added a comment - +1
        Swarnim Kulkarni made changes -
        Attachment HIVE-4995.1.patch.txt [ 12596171 ]
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12595810/HIVE-4995.1.patch.txt

        ERROR: -1 due to 1 failed/errored test(s), 2759 tests executed
        Failed tests:

        org.apache.hcatalog.mapreduce.TestSequenceFileReadWrite.testTextTableWriteReadMR
        

        Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/301/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/301/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests failed with: TestsFailedException: 1 tests failed
        

        This message is automatically generated.

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12595810/HIVE-4995.1.patch.txt ERROR: -1 due to 1 failed/errored test(s), 2759 tests executed Failed tests: org.apache.hcatalog.mapreduce.TestSequenceFileReadWrite.testTextTableWriteReadMR Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/301/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/301/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 1 tests failed This message is automatically generated.
        Hide
        Swarnim Kulkarni added a comment -

        This should be ready for review.

        Show
        Swarnim Kulkarni added a comment - This should be ready for review.
        Swarnim Kulkarni made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Swarnim Kulkarni added a comment -
        Show
        Swarnim Kulkarni added a comment - Review request: https://reviews.apache.org/r/13263/
        Swarnim Kulkarni made changes -
        Field Original Value New Value
        Attachment HIVE-4995.1.patch.txt [ 12595810 ]
        Swarnim Kulkarni created issue -

          People

          • Assignee:
            Swarnim Kulkarni
            Reporter:
            Swarnim Kulkarni
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development