Hive
  1. Hive
  2. HIVE-4995

select * may incorrectly return empty fields with hbase-handler

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.11.0
    • Fix Version/s: 0.12.0
    • Component/s: HBase Handler
    • Labels:
      None

      Description

      HIVE-3725 added capability to pull hbase columns with prefixes. However the way the current logic to add columns stands in HiveHBaseTableInput format, it might cause some columns to incorrectly display empty fields.

      Consider the following query:

      CREATE EXTERNAL TABLE test_table(key string, value1 map<string,string>, value2 string)
      ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe' 
      STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' 
      WITH SERDEPROPERTIES 
      ("hbase.columns.mapping" = ":key,cf-a:prefix.*,cf-a:another_col") 
      TBLPROPERTIES ("hbase.table.name" = "test_table");
      

      Given the existing logic in HiveHBaseTableInputFormat:

      for (int i = 0; i < columnsMapping.size(); i++) 
      {
              ColumnMapping colMap = columnsMapping.get(i);
              if (colMap.hbaseRowKey) {
                continue;
              }
      
              if (colMap.qualifierName == null) {
                scan.addFamily(colMap.familyNameBytes);
              } else {
                scan.addColumn(colMap.familyNameBytes, colMap.qualifierNameBytes);
              }
      }
      

      So for the above query, the 'addFamily' will be called first followed by 'addColumn' for the column family "cf-a". This will wipe away whatever we had set with the 'addFamily' call in the previous step resulting in an empty column when queried.

      1. HIVE-4995.1.patch.txt
        19 kB
        Swarnim Kulkarni
      2. HIVE-4995.1.patch.txt
        19 kB
        Swarnim Kulkarni

        Activity

        Hide
        Ashutosh Chauhan added a comment -

        This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

        Show
        Ashutosh Chauhan added a comment - This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hive-trunk-h0.21 #2249 (See https://builds.apache.org/job/Hive-trunk-h0.21/2249/)
        HIVE-4995: select * may incorrectly return empty fields with hbase-handler (Swarnim Kulkarni via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1510973)

        • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
        • /hive/trunk/hbase-handler/src/test/queries/positive/hbase_binary_map_queries_prefix.q
        • /hive/trunk/hbase-handler/src/test/results/positive/hbase_binary_map_queries_prefix.q.out
        Show
        Hudson added a comment - FAILURE: Integrated in Hive-trunk-h0.21 #2249 (See https://builds.apache.org/job/Hive-trunk-h0.21/2249/ ) HIVE-4995 : select * may incorrectly return empty fields with hbase-handler (Swarnim Kulkarni via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1510973 ) /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java /hive/trunk/hbase-handler/src/test/queries/positive/hbase_binary_map_queries_prefix.q /hive/trunk/hbase-handler/src/test/results/positive/hbase_binary_map_queries_prefix.q.out
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #119 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/119/)
        HIVE-4995: select * may incorrectly return empty fields with hbase-handler (Swarnim Kulkarni via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1510973)

        • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
        • /hive/trunk/hbase-handler/src/test/queries/positive/hbase_binary_map_queries_prefix.q
        • /hive/trunk/hbase-handler/src/test/results/positive/hbase_binary_map_queries_prefix.q.out
        Show
        Hudson added a comment - SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #119 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/119/ ) HIVE-4995 : select * may incorrectly return empty fields with hbase-handler (Swarnim Kulkarni via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1510973 ) /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java /hive/trunk/hbase-handler/src/test/queries/positive/hbase_binary_map_queries_prefix.q /hive/trunk/hbase-handler/src/test/results/positive/hbase_binary_map_queries_prefix.q.out
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hive-trunk-hadoop2-ptest #48 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/48/)
        HIVE-4995: select * may incorrectly return empty fields with hbase-handler (Swarnim Kulkarni via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1510973)

        • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
        • /hive/trunk/hbase-handler/src/test/queries/positive/hbase_binary_map_queries_prefix.q
        • /hive/trunk/hbase-handler/src/test/results/positive/hbase_binary_map_queries_prefix.q.out
        Show
        Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2-ptest #48 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/48/ ) HIVE-4995 : select * may incorrectly return empty fields with hbase-handler (Swarnim Kulkarni via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1510973 ) /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java /hive/trunk/hbase-handler/src/test/queries/positive/hbase_binary_map_queries_prefix.q /hive/trunk/hbase-handler/src/test/results/positive/hbase_binary_map_queries_prefix.q.out
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hive-trunk-hadoop2 #332 (See https://builds.apache.org/job/Hive-trunk-hadoop2/332/)
        HIVE-4995: select * may incorrectly return empty fields with hbase-handler (Swarnim Kulkarni via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1510973)

        • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
        • /hive/trunk/hbase-handler/src/test/queries/positive/hbase_binary_map_queries_prefix.q
        • /hive/trunk/hbase-handler/src/test/results/positive/hbase_binary_map_queries_prefix.q.out
        Show
        Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2 #332 (See https://builds.apache.org/job/Hive-trunk-hadoop2/332/ ) HIVE-4995 : select * may incorrectly return empty fields with hbase-handler (Swarnim Kulkarni via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1510973 ) /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java /hive/trunk/hbase-handler/src/test/queries/positive/hbase_binary_map_queries_prefix.q /hive/trunk/hbase-handler/src/test/results/positive/hbase_binary_map_queries_prefix.q.out
        Hide
        Brock Noland added a comment -

        Committed to trunk! Thank you for your contribution!

        Show
        Brock Noland added a comment - Committed to trunk! Thank you for your contribution!
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12596171/HIVE-4995.1.patch.txt

        ERROR: -1 due to 1 failed/errored test(s), 2760 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1
        

        Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/310/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/310/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests failed with: TestsFailedException: 1 tests failed
        

        This message is automatically generated.

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12596171/HIVE-4995.1.patch.txt ERROR: -1 due to 1 failed/errored test(s), 2760 tests executed Failed tests: org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1 Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/310/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/310/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 1 tests failed This message is automatically generated.
        Hide
        Brock Noland added a comment -

        +1

        Show
        Brock Noland added a comment - +1
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12595810/HIVE-4995.1.patch.txt

        ERROR: -1 due to 1 failed/errored test(s), 2759 tests executed
        Failed tests:

        org.apache.hcatalog.mapreduce.TestSequenceFileReadWrite.testTextTableWriteReadMR
        

        Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/301/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/301/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests failed with: TestsFailedException: 1 tests failed
        

        This message is automatically generated.

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12595810/HIVE-4995.1.patch.txt ERROR: -1 due to 1 failed/errored test(s), 2759 tests executed Failed tests: org.apache.hcatalog.mapreduce.TestSequenceFileReadWrite.testTextTableWriteReadMR Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/301/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/301/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 1 tests failed This message is automatically generated.
        Hide
        Swarnim Kulkarni added a comment -

        This should be ready for review.

        Show
        Swarnim Kulkarni added a comment - This should be ready for review.
        Hide
        Swarnim Kulkarni added a comment -
        Show
        Swarnim Kulkarni added a comment - Review request: https://reviews.apache.org/r/13263/

          People

          • Assignee:
            Swarnim Kulkarni
            Reporter:
            Swarnim Kulkarni
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development