Hive
  1. Hive
  2. HIVE-4331 Integrated StorageHandler for Hive and HCat using the HiveStorageHandler
  3. HIVE-5260

Introduce HivePassThroughOutputFormat that allows Hive to use general purpose OutputFormats instead of HiveOutputFormats in StorageHandlers

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.0
    • Fix Version/s: 0.12.0
    • Component/s: HBase Handler, HCatalog
    • Labels:
      None

      Description

      This is a task being created to address the hive-side of HIVE-4331.

      1. HIVE-5260.2.patch
        34 kB
        Viraj Bhat
      2. HIVE-5260.patch
        35 kB
        Viraj Bhat

        Issue Links

          Activity

          Sushanth Sowmyan created issue -
          Sushanth Sowmyan made changes -
          Field Original Value New Value
          Component/s HBase Handler [ 12313461 ]
          Component/s HCatalog [ 12320409 ]
          Sushanth Sowmyan made changes -
          Assignee Viraj Bhat [ viraj ]
          Hide
          Sushanth Sowmyan added a comment -

          Phabricator link put up by viraj : https://reviews.facebook.net/D12063

          Show
          Sushanth Sowmyan added a comment - Phabricator link put up by viraj : https://reviews.facebook.net/D12063
          Hide
          Sushanth Sowmyan added a comment -

          I'm +1 on this. Ashutosh Chauhan, could you please check this?

          Show
          Sushanth Sowmyan added a comment - I'm +1 on this. Ashutosh Chauhan , could you please check this?
          Hide
          Ashutosh Chauhan added a comment -

          I don't see the feedback provided on https://reviews.facebook.net/D12063#comment-2 & https://reviews.facebook.net/D12063#comment-3 getting addressed in latest patch.

          Show
          Ashutosh Chauhan added a comment - I don't see the feedback provided on https://reviews.facebook.net/D12063#comment-2 & https://reviews.facebook.net/D12063#comment-3 getting addressed in latest patch.
          Hide
          Viraj Bhat added a comment -

          Hi Ashutosh are you seeing the latest revision? Viraj

          Show
          Viraj Bhat added a comment - Hi Ashutosh are you seeing the latest revision? Viraj
          Hide
          Sushanth Sowmyan added a comment -

          Ashutosh, FYI, I had issues with the diff between versions of diffs by Phabricator not showing correct diffs. I had to download the unified patch Viraj put up (HIVE-4331.patch) to see the changes.

          Show
          Sushanth Sowmyan added a comment - Ashutosh, FYI, I had issues with the diff between versions of diffs by Phabricator not showing correct diffs. I had to download the unified patch Viraj put up ( HIVE-4331 .patch) to see the changes.
          Sushanth Sowmyan made changes -
          Link This issue blocks HIVE-5261 [ HIVE-5261 ]
          Hide
          Sushanth Sowmyan added a comment -

          Per discussions between Viraj, Ashutosh and myself, we need a couple more changes on this patch. Viraj is currently respinning his patch with these changes:

          Line numbers addendum to Ashutosh's original review:

          73 : ok

          105 : can be improved by doing inner-if once, but will not require respinning by itself. If we have to respin though, we should do this.

          configureTableJobProperties : HBaseConfiguration.addHbaseResources seems to take Configuration, which makes it unnecessary to create copyOfConf. In this case, avoid copy, and avoid the if two lines later as well, where we check if it's an instance of JobConf or not. Respin.

          > possible to change addHBaseDelegationToken to accept Configuration, and instantiate a JobContext to pass into User.getCurrent().obtainAuthTokenForJob(conf,job) form.

          341 : similar comment

          Refactor to do similar things outside the if(), and only different things inside the if-else for inputjobprops/outputjobprops

          100 : fine

          123 : replace concurrenthashmap with string, keep in hivefileformatutils

          Show
          Sushanth Sowmyan added a comment - Per discussions between Viraj, Ashutosh and myself, we need a couple more changes on this patch. Viraj is currently respinning his patch with these changes: Line numbers addendum to Ashutosh's original review: 73 : ok 105 : can be improved by doing inner-if once, but will not require respinning by itself. If we have to respin though, we should do this. configureTableJobProperties : HBaseConfiguration.addHbaseResources seems to take Configuration, which makes it unnecessary to create copyOfConf. In this case, avoid copy, and avoid the if two lines later as well, where we check if it's an instance of JobConf or not. Respin. > possible to change addHBaseDelegationToken to accept Configuration, and instantiate a JobContext to pass into User.getCurrent().obtainAuthTokenForJob(conf,job) form. 341 : similar comment Refactor to do similar things outside the if(), and only different things inside the if-else for inputjobprops/outputjobprops 100 : fine 123 : replace concurrenthashmap with string, keep in hivefileformatutils
          Hide
          Viraj Bhat added a comment -

          Hi Sushanth and Ashutosh,
          Thanks for taking time to discuss the changes at length. I have not attached the patch on the Hive side to take care of things as suggested in the previous comment. We are skipping phabricator. I will make this patch available and let the PRE-COMMIT take over.
          Thanks again
          Viraj

          Show
          Viraj Bhat added a comment - Hi Sushanth and Ashutosh, Thanks for taking time to discuss the changes at length. I have not attached the patch on the Hive side to take care of things as suggested in the previous comment. We are skipping phabricator. I will make this patch available and let the PRE-COMMIT take over. Thanks again Viraj
          Hide
          Viraj Bhat added a comment -

          New patch on the hive side to take care of the comments on 9/10/2013

          Show
          Viraj Bhat added a comment - New patch on the hive side to take care of the comments on 9/10/2013
          Viraj Bhat made changes -
          Attachment HIVE-5260.patch [ 12602666 ]
          Viraj Bhat made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Ashutosh Chauhan added a comment -

          +1

          Show
          Ashutosh Chauhan added a comment - +1
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12602666/HIVE-5260.patch

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/702/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/702/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Tests failed with: NonZeroExitCodeException: Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]]
          + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + cd /data/hive-ptest/working/
          + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-702/source-prep.txt
          + mkdir -p maven ivy
          + [[ svn = \s\v\n ]]
          + [[ -n '' ]]
          + [[ -d apache-svn-trunk-source ]]
          + [[ ! -d apache-svn-trunk-source/.svn ]]
          + [[ ! -d apache-svn-trunk-source ]]
          + cd apache-svn-trunk-source
          + svn revert -R .
          ++ egrep -v '^X|^Performing status on external'
          ++ awk '{print $2}'
          ++ svn status --no-ignore
          + rm -rf
          + svn update
          
          Fetching external item into 'hcatalog/src/test/e2e/harness'
          External at revision 1522097.
          
          At revision 1522097.
          + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
          + patchFilePath=/data/hive-ptest/working/scratch/build.patch
          + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
          + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
          + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
          The patch does not appear to apply with p0 to p2
          + exit 1
          '
          

          This message is automatically generated.

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12602666/HIVE-5260.patch Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/702/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/702/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Tests failed with: NonZeroExitCodeException: Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]] + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-702/source-prep.txt + mkdir -p maven ivy + [[ svn = \s\v\n ]] + [[ -n '' ]] + [[ -d apache-svn-trunk-source ]] + [[ ! -d apache-svn-trunk-source/.svn ]] + [[ ! -d apache-svn-trunk-source ]] + cd apache-svn-trunk-source + svn revert -R . ++ egrep -v '^X|^Performing status on external' ++ awk '{print $2}' ++ svn status --no-ignore + rm -rf + svn update Fetching external item into 'hcatalog/src/test/e2e/harness' External at revision 1522097. At revision 1522097. + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0 to p2 + exit 1 ' This message is automatically generated.
          Hide
          Viraj Bhat added a comment -

          The previous message was caused because the git apply patch worked well but the patch -p1 command does not. Attaching another patch.
          Viraj

          Show
          Viraj Bhat added a comment - The previous message was caused because the git apply patch worked well but the patch -p1 command does not. Attaching another patch. Viraj
          Hide
          Viraj Bhat added a comment -

          patch which will apply cleanly

          Show
          Viraj Bhat added a comment - patch which will apply cleanly
          Viraj Bhat made changes -
          Attachment HIVE-5260.2.patch [ 12602727 ]
          Hide
          Sushanth Sowmyan added a comment -

          I'm +1 on the latest changes - they are in accordance with the requirements discussion we had yesterday.

          Show
          Sushanth Sowmyan added a comment - I'm +1 on the latest changes - they are in accordance with the requirements discussion we had yesterday.
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12602727/HIVE-5260.2.patch

          SUCCESS: +1 3091 tests passed

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/709/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/709/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12602727/HIVE-5260.2.patch SUCCESS: +1 3091 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/709/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/709/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated.
          Hide
          Ashutosh Chauhan added a comment -

          Committed to trunk. Thanks, Viraj!

          Show
          Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Viraj!
          Ashutosh Chauhan made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.13.0 [ 12324986 ]
          Fix Version/s 0.12.0 [ 12324312 ]
          Resolution Fixed [ 1 ]
          Hide
          Viraj Bhat added a comment -

          Thanks Ashutosh and Sushanth for your help.
          Viraj

          Show
          Viraj Bhat added a comment - Thanks Ashutosh and Sushanth for your help. Viraj
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-hadoop2-ptest #96 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/96/)
          HIVE-5260 : Introduce HivePassThroughOutputFormat that allows Hive to use general purpose OutputFormats instead of HiveOutputFormats in StorageHandlers (Viraj Bhat via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522574)

          • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
          • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
          • /hive/trunk/hbase-handler/src/test/results/positive/external_table_ppd.q.out
          • /hive/trunk/hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughOutputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughRecordWriter.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2-ptest #96 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/96/ ) HIVE-5260 : Introduce HivePassThroughOutputFormat that allows Hive to use general purpose OutputFormats instead of HiveOutputFormats in StorageHandlers (Viraj Bhat via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522574 ) /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java /hive/trunk/hbase-handler/src/test/results/positive/external_table_ppd.q.out /hive/trunk/hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughOutputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughRecordWriter.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-hadoop1-ptest #163 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/163/)
          HIVE-5260 : Introduce HivePassThroughOutputFormat that allows Hive to use general purpose OutputFormats instead of HiveOutputFormats in StorageHandlers (Viraj Bhat via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522574)

          • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
          • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
          • /hive/trunk/hbase-handler/src/test/results/positive/external_table_ppd.q.out
          • /hive/trunk/hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughOutputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughRecordWriter.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop1-ptest #163 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/163/ ) HIVE-5260 : Introduce HivePassThroughOutputFormat that allows Hive to use general purpose OutputFormats instead of HiveOutputFormats in StorageHandlers (Viraj Bhat via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522574 ) /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java /hive/trunk/hbase-handler/src/test/results/positive/external_table_ppd.q.out /hive/trunk/hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughOutputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughRecordWriter.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java
          Hide
          Sushanth Sowmyan added a comment -

          Thejas M Nair/Ashutosh Chauhan : Could we get this on to 0.12 as well? We need it there to be able to deprecate the hbase storage handler as of 0.12.

          Show
          Sushanth Sowmyan added a comment - Thejas M Nair / Ashutosh Chauhan : Could we get this on to 0.12 as well? We need it there to be able to deprecate the hbase storage handler as of 0.12.
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-h0.21 #2327 (See https://builds.apache.org/job/Hive-trunk-h0.21/2327/)
          HIVE-5260 : Introduce HivePassThroughOutputFormat that allows Hive to use general purpose OutputFormats instead of HiveOutputFormats in StorageHandlers (Viraj Bhat via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522574)

          • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
          • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
          • /hive/trunk/hbase-handler/src/test/results/positive/external_table_ppd.q.out
          • /hive/trunk/hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughOutputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughRecordWriter.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-h0.21 #2327 (See https://builds.apache.org/job/Hive-trunk-h0.21/2327/ ) HIVE-5260 : Introduce HivePassThroughOutputFormat that allows Hive to use general purpose OutputFormats instead of HiveOutputFormats in StorageHandlers (Viraj Bhat via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522574 ) /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java /hive/trunk/hbase-handler/src/test/results/positive/external_table_ppd.q.out /hive/trunk/hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughOutputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughRecordWriter.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java
          Hide
          Viraj Bhat added a comment -

          Sushanth I think the patch to apply to Hive 0.12 as well. I would be good to have this in Hive 0.12

          Show
          Viraj Bhat added a comment - Sushanth I think the patch to apply to Hive 0.12 as well. I would be good to have this in Hive 0.12
          Hide
          Hudson added a comment -

          ABORTED: Integrated in Hive-trunk-hadoop2 #425 (See https://builds.apache.org/job/Hive-trunk-hadoop2/425/)
          HIVE-5260 : Introduce HivePassThroughOutputFormat that allows Hive to use general purpose OutputFormats instead of HiveOutputFormats in StorageHandlers (Viraj Bhat via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522574)

          • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
          • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
          • /hive/trunk/hbase-handler/src/test/results/positive/external_table_ppd.q.out
          • /hive/trunk/hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughOutputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughRecordWriter.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java
          Show
          Hudson added a comment - ABORTED: Integrated in Hive-trunk-hadoop2 #425 (See https://builds.apache.org/job/Hive-trunk-hadoop2/425/ ) HIVE-5260 : Introduce HivePassThroughOutputFormat that allows Hive to use general purpose OutputFormats instead of HiveOutputFormats in StorageHandlers (Viraj Bhat via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522574 ) /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java /hive/trunk/hbase-handler/src/test/results/positive/external_table_ppd.q.out /hive/trunk/hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughOutputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughRecordWriter.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java
          Thejas M Nair made changes -
          Fix Version/s 0.12.0 [ 12324312 ]
          Fix Version/s 0.13.0 [ 12324986 ]
          Hide
          Thejas M Nair added a comment -

          Patch committed to 0.12 branch as requested.

          Show
          Thejas M Nair added a comment - Patch committed to 0.12 branch as requested.
          Hide
          Viraj Bhat added a comment -

          Thejas; Thanks for the commit to branch 0.12

          Show
          Viraj Bhat added a comment - Thejas; Thanks for the commit to branch 0.12
          Hide
          Ashutosh Chauhan added a comment -

          This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

          Show
          Ashutosh Chauhan added a comment - This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.
          Ashutosh Chauhan made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Show
          Brock Noland added a comment - Hey guys, I am struggling to understand this code. Specifically: 1) We call addHBaseResource three separate times, twice for each code path: https://github.com/apache/hive/blob/3b96b3204e5298bcc252bf6f0f5180bb6119bff1/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java#L331 https://github.com/apache/hive/blob/3b96b3204e5298bcc252bf6f0f5180bb6119bff1/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java#L337 https://github.com/apache/hive/blob/3b96b3204e5298bcc252bf6f0f5180bb6119bff1/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java#L346 2) We copy the jobConf, call addHBaseResources a second time, and then never use the object? https://github.com/apache/hive/blob/3b96b3204e5298bcc252bf6f0f5180bb6119bff1/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java#L345 Any idea what is the purpose of doing these things?
          Brock Noland made changes -
          Link This issue is related to HIVE-6115 [ HIVE-6115 ]
          Hide
          Viraj Bhat added a comment -

          Hi Brock,
          Sorry for getting back to you late on this. Thanks for cleaning the code on this one.
          I think during the merge of the HBaseStorageHandler, some of the code might have got copied in 2 places, before I created a function for it.
          So it is redundant to call: addHBaseResources in 331 which in turn calls later calls HBaseConfiguration.addHbaseResources and also do the same while configuring both input and output job properties.
          You can remove line 345. I think this is a typo.
          Viraj

          Show
          Viraj Bhat added a comment - Hi Brock, Sorry for getting back to you late on this. Thanks for cleaning the code on this one. I think during the merge of the HBaseStorageHandler, some of the code might have got copied in 2 places, before I created a function for it. So it is redundant to call: addHBaseResources in 331 which in turn calls later calls HBaseConfiguration.addHbaseResources and also do the same while configuring both input and output job properties. You can remove line 345. I think this is a typo. Viraj

            People

            • Assignee:
              Viraj Bhat
              Reporter:
              Sushanth Sowmyan
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development