Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.2, 1.4.3
    • Fix Version/s: 1.4.4
    • Component/s: None
    • Labels:
      None
    • Environment:

      All 1.x sqoop version

      Description

      Apache HCatalog is a table and storage management service that provides a shared schema, data types and table abstraction freeing users from being concerned about where or how their data is stored. It provides interoperability across Pig, Map Reduce, and Hive.

      A sqoop hcatalog connector will help in supporting storage formats that are abstracted by HCatalog.

      1. SQOOP HCatalog Integration.pdf
        119 kB
        Venkat Ranganathan
      2. SQOOP-931.patch
        174 kB
        Venkat Ranganathan
      3. SQOOP HCatalog Integration - 2.pdf
        132 kB
        Venkat Ranganathan
      4. SQOOP HCatalog Integration - 3.pdf
        134 kB
        Venkat Ranganathan
      5. SQOOP-931.patch.14
        236 kB
        Venkat Ranganathan

        Issue Links

          Activity

          Hide
          Jarek Jarcec Cecho added a comment -

          Hi Venkat Ranganathan,
          thank you for opening this ticket, I believe that HCatalog integration is definitely something worth exploring. Would you mind sharing your thoughts about integrating as a new Connector?

          The reason I'm asking is that I personally was thinking about rather integrating with HCatalog on a "storage" level rather than connector level. For example I was thinking about use case to allow users import data from MySQL and store them in HCatalog similarly as storing them in Hive, HBase or other Hadoop ecosystem tool/format.

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - Hi Venkat Ranganathan , thank you for opening this ticket, I believe that HCatalog integration is definitely something worth exploring. Would you mind sharing your thoughts about integrating as a new Connector? The reason I'm asking is that I personally was thinking about rather integrating with HCatalog on a "storage" level rather than connector level. For example I was thinking about use case to allow users import data from MySQL and store them in HCatalog similarly as storing them in Hive, HBase or other Hadoop ecosystem tool/format. Jarcec
          Hide
          Venkat Ranganathan added a comment -

          I will be uploading my design shortly. Yes, the intention is to make use of HCatalog storage abstraction features so that storage formats readily supported by Hcatalog (like RCFile/ORCFile etc) are available as Sqoop sources and destinations.

          I was not really sure how to categorize it. Yes, we are not looking at a JDBC provider model for Hcatalog like the connection manager etc, but we need to support the ability to use HCatalog specific input and output formats and may also do some schema mapping.

          May be I will reclassify it as general for now to avoid confusion

          Show
          Venkat Ranganathan added a comment - I will be uploading my design shortly. Yes, the intention is to make use of HCatalog storage abstraction features so that storage formats readily supported by Hcatalog (like RCFile/ORCFile etc) are available as Sqoop sources and destinations. I was not really sure how to categorize it. Yes, we are not looking at a JDBC provider model for Hcatalog like the connection manager etc, but we need to support the ability to use HCatalog specific input and output formats and may also do some schema mapping. May be I will reclassify it as general for now to avoid confusion
          Hide
          Jarek Jarcec Cecho added a comment -

          Och I see, that make sense. Thank you for working on this sir!

          Show
          Jarek Jarcec Cecho added a comment - Och I see, that make sense. Thank you for working on this sir!
          Hide
          Venkat Ranganathan added a comment -

          Document describing the Sqoop HCatalog integration

          Show
          Venkat Ranganathan added a comment - Document describing the Sqoop HCatalog integration
          Hide
          Jarek Jarcec Cecho added a comment -

          Hi Venkat Ranganathan,
          thank you very much for working on this! I've read the proposal and I do have couple of notes:

          • Can we introduce --hcatalog-database parameter similarly as we added for Hive in SQOOP-912? I do have two main reasons for that - having the database inside the table parameter is inconsistent with the rest of the framework (--table will not accept database) which is very confusing for the end user and it unfortunately won't work for import-all-tables tool.
          • I do understand the reasoning with using HCatalog as a nice way to get support for all additional output types. However I feel that forcing user to create the table definition prior import is against the idea of Sqoop and might be show stopper for most of the users. The most obvious advantage of using Sqoop is that it can propagate all the metadata for user automatically. I would strongly prefer to have option like --create-hcatalog-table from day one.
          • I would like to understand what are the possible implications of not supporting --drop-hive-import-delims. The reason why we've introduced this parameter is that we were creating "shadows" rows as rows containing new line character were spitted into two lines and thus were seen as two rows by Hive. I assume that this is not an issue for all output types supported by HCatalog (Avro, SequenceFile), but what about normal text files?
          • Not supporting --direct option is a bummer, but I guess that we can live with it. Would be possible in such case to just import data using usual means and load them into HCatalog similarly as we're doing in Hive? I do understand that this will limit our option of reusing HCatalog SerDe's in this case.
          • It seems that we're proposing only manual tests that requires setup of third party dependencies. As this is very significant feature, I would argue for having normal tests that will be running during usual jenkins builds. Can we reuse some sort of MiniHCatalogCluster like for Hadoop/Mr/Hive case?

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - Hi Venkat Ranganathan , thank you very much for working on this! I've read the proposal and I do have couple of notes: Can we introduce --hcatalog-database parameter similarly as we added for Hive in SQOOP-912 ? I do have two main reasons for that - having the database inside the table parameter is inconsistent with the rest of the framework ( --table will not accept database) which is very confusing for the end user and it unfortunately won't work for import-all-tables tool. I do understand the reasoning with using HCatalog as a nice way to get support for all additional output types. However I feel that forcing user to create the table definition prior import is against the idea of Sqoop and might be show stopper for most of the users. The most obvious advantage of using Sqoop is that it can propagate all the metadata for user automatically. I would strongly prefer to have option like --create-hcatalog-table from day one. I would like to understand what are the possible implications of not supporting --drop-hive-import-delims . The reason why we've introduced this parameter is that we were creating "shadows" rows as rows containing new line character were spitted into two lines and thus were seen as two rows by Hive. I assume that this is not an issue for all output types supported by HCatalog (Avro, SequenceFile), but what about normal text files? Not supporting --direct option is a bummer, but I guess that we can live with it. Would be possible in such case to just import data using usual means and load them into HCatalog similarly as we're doing in Hive? I do understand that this will limit our option of reusing HCatalog SerDe's in this case. It seems that we're proposing only manual tests that requires setup of third party dependencies. As this is very significant feature, I would argue for having normal tests that will be running during usual jenkins builds. Can we reuse some sort of MiniHCatalogCluster like for Hadoop/Mr/Hive case? Jarcec
          Hide
          Venkat Ranganathan added a comment -

          THanks Jarek Jarcec Cecho for your review and comments.

          I initially had the database separately, but the hcatalog team thought it made more sense to have them together (as they process tables with that format). But adding a database column should not be difficult if we need to make it more compliant.

          It is true that schema inference is a great feature. I thought of adding it in a follow on JIRA with some additional constructs such that we still give the user storage type independence if they so desire. For example if they want all their tables to be whatever they choose (and supported by HCatalog) as the default and pre-create tables if specific output file type is desired. I will create a SUB task and get it in this task itself.

          The main issue with not supporting hive-drop-delims is that string columns with embedded delimiter chars and using delimited text format will have the fidelity issues that the current users have. I considered that but was not sure if it was worth the extra processing for all output types. I wanted to sqoop code to be agnostic of the storage format (so not worry about querying the metadata on the storage info), and users still have the option of using the current hive import to deal with that case if it is so desired which is well understood.

          Direct option does not deal with sqoop record type, so we have to come up with a HCat implementation for each connection manager based on its input/output row formats after parsing it. For example, in the case of Netezza direct mode, Sqoop ORM scheme is not involved so we don't even generate the jar files. I think the existing Hive import mechanism can be used where applicable (not sure if it works with all connection managers, but since the output is text format, the existing hive import support should help with that). As you know, HCatalog uses hive metastore, so such tables are also available to HCatalog users.

          Regarding running as part of the normal test suite, I totally understand. I also did not want it to be a manual test. If you look at the test utils, I use a mini cluster like MR job to do the loading into HCatalog and reading off HCatalog. HCatalog does not have a HCatalogMiniCluster (for unit testing). When I first tried to run everything in local mode (which is supported), then the hive tests failed (because we depend on lack of some classes to distinguish between hive external cli or in process invocation). That is why I had to exclude some of the Hive classes from the dependencies to make it run all unit tests. Let me see if there is a way to accommodate both use cases (by introducing additional test parameters to force external Hive CLI usage that uses the mock Hive utils that we have in the unit test framework) and still get HCatalog run as part of the unit tests.

          Thanks
          Venkat

          Show
          Venkat Ranganathan added a comment - THanks Jarek Jarcec Cecho for your review and comments. I initially had the database separately, but the hcatalog team thought it made more sense to have them together (as they process tables with that format). But adding a database column should not be difficult if we need to make it more compliant. It is true that schema inference is a great feature. I thought of adding it in a follow on JIRA with some additional constructs such that we still give the user storage type independence if they so desire. For example if they want all their tables to be whatever they choose (and supported by HCatalog) as the default and pre-create tables if specific output file type is desired. I will create a SUB task and get it in this task itself. The main issue with not supporting hive-drop-delims is that string columns with embedded delimiter chars and using delimited text format will have the fidelity issues that the current users have. I considered that but was not sure if it was worth the extra processing for all output types. I wanted to sqoop code to be agnostic of the storage format (so not worry about querying the metadata on the storage info), and users still have the option of using the current hive import to deal with that case if it is so desired which is well understood. Direct option does not deal with sqoop record type, so we have to come up with a HCat implementation for each connection manager based on its input/output row formats after parsing it. For example, in the case of Netezza direct mode, Sqoop ORM scheme is not involved so we don't even generate the jar files. I think the existing Hive import mechanism can be used where applicable (not sure if it works with all connection managers, but since the output is text format, the existing hive import support should help with that). As you know, HCatalog uses hive metastore, so such tables are also available to HCatalog users. Regarding running as part of the normal test suite, I totally understand. I also did not want it to be a manual test. If you look at the test utils, I use a mini cluster like MR job to do the loading into HCatalog and reading off HCatalog. HCatalog does not have a HCatalogMiniCluster (for unit testing). When I first tried to run everything in local mode (which is supported), then the hive tests failed (because we depend on lack of some classes to distinguish between hive external cli or in process invocation). That is why I had to exclude some of the Hive classes from the dependencies to make it run all unit tests. Let me see if there is a way to accommodate both use cases (by introducing additional test parameters to force external Hive CLI usage that uses the mock Hive utils that we have in the unit test framework) and still get HCatalog run as part of the unit tests. Thanks Venkat
          Hide
          Venkat Ranganathan added a comment -

          I have attached a new document with additional details on automatic table creation and expalining the delimiter support. In future, it should be possible to combine the Hive/HCatalog support into a single implementation.

          Show
          Venkat Ranganathan added a comment - I have attached a new document with additional details on automatic table creation and expalining the delimiter support. In future, it should be possible to combine the Hive/HCatalog support into a single implementation.
          Hide
          Venkat Ranganathan added a comment - - edited

          Hi Jarek Jarcec Cecho

          I think supporting hive-drop-delims or hive-delims-replacement is easy for HCatalog (after going through the code) and it will be an issue only when we support direct mode. So, I agree that we can support hive-drop-delims but we will describe that this is only needed for text format. We will add a warning that if hive-drop-delims or --hive-delims-replacment is set that the data fiedlity will be lost. Sorry for the confusion

          Thanks

          Venkat

          Show
          Venkat Ranganathan added a comment - - edited Hi Jarek Jarcec Cecho I think supporting hive-drop-delims or hive-delims-replacement is easy for HCatalog (after going through the code) and it will be an issue only when we support direct mode. So, I agree that we can support hive-drop-delims but we will describe that this is only needed for text format. We will add a warning that if hive-drop-delims or --hive-delims-replacment is set that the data fiedlity will be lost. Sorry for the confusion Thanks Venkat
          Hide
          Venkat Ranganathan added a comment -

          Added a new file with changes based on review comments

          Show
          Venkat Ranganathan added a comment - Added a new file with changes based on review comments
          Hide
          Venkat Ranganathan added a comment - - edited

          I have updated the patch to review board based on review comments and also updated the document and attached to this JIRA

          All unit tests pass - no new checkstyle violations introduced by this patch.

          Show
          Venkat Ranganathan added a comment - - edited I have updated the patch to review board based on review comments and also updated the document and attached to this JIRA All unit tests pass - no new checkstyle violations introduced by this patch.
          Hide
          Jarek Jarcec Cecho added a comment -

          Venkat Ranganathan, thank you very much for incorporating all my comments into the design document! I think that the document is great and I'll move to reviewing the code itself.

          Show
          Jarek Jarcec Cecho added a comment - Venkat Ranganathan , thank you very much for incorporating all my comments into the design document! I think that the document is great and I'll move to reviewing the code itself.
          Hide
          Venkat Ranganathan added a comment -

          Thanks Jarek. I tried to incorporate the changes. I am updating the patch with additional fixes (that was found when testing with different DB products and versions) and also fixed some trailing blanks in the ivy settings.

          Show
          Venkat Ranganathan added a comment - Thanks Jarek. I tried to incorporate the changes. I am updating the patch with additional fixes (that was found when testing with different DB products and versions) and also fixed some trailing blanks in the ivy settings.
          Hide
          Venkat Ranganathan added a comment -

          Latest patch that addresses all review comments uploaded. Thanks Jarek

          Show
          Venkat Ranganathan added a comment - Latest patch that addresses all review comments uploaded. Thanks Jarek
          Hide
          ASF subversion and git services added a comment -

          Commit 5e88d43b5af024c1b9fd82029f7de4c325dcf009 in branch refs/heads/trunk from Jarek Jarcec Cecho
          [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=5e88d43 ]

          SQOOP-931: Integrate HCatalog with Sqoop

          (Venkat Ranganathan via Jarek Jarcec Cecho)

          Show
          ASF subversion and git services added a comment - Commit 5e88d43b5af024c1b9fd82029f7de4c325dcf009 in branch refs/heads/trunk from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=5e88d43 ] SQOOP-931 : Integrate HCatalog with Sqoop (Venkat Ranganathan via Jarek Jarcec Cecho)
          Hide
          Jarek Jarcec Cecho added a comment -

          Thank you Venkat for all your effort on this feature!

          Show
          Jarek Jarcec Cecho added a comment - Thank you Venkat for all your effort on this feature!
          Hide
          Venkat Ranganathan added a comment -

          Thanks Jarek for reviewing this patch and committing it.

          Venkat

          Show
          Venkat Ranganathan added a comment - Thanks Jarek for reviewing this patch and committing it. Venkat
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop200 #655 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/655/)
          SQOOP-931: Integrate HCatalog with Sqoop (Revision 5e88d43b5af024c1b9fd82029f7de4c325dcf009)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=5e88d43b5af024c1b9fd82029f7de4c325dcf009
          Files :

          • ivy/ivysettings.xml
          • src/java/org/apache/sqoop/mapreduce/ExportJobBase.java
          • src/java/org/apache/sqoop/tool/ExportTool.java
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportMapper.java
          • src/test/com/cloudera/sqoop/hive/TestHiveImport.java
          • src/java/org/apache/sqoop/SqoopOptions.java
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportFormat.java
          • src/docs/user/SqoopUserGuide.txt
          • src/test/org/apache/sqoop/hcat/HCatalogTestUtils.java
          • src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
          • src/java/org/apache/sqoop/manager/ConnManager.java
          • bin/configure-sqoop
          • src/test/org/apache/sqoop/hcat/HCatalogExportTest.java
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatRecordReader.java
          • src/java/org/apache/sqoop/tool/ImportTool.java
          • src/test/org/apache/sqoop/hcat/HCatalogImportTest.java
          • src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java
          • src/test/org/apache/sqoop/hcat/TestHCatalogBasic.java
          • src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java
          • src/perftest/ExportStressTest.java
          • src/java/org/apache/sqoop/hive/HiveImport.java
          • src/java/org/apache/sqoop/mapreduce/JobBase.java
          • src/java/org/apache/sqoop/config/ConfigurationConstants.java
          • ivy.xml
          • testdata/hcatalog/conf/log4j.properties
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportMapper.java
          • src/docs/user/hcatalog.txt
          • src/test/com/cloudera/sqoop/ThirdPartyTests.java
          • build.xml
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatInputSplit.java
          • src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          • src/java/org/apache/sqoop/tool/CodeGenTool.java
          • bin/configure-sqoop.cmd
          • src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java
          • testdata/hcatalog/conf/hive-site.xml
          • testdata/hcatalog/conf/hive-log4j.properties
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop200 #655 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/655/ ) SQOOP-931 : Integrate HCatalog with Sqoop (Revision 5e88d43b5af024c1b9fd82029f7de4c325dcf009) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=5e88d43b5af024c1b9fd82029f7de4c325dcf009 Files : ivy/ivysettings.xml src/java/org/apache/sqoop/mapreduce/ExportJobBase.java src/java/org/apache/sqoop/tool/ExportTool.java src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportMapper.java src/test/com/cloudera/sqoop/hive/TestHiveImport.java src/java/org/apache/sqoop/SqoopOptions.java src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportFormat.java src/docs/user/SqoopUserGuide.txt src/test/org/apache/sqoop/hcat/HCatalogTestUtils.java src/java/org/apache/sqoop/mapreduce/ImportJobBase.java src/java/org/apache/sqoop/manager/ConnManager.java bin/configure-sqoop src/test/org/apache/sqoop/hcat/HCatalogExportTest.java src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatRecordReader.java src/java/org/apache/sqoop/tool/ImportTool.java src/test/org/apache/sqoop/hcat/HCatalogImportTest.java src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java src/test/org/apache/sqoop/hcat/TestHCatalogBasic.java src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java src/perftest/ExportStressTest.java src/java/org/apache/sqoop/hive/HiveImport.java src/java/org/apache/sqoop/mapreduce/JobBase.java src/java/org/apache/sqoop/config/ConfigurationConstants.java ivy.xml testdata/hcatalog/conf/log4j.properties src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportMapper.java src/docs/user/hcatalog.txt src/test/com/cloudera/sqoop/ThirdPartyTests.java build.xml src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatInputSplit.java src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/java/org/apache/sqoop/tool/CodeGenTool.java bin/configure-sqoop.cmd src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java testdata/hcatalog/conf/hive-site.xml testdata/hcatalog/conf/hive-log4j.properties src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop20 #651 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/651/)
          SQOOP-931: Integrate HCatalog with Sqoop (Revision 5e88d43b5af024c1b9fd82029f7de4c325dcf009)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=5e88d43b5af024c1b9fd82029f7de4c325dcf009
          Files :

          • src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportMapper.java
          • ivy/ivysettings.xml
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java
          • src/java/org/apache/sqoop/config/ConfigurationConstants.java
          • build.xml
          • src/perftest/ExportStressTest.java
          • ivy.xml
          • testdata/hcatalog/conf/hive-site.xml
          • src/test/org/apache/sqoop/hcat/HCatalogExportTest.java
          • src/java/org/apache/sqoop/hive/HiveImport.java
          • src/docs/user/hcatalog.txt
          • src/docs/user/SqoopUserGuide.txt
          • src/test/org/apache/sqoop/hcat/HCatalogImportTest.java
          • src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
          • src/java/org/apache/sqoop/tool/CodeGenTool.java
          • src/java/org/apache/sqoop/tool/ImportTool.java
          • src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java
          • src/java/org/apache/sqoop/manager/ConnManager.java
          • src/java/org/apache/sqoop/mapreduce/JobBase.java
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatInputSplit.java
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportFormat.java
          • testdata/hcatalog/conf/hive-log4j.properties
          • src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java
          • bin/configure-sqoop.cmd
          • src/test/com/cloudera/sqoop/ThirdPartyTests.java
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportMapper.java
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          • src/java/org/apache/sqoop/mapreduce/ExportJobBase.java
          • src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java
          • testdata/hcatalog/conf/log4j.properties
          • src/java/org/apache/sqoop/SqoopOptions.java
          • bin/configure-sqoop
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatRecordReader.java
          • src/java/org/apache/sqoop/tool/ExportTool.java
          • src/test/org/apache/sqoop/hcat/TestHCatalogBasic.java
          • src/test/com/cloudera/sqoop/hive/TestHiveImport.java
          • src/test/org/apache/sqoop/hcat/HCatalogTestUtils.java
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop20 #651 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/651/ ) SQOOP-931 : Integrate HCatalog with Sqoop (Revision 5e88d43b5af024c1b9fd82029f7de4c325dcf009) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=5e88d43b5af024c1b9fd82029f7de4c325dcf009 Files : src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportMapper.java ivy/ivysettings.xml src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java src/java/org/apache/sqoop/config/ConfigurationConstants.java build.xml src/perftest/ExportStressTest.java ivy.xml testdata/hcatalog/conf/hive-site.xml src/test/org/apache/sqoop/hcat/HCatalogExportTest.java src/java/org/apache/sqoop/hive/HiveImport.java src/docs/user/hcatalog.txt src/docs/user/SqoopUserGuide.txt src/test/org/apache/sqoop/hcat/HCatalogImportTest.java src/java/org/apache/sqoop/mapreduce/ImportJobBase.java src/java/org/apache/sqoop/tool/CodeGenTool.java src/java/org/apache/sqoop/tool/ImportTool.java src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java src/java/org/apache/sqoop/manager/ConnManager.java src/java/org/apache/sqoop/mapreduce/JobBase.java src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatInputSplit.java src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportFormat.java testdata/hcatalog/conf/hive-log4j.properties src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java bin/configure-sqoop.cmd src/test/com/cloudera/sqoop/ThirdPartyTests.java src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportMapper.java src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/java/org/apache/sqoop/mapreduce/ExportJobBase.java src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java testdata/hcatalog/conf/log4j.properties src/java/org/apache/sqoop/SqoopOptions.java bin/configure-sqoop src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatRecordReader.java src/java/org/apache/sqoop/tool/ExportTool.java src/test/org/apache/sqoop/hcat/TestHCatalogBasic.java src/test/com/cloudera/sqoop/hive/TestHiveImport.java src/test/org/apache/sqoop/hcat/HCatalogTestUtils.java
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop100 #654 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/654/)
          SQOOP-931: Integrate HCatalog with Sqoop (Revision 5e88d43b5af024c1b9fd82029f7de4c325dcf009)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=5e88d43b5af024c1b9fd82029f7de4c325dcf009
          Files :

          • src/test/com/cloudera/sqoop/hive/TestHiveImport.java
          • src/docs/user/hcatalog.txt
          • bin/configure-sqoop.cmd
          • ivy/ivysettings.xml
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatRecordReader.java
          • testdata/hcatalog/conf/log4j.properties
          • src/test/org/apache/sqoop/hcat/HCatalogTestUtils.java
          • src/test/org/apache/sqoop/hcat/HCatalogImportTest.java
          • src/test/com/cloudera/sqoop/ThirdPartyTests.java
          • testdata/hcatalog/conf/hive-site.xml
          • src/java/org/apache/sqoop/hive/HiveImport.java
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          • testdata/hcatalog/conf/hive-log4j.properties
          • build.xml
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatInputSplit.java
          • src/java/org/apache/sqoop/tool/CodeGenTool.java
          • src/test/org/apache/sqoop/hcat/HCatalogExportTest.java
          • src/java/org/apache/sqoop/mapreduce/ExportJobBase.java
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportMapper.java
          • src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java
          • src/test/org/apache/sqoop/hcat/TestHCatalogBasic.java
          • src/java/org/apache/sqoop/SqoopOptions.java
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportFormat.java
          • src/java/org/apache/sqoop/config/ConfigurationConstants.java
          • src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java
          • ivy.xml
          • src/java/org/apache/sqoop/tool/ExportTool.java
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java
          • src/perftest/ExportStressTest.java
          • src/docs/user/SqoopUserGuide.txt
          • src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
          • src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java
          • bin/configure-sqoop
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportMapper.java
          • src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java
          • src/java/org/apache/sqoop/mapreduce/JobBase.java
          • src/java/org/apache/sqoop/tool/ImportTool.java
          • src/java/org/apache/sqoop/manager/ConnManager.java
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop100 #654 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/654/ ) SQOOP-931 : Integrate HCatalog with Sqoop (Revision 5e88d43b5af024c1b9fd82029f7de4c325dcf009) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=5e88d43b5af024c1b9fd82029f7de4c325dcf009 Files : src/test/com/cloudera/sqoop/hive/TestHiveImport.java src/docs/user/hcatalog.txt bin/configure-sqoop.cmd ivy/ivysettings.xml src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatRecordReader.java testdata/hcatalog/conf/log4j.properties src/test/org/apache/sqoop/hcat/HCatalogTestUtils.java src/test/org/apache/sqoop/hcat/HCatalogImportTest.java src/test/com/cloudera/sqoop/ThirdPartyTests.java testdata/hcatalog/conf/hive-site.xml src/java/org/apache/sqoop/hive/HiveImport.java src/java/org/apache/sqoop/tool/BaseSqoopTool.java testdata/hcatalog/conf/hive-log4j.properties build.xml src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatInputSplit.java src/java/org/apache/sqoop/tool/CodeGenTool.java src/test/org/apache/sqoop/hcat/HCatalogExportTest.java src/java/org/apache/sqoop/mapreduce/ExportJobBase.java src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportMapper.java src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java src/test/org/apache/sqoop/hcat/TestHCatalogBasic.java src/java/org/apache/sqoop/SqoopOptions.java src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportFormat.java src/java/org/apache/sqoop/config/ConfigurationConstants.java src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java ivy.xml src/java/org/apache/sqoop/tool/ExportTool.java src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java src/perftest/ExportStressTest.java src/docs/user/SqoopUserGuide.txt src/java/org/apache/sqoop/mapreduce/ImportJobBase.java src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java bin/configure-sqoop src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportMapper.java src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java src/java/org/apache/sqoop/mapreduce/JobBase.java src/java/org/apache/sqoop/tool/ImportTool.java src/java/org/apache/sqoop/manager/ConnManager.java
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop23 #847 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/847/)
          SQOOP-931: Integrate HCatalog with Sqoop (Revision 5e88d43b5af024c1b9fd82029f7de4c325dcf009)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=5e88d43b5af024c1b9fd82029f7de4c325dcf009
          Files :

          • src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java
          • bin/configure-sqoop
          • src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java
          • src/test/com/cloudera/sqoop/ThirdPartyTests.java
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportFormat.java
          • src/java/org/apache/sqoop/SqoopOptions.java
          • src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java
          • testdata/hcatalog/conf/hive-log4j.properties
          • src/test/org/apache/sqoop/hcat/HCatalogExportTest.java
          • src/docs/user/hcatalog.txt
          • src/java/org/apache/sqoop/hive/HiveImport.java
          • build.xml
          • src/java/org/apache/sqoop/tool/ExportTool.java
          • testdata/hcatalog/conf/log4j.properties
          • src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java
          • src/docs/user/SqoopUserGuide.txt
          • bin/configure-sqoop.cmd
          • src/java/org/apache/sqoop/mapreduce/JobBase.java
          • src/perftest/ExportStressTest.java
          • src/java/org/apache/sqoop/mapreduce/ExportJobBase.java
          • src/test/com/cloudera/sqoop/hive/TestHiveImport.java
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatInputSplit.java
          • src/test/org/apache/sqoop/hcat/HCatalogTestUtils.java
          • src/java/org/apache/sqoop/config/ConfigurationConstants.java
          • ivy/ivysettings.xml
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportMapper.java
          • src/java/org/apache/sqoop/tool/CodeGenTool.java
          • src/test/org/apache/sqoop/hcat/HCatalogImportTest.java
          • src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportMapper.java
          • src/test/org/apache/sqoop/hcat/TestHCatalogBasic.java
          • src/java/org/apache/sqoop/manager/ConnManager.java
          • testdata/hcatalog/conf/hive-site.xml
          • ivy.xml
          • src/java/org/apache/sqoop/tool/ImportTool.java
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatRecordReader.java
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop23 #847 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/847/ ) SQOOP-931 : Integrate HCatalog with Sqoop (Revision 5e88d43b5af024c1b9fd82029f7de4c325dcf009) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=5e88d43b5af024c1b9fd82029f7de4c325dcf009 Files : src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java bin/configure-sqoop src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java src/test/com/cloudera/sqoop/ThirdPartyTests.java src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportFormat.java src/java/org/apache/sqoop/SqoopOptions.java src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java testdata/hcatalog/conf/hive-log4j.properties src/test/org/apache/sqoop/hcat/HCatalogExportTest.java src/docs/user/hcatalog.txt src/java/org/apache/sqoop/hive/HiveImport.java build.xml src/java/org/apache/sqoop/tool/ExportTool.java testdata/hcatalog/conf/log4j.properties src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java src/docs/user/SqoopUserGuide.txt bin/configure-sqoop.cmd src/java/org/apache/sqoop/mapreduce/JobBase.java src/perftest/ExportStressTest.java src/java/org/apache/sqoop/mapreduce/ExportJobBase.java src/test/com/cloudera/sqoop/hive/TestHiveImport.java src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatInputSplit.java src/test/org/apache/sqoop/hcat/HCatalogTestUtils.java src/java/org/apache/sqoop/config/ConfigurationConstants.java ivy/ivysettings.xml src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportMapper.java src/java/org/apache/sqoop/tool/CodeGenTool.java src/test/org/apache/sqoop/hcat/HCatalogImportTest.java src/java/org/apache/sqoop/mapreduce/ImportJobBase.java src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportMapper.java src/test/org/apache/sqoop/hcat/TestHCatalogBasic.java src/java/org/apache/sqoop/manager/ConnManager.java testdata/hcatalog/conf/hive-site.xml ivy.xml src/java/org/apache/sqoop/tool/ImportTool.java src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatRecordReader.java src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          Hide
          Sonya Ling added a comment -

          I tried sqoop-1.4.4 for this feature today. I have Hive 0.11.0 that has hcatalog merged into it. I have Hadoop 2.0.0-cdh4.3.0. I run statement similar to the followings:

          sqoop import --connect jdbc:mysql://<host>/<database> --username <user> --password <password> --table transaction --hcatalog-table transaction --create-hcatalog-table --where "date >= 08-01-2012" --hive-partition-key date

          It went through and started hadoop job but halt due the following error:
          Exception in thread "main" java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
          at org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.getJobInfo(HCatBaseOutputFormat.java:94)
          at org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.getOutputFormat(HCatBaseOutputFormat.java:82)
          at org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.checkOutputSpecs(HCatBaseOutputFormat.java:72)
          at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:417)

          I know this is due to Hadoop version conflict issue. HCatBaseOutputFormat is expecting Hadoop 1.0.x but I have Hadoop 2.0.x. I saw similar error when I ran Ozzie. I could get around by add mapred.mapper.new-api and mapred.reducer.new-api to true in workflow.xml of ozzie. I added the same properties to proto-hive-site.xml in hcatalog/etc. It did not work. I checked hcatalog 0.50 source codes. It does not use those properties.

          How can I get around this issue? Please advise. Thanks.

          Show
          Sonya Ling added a comment - I tried sqoop-1.4.4 for this feature today. I have Hive 0.11.0 that has hcatalog merged into it. I have Hadoop 2.0.0-cdh4.3.0. I run statement similar to the followings: sqoop import --connect jdbc:mysql://<host>/<database> --username <user> --password <password> --table transaction --hcatalog-table transaction --create-hcatalog-table --where "date >= 08-01-2012" --hive-partition-key date It went through and started hadoop job but halt due the following error: Exception in thread "main" java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected at org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.getJobInfo(HCatBaseOutputFormat.java:94) at org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.getOutputFormat(HCatBaseOutputFormat.java:82) at org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.checkOutputSpecs(HCatBaseOutputFormat.java:72) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:417) I know this is due to Hadoop version conflict issue. HCatBaseOutputFormat is expecting Hadoop 1.0.x but I have Hadoop 2.0.x. I saw similar error when I ran Ozzie. I could get around by add mapred.mapper.new-api and mapred.reducer.new-api to true in workflow.xml of ozzie. I added the same properties to proto-hive-site.xml in hcatalog/etc. It did not work. I checked hcatalog 0.50 source codes. It does not use those properties. How can I get around this issue? Please advise. Thanks.
          Hide
          Venkat Ranganathan added a comment -

          You need HCatalog Hadoop 2.0 artifacts. We have a JIRA issue open in Hive for fixing this. Until it is resolved, one option is that you can build HCatalog yourself.

          I know that HDP 2.0 Community Preview(1) has HCatalog 0.11 built with Hadoop 2.0 and there could be similar options with other distributions as well.

          Thanks

          Venkat
          (1) http://hortonworks.com/products/hdp/hdp2/

          Show
          Venkat Ranganathan added a comment - You need HCatalog Hadoop 2.0 artifacts. We have a JIRA issue open in Hive for fixing this. Until it is resolved, one option is that you can build HCatalog yourself. I know that HDP 2.0 Community Preview(1) has HCatalog 0.11 built with Hadoop 2.0 and there could be similar options with other distributions as well. Thanks Venkat (1) http://hortonworks.com/products/hdp/hdp2/
          Hide
          Sonya Ling added a comment -

          I did set up maven Hadoop200 profile with all Hadoop 2.0 artifacts. With some twist and turn, I did get mvn package successful for all hcatalog modules. Then I duplicate hcatalog folder into hcatalog-200 folder under hive and copy all jars from my build to that hcatalog-200 folder. Change HCAT_HOME to hcatalog-200.

          I re-run the same sqoop command. This time I got the following error right away.
          I got 13/08/14 19:19:07 INFO hcat.SqoopHCatUtilities: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
          13/08/14 19:19:08 INFO hcat.SqoopHCatUtilities: FAILED: SemanticException Operation not supported.
          13/08/14 19:19:08 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: HCat exited with status 64
          at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.executeExternalHCatProgram(SqoopHCatUtilities.java:931)
          at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.launchHCatCli(SqoopHCatUtilities.java:884)
          at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.createHCatTable(SqoopHCatUtilities.java:521)
          at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.configureHCat(SqoopHCatUtilities.java:294)
          at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.configureImportOutputFormat(SqoopHCatUtilities.java:658)

          Let me know if I missed something. THANKS a lot.

          Show
          Sonya Ling added a comment - I did set up maven Hadoop200 profile with all Hadoop 2.0 artifacts. With some twist and turn, I did get mvn package successful for all hcatalog modules. Then I duplicate hcatalog folder into hcatalog-200 folder under hive and copy all jars from my build to that hcatalog-200 folder. Change HCAT_HOME to hcatalog-200. I re-run the same sqoop command. This time I got the following error right away. I got 13/08/14 19:19:07 INFO hcat.SqoopHCatUtilities: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 13/08/14 19:19:08 INFO hcat.SqoopHCatUtilities: FAILED: SemanticException Operation not supported. 13/08/14 19:19:08 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: HCat exited with status 64 at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.executeExternalHCatProgram(SqoopHCatUtilities.java:931) at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.launchHCatCli(SqoopHCatUtilities.java:884) at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.createHCatTable(SqoopHCatUtilities.java:521) at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.configureHCat(SqoopHCatUtilities.java:294) at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.configureImportOutputFormat(SqoopHCatUtilities.java:658) Let me know if I missed something. THANKS a lot.
          Hide
          Venkat Ranganathan added a comment -

          Moving this to dev@sqoop.apache.org as this is on the usage of this

          Show
          Venkat Ranganathan added a comment - Moving this to dev@sqoop.apache.org as this is on the usage of this
          Hide
          Sonya Ling added a comment -

          I got Hadoop 2.0.0-cdh4.3.0 work with sqoop-1.4.4 (HCatlog integration). It populates records with dynamic partitions beautifully.

          Building hcatalog with 2.0 artifact yourself is not a good idea because you might get other error like'Caused by: java.lang.ClassNotFoundException: org.apache.hcatalog.shims.HCatHadoopShims23' error (not to say you need to manipulate maven to get maven build successfully). Instead, you should get hcatalog-cdh4 (the same version as your Hadoop, Hive etc.). That ensures everything is compatible.

          You need to manually create your partitioned table either in hcat or hive beforehand since --create-hcatalog-table won't create partitioned table for you. Then, execute sqoop script like the following example:

          sqoop import --connect jdbc:mysql://<host>/<database> --username <user> --password <password> --table <sql-table> --where <where clause> --split-by <split-field> --hcatalog-database <hcat-database> --hcatalog-table <hcat-table>

          The important thing is NOT TO put -hive-partition-key. That's for static partition, stated clearly in document. Dynamic partions would work like charm.

          Thanks for all the helps.
          Cheers.

          Show
          Sonya Ling added a comment - I got Hadoop 2.0.0-cdh4.3.0 work with sqoop-1.4.4 (HCatlog integration). It populates records with dynamic partitions beautifully. Building hcatalog with 2.0 artifact yourself is not a good idea because you might get other error like'Caused by: java.lang.ClassNotFoundException: org.apache.hcatalog.shims.HCatHadoopShims23' error (not to say you need to manipulate maven to get maven build successfully). Instead, you should get hcatalog-cdh4 (the same version as your Hadoop, Hive etc.). That ensures everything is compatible. You need to manually create your partitioned table either in hcat or hive beforehand since --create-hcatalog-table won't create partitioned table for you. Then, execute sqoop script like the following example: sqoop import --connect jdbc:mysql://<host>/<database> --username <user> --password <password> --table <sql-table> --where <where clause> --split-by <split-field> --hcatalog-database <hcat-database> --hcatalog-table <hcat-table> The important thing is NOT TO put -hive-partition-key. That's for static partition, stated clearly in document. Dynamic partions would work like charm. Thanks for all the helps. Cheers.
          Hide
          Venkat Ranganathan added a comment -

          Thanks for the update Sonya Ling. It is awesome that the solution worked for you beautifully. The documentation should describe the functionality

          Unfortunately, I don't know about vendor specific distributions (unless of course HDP versions as I work for Hortonworks )

          Thanks

          Venkat

          Show
          Venkat Ranganathan added a comment - Thanks for the update Sonya Ling . It is awesome that the solution worked for you beautifully. The documentation should describe the functionality Unfortunately, I don't know about vendor specific distributions (unless of course HDP versions as I work for Hortonworks ) Thanks Venkat
          Hide
          Venkat Ranganathan added a comment -

          Also just to clarify create-hcatalog-table will only create partitioned table if there was a static partiion key provided. Also, there is a storage stanza that can be provided to provide storage format information.

          Show
          Venkat Ranganathan added a comment - Also just to clarify create-hcatalog-table will only create partitioned table if there was a static partiion key provided. Also, there is a storage stanza that can be provided to provide storage format information.

            People

            • Assignee:
              Venkat Ranganathan
              Reporter:
              Venkat Ranganathan
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development