Sqoop
  1. Sqoop
  2. SQOOP-846

Provide a connector for Netezza appliances

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.2
    • Fix Version/s: 1.4.4
    • Component/s: None
    • Labels:
    • Environment:

      Netezza environments

      Description

      Netezza Datawarehouse appliance is a well known data warehouse appliance product and it would be good to have a connector for Netezza in Sqoop

      1. Netezza Connector Doc.pdf
        155 kB
        Venkat Ranganathan
      2. SQOOP-846.patch
        89 kB
        Venkat Ranganathan
      3. SQOOP-846.patch.1
        96 kB
        Venkat Ranganathan
      4. SQOOP-846.patch.3
        88 kB
        Venkat Ranganathan
      5. SQOOP-846.patch.6
        92 kB
        Venkat Ranganathan
      6. SQOOP-846.patch.7
        92 kB
        Venkat Ranganathan
      7. SQOOP-846.patch.9
        91 kB
        Venkat Ranganathan

        Issue Links

          Activity

          Hide
          Louis Mau added a comment -

          This comment is prompted by one of the comment in SQOOP-931 i.e. to be able to use HCat support with Direct mode. The current NetezzaExternalTableImportMapper implementation iterates through the external table pipe inside the map() method. All the external table support are implemented in private methods. Is it possible to move these private methods into NetezzaExternalTableRecordReader and relies on the Mapper to do the iteration? This will make it easier to reuse the external table implementation in other context.

          Show
          Louis Mau added a comment - This comment is prompted by one of the comment in SQOOP-931 i.e. to be able to use HCat support with Direct mode. The current NetezzaExternalTableImportMapper implementation iterates through the external table pipe inside the map() method. All the external table support are implemented in private methods. Is it possible to move these private methods into NetezzaExternalTableRecordReader and relies on the Mapper to do the iteration? This will make it easier to reuse the external table implementation in other context.
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop100 #590 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/590/)
          SQOOP-846: Provide a connector for Netezza appliances (Revision 0d5f73ad8dccd4630e4762361513b1583b2b1a2f)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=0d5f73ad8dccd4630e4762361513b1583b2b1a2f
          Files :

          • src/java/org/apache/sqoop/manager/MySQLUtils.java
          • src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableImportJob.java
          • src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaDBDataSliceSplitter.java
          • src/test/com/cloudera/sqoop/manager/NetezzaTestUtils.java
          • src/test/com/cloudera/sqoop/manager/NetezzaImportManualTest.java
          • src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableExportJob.java
          • src/java/org/apache/sqoop/lib/DelimiterSet.java
          • src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaJDBCStatementRunner.java
          • src/java/org/apache/sqoop/manager/DirectNetezzaManager.java
          • src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableImportMapper.java
          • src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputSplit.java
          • src/test/com/cloudera/sqoop/manager/DirectNetezzaExportManualTest.java
          • src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputFormat.java
          • src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableTextExportMapper.java
          • src/java/org/apache/sqoop/manager/NetezzaManager.java
          • src/docs/user/connectors.txt
          • src/java/org/apache/sqoop/mapreduce/netezza/NetezzaDataDrivenDBInputFormat.java
          • src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableRecordExportMapper.java
          • src/java/org/apache/sqoop/manager/DefaultManagerFactory.java
          • src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableExportMapper.java
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop100 #590 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/590/ ) SQOOP-846 : Provide a connector for Netezza appliances (Revision 0d5f73ad8dccd4630e4762361513b1583b2b1a2f) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=0d5f73ad8dccd4630e4762361513b1583b2b1a2f Files : src/java/org/apache/sqoop/manager/MySQLUtils.java src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableImportJob.java src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaDBDataSliceSplitter.java src/test/com/cloudera/sqoop/manager/NetezzaTestUtils.java src/test/com/cloudera/sqoop/manager/NetezzaImportManualTest.java src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableExportJob.java src/java/org/apache/sqoop/lib/DelimiterSet.java src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaJDBCStatementRunner.java src/java/org/apache/sqoop/manager/DirectNetezzaManager.java src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableImportMapper.java src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputSplit.java src/test/com/cloudera/sqoop/manager/DirectNetezzaExportManualTest.java src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputFormat.java src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableTextExportMapper.java src/java/org/apache/sqoop/manager/NetezzaManager.java src/docs/user/connectors.txt src/java/org/apache/sqoop/mapreduce/netezza/NetezzaDataDrivenDBInputFormat.java src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableRecordExportMapper.java src/java/org/apache/sqoop/manager/DefaultManagerFactory.java src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableExportMapper.java
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop23 #790 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/790/)
          SQOOP-846: Provide a connector for Netezza appliances (Revision 0d5f73ad8dccd4630e4762361513b1583b2b1a2f)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=0d5f73ad8dccd4630e4762361513b1583b2b1a2f
          Files :

          • src/test/com/cloudera/sqoop/manager/DirectNetezzaExportManualTest.java
          • src/java/org/apache/sqoop/manager/NetezzaManager.java
          • src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputSplit.java
          • src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableExportJob.java
          • src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaJDBCStatementRunner.java
          • src/test/com/cloudera/sqoop/manager/NetezzaTestUtils.java
          • src/java/org/apache/sqoop/manager/DefaultManagerFactory.java
          • src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableRecordExportMapper.java
          • src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableExportMapper.java
          • src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputFormat.java
          • src/java/org/apache/sqoop/mapreduce/netezza/NetezzaDataDrivenDBInputFormat.java
          • src/test/com/cloudera/sqoop/manager/NetezzaImportManualTest.java
          • src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableImportMapper.java
          • src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableImportJob.java
          • src/docs/user/connectors.txt
          • src/java/org/apache/sqoop/manager/DirectNetezzaManager.java
          • src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableTextExportMapper.java
          • src/java/org/apache/sqoop/lib/DelimiterSet.java
          • src/java/org/apache/sqoop/manager/MySQLUtils.java
          • src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaDBDataSliceSplitter.java
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop23 #790 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/790/ ) SQOOP-846 : Provide a connector for Netezza appliances (Revision 0d5f73ad8dccd4630e4762361513b1583b2b1a2f) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=0d5f73ad8dccd4630e4762361513b1583b2b1a2f Files : src/test/com/cloudera/sqoop/manager/DirectNetezzaExportManualTest.java src/java/org/apache/sqoop/manager/NetezzaManager.java src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputSplit.java src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableExportJob.java src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaJDBCStatementRunner.java src/test/com/cloudera/sqoop/manager/NetezzaTestUtils.java src/java/org/apache/sqoop/manager/DefaultManagerFactory.java src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableRecordExportMapper.java src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableExportMapper.java src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputFormat.java src/java/org/apache/sqoop/mapreduce/netezza/NetezzaDataDrivenDBInputFormat.java src/test/com/cloudera/sqoop/manager/NetezzaImportManualTest.java src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableImportMapper.java src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableImportJob.java src/docs/user/connectors.txt src/java/org/apache/sqoop/manager/DirectNetezzaManager.java src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableTextExportMapper.java src/java/org/apache/sqoop/lib/DelimiterSet.java src/java/org/apache/sqoop/manager/MySQLUtils.java src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaDBDataSliceSplitter.java
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop200 #576 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/576/)
          SQOOP-846: Provide a connector for Netezza appliances (Revision 0d5f73ad8dccd4630e4762361513b1583b2b1a2f)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=0d5f73ad8dccd4630e4762361513b1583b2b1a2f
          Files :

          • src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableExportMapper.java
          • src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableExportJob.java
          • src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableImportJob.java
          • src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaJDBCStatementRunner.java
          • src/test/com/cloudera/sqoop/manager/NetezzaImportManualTest.java
          • src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableTextExportMapper.java
          • src/docs/user/connectors.txt
          • src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaDBDataSliceSplitter.java
          • src/java/org/apache/sqoop/manager/MySQLUtils.java
          • src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableImportMapper.java
          • src/test/com/cloudera/sqoop/manager/DirectNetezzaExportManualTest.java
          • src/java/org/apache/sqoop/manager/NetezzaManager.java
          • src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableRecordExportMapper.java
          • src/test/com/cloudera/sqoop/manager/NetezzaTestUtils.java
          • src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputFormat.java
          • src/java/org/apache/sqoop/manager/DefaultManagerFactory.java
          • src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputSplit.java
          • src/java/org/apache/sqoop/lib/DelimiterSet.java
          • src/java/org/apache/sqoop/mapreduce/netezza/NetezzaDataDrivenDBInputFormat.java
          • src/java/org/apache/sqoop/manager/DirectNetezzaManager.java
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop200 #576 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/576/ ) SQOOP-846 : Provide a connector for Netezza appliances (Revision 0d5f73ad8dccd4630e4762361513b1583b2b1a2f) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=0d5f73ad8dccd4630e4762361513b1583b2b1a2f Files : src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableExportMapper.java src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableExportJob.java src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableImportJob.java src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaJDBCStatementRunner.java src/test/com/cloudera/sqoop/manager/NetezzaImportManualTest.java src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableTextExportMapper.java src/docs/user/connectors.txt src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaDBDataSliceSplitter.java src/java/org/apache/sqoop/manager/MySQLUtils.java src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableImportMapper.java src/test/com/cloudera/sqoop/manager/DirectNetezzaExportManualTest.java src/java/org/apache/sqoop/manager/NetezzaManager.java src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableRecordExportMapper.java src/test/com/cloudera/sqoop/manager/NetezzaTestUtils.java src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputFormat.java src/java/org/apache/sqoop/manager/DefaultManagerFactory.java src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputSplit.java src/java/org/apache/sqoop/lib/DelimiterSet.java src/java/org/apache/sqoop/mapreduce/netezza/NetezzaDataDrivenDBInputFormat.java src/java/org/apache/sqoop/manager/DirectNetezzaManager.java
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop20 #579 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/579/)
          SQOOP-846: Provide a connector for Netezza appliances (Revision 0d5f73ad8dccd4630e4762361513b1583b2b1a2f)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=0d5f73ad8dccd4630e4762361513b1583b2b1a2f
          Files :

          • src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableImportJob.java
          • src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaDBDataSliceSplitter.java
          • src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableExportMapper.java
          • src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableTextExportMapper.java
          • src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaJDBCStatementRunner.java
          • src/java/org/apache/sqoop/manager/NetezzaManager.java
          • src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputSplit.java
          • src/java/org/apache/sqoop/lib/DelimiterSet.java
          • src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableExportJob.java
          • src/docs/user/connectors.txt
          • src/java/org/apache/sqoop/mapreduce/netezza/NetezzaDataDrivenDBInputFormat.java
          • src/java/org/apache/sqoop/manager/MySQLUtils.java
          • src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableRecordExportMapper.java
          • src/test/com/cloudera/sqoop/manager/NetezzaImportManualTest.java
          • src/java/org/apache/sqoop/manager/DefaultManagerFactory.java
          • src/test/com/cloudera/sqoop/manager/DirectNetezzaExportManualTest.java
          • src/test/com/cloudera/sqoop/manager/NetezzaTestUtils.java
          • src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableImportMapper.java
          • src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputFormat.java
          • src/java/org/apache/sqoop/manager/DirectNetezzaManager.java
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop20 #579 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/579/ ) SQOOP-846 : Provide a connector for Netezza appliances (Revision 0d5f73ad8dccd4630e4762361513b1583b2b1a2f) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=0d5f73ad8dccd4630e4762361513b1583b2b1a2f Files : src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableImportJob.java src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaDBDataSliceSplitter.java src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableExportMapper.java src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableTextExportMapper.java src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaJDBCStatementRunner.java src/java/org/apache/sqoop/manager/NetezzaManager.java src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputSplit.java src/java/org/apache/sqoop/lib/DelimiterSet.java src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableExportJob.java src/docs/user/connectors.txt src/java/org/apache/sqoop/mapreduce/netezza/NetezzaDataDrivenDBInputFormat.java src/java/org/apache/sqoop/manager/MySQLUtils.java src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableRecordExportMapper.java src/test/com/cloudera/sqoop/manager/NetezzaImportManualTest.java src/java/org/apache/sqoop/manager/DefaultManagerFactory.java src/test/com/cloudera/sqoop/manager/DirectNetezzaExportManualTest.java src/test/com/cloudera/sqoop/manager/NetezzaTestUtils.java src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableImportMapper.java src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputFormat.java src/java/org/apache/sqoop/manager/DirectNetezzaManager.java
          Hide
          Venkat Ranganathan added a comment -

          Thanks a lot Jarek. Time to work on the follow on JIRAS

          Show
          Venkat Ranganathan added a comment - Thanks a lot Jarek. Time to work on the follow on JIRAS
          Jarek Jarcec Cecho made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Jarek Jarcec Cecho added a comment -
          Show
          Jarek Jarcec Cecho added a comment - The patch is in: https://git-wip-us.apache.org/repos/asf?p=sqoop.git;a=commit;h=0d5f73ad8dccd4630e4762361513b1583b2b1a2f Thank you Venkat for your hard work! Jarcec
          Jarek Jarcec Cecho made changes -
          Fix Version/s 1.4.4 [ 12324082 ]
          Fix Version/s 1.4.3 [ 12322644 ]
          Venkat Ranganathan made changes -
          Link This issue blocks SQOOP-929 [ SQOOP-929 ]
          Hide
          Venkat Ranganathan added a comment -

          Latest patch - with doc changes and additional changes from review

          Show
          Venkat Ranganathan added a comment - Latest patch - with doc changes and additional changes from review
          Venkat Ranganathan made changes -
          Attachment SQOOP-846.patch.9 [ 12571786 ]
          Venkat Ranganathan made changes -
          Attachment SQOOP-846.patch.7 [ 12571373 ]
          Hide
          Venkat Ranganathan added a comment -

          Latest patch with DirectMySQLTest removed from the patch

          Show
          Venkat Ranganathan added a comment - Latest patch with DirectMySQLTest removed from the patch
          Venkat Ranganathan made changes -
          Attachment SQOOP-846.patch.6 [ 12571336 ]
          Hide
          Venkat Ranganathan added a comment -

          Latest patch

          Show
          Venkat Ranganathan added a comment - Latest patch
          Hide
          Venkat Ranganathan added a comment -

          Added a new document with specifics on why updateTable is limited to one mapper by adding a link to the enzee community discussion on this

          Show
          Venkat Ranganathan added a comment - Added a new document with specifics on why updateTable is limited to one mapper by adding a link to the enzee community discussion on this
          Venkat Ranganathan made changes -
          Attachment Netezza Connector Doc.pdf [ 12570435 ]
          Venkat Ranganathan made changes -
          Attachment Netezza Connector Doc.pdf [ 12571273 ]
          Venkat Ranganathan made changes -
          Attachment SQOOP-846.patch.3 [ 12570858 ]
          Hide
          Venkat Ranganathan added a comment -

          Adding latest patch

          Show
          Venkat Ranganathan added a comment - Adding latest patch
          Hide
          Venkat Ranganathan added a comment -

          I see that I introduced some whitespace issues in the previous patch. Let me handle them also and create a new patch

          Show
          Venkat Ranganathan added a comment - I see that I introduced some whitespace issues in the previous patch. Let me handle them also and create a new patch
          Hide
          Venkat Ranganathan added a comment -

          Thanks Jarcec

          I have updated the document and updated the patch here - will upload the new patch to the review board as well

          Thanks

          Show
          Venkat Ranganathan added a comment - Thanks Jarcec I have updated the document and updated the patch here - will upload the new patch to the review board as well Thanks
          Venkat Ranganathan made changes -
          Attachment SQOOP-846.patch.1 [ 12570436 ]
          Venkat Ranganathan made changes -
          Attachment Netezza Connector Doc.pdf [ 12570435 ]
          Venkat Ranganathan made changes -
          Attachment Netezza Connector Doc.pdf [ 12570202 ]
          Hide
          Jarek Jarcec Cecho added a comment -

          That sounds as a good plan, thank you for working on this!

          Show
          Jarek Jarcec Cecho added a comment - That sounds as a good plan, thank you for working on this!
          Hide
          Venkat Ranganathan added a comment -

          Thanks Jarcec. Good observation. I debated about this option initially, but used this as it was not clear how many of those options are really used in practice. I think I tried to pick the defaults for some of the options and let the useful ones be specified, (max error threshold and log dir). I can remove that option and we can add additional specific options based on usage

          Thanks
          Venkat

          Show
          Venkat Ranganathan added a comment - Thanks Jarcec. Good observation. I debated about this option initially, but used this as it was not clear how many of those options are really used in practice. I think I tried to pick the defaults for some of the options and let the useful ones be specified, (max error threshold and log dir). I can remove that option and we can add additional specific options based on usage Thanks Venkat
          Hide
          Jarek Jarcec Cecho added a comment -

          Hi Venkat,
          thank you very much for sharing the design doc. It's very well written document, thank you for doing that! I do have one comment.

          The connector will provide one extra argument option -additional­external-‐table-­options.

          I'm not sure that this a good idea. Some of the arguments might break the behavior and I'm not sure that users should have this kind of power. I would much more prefer to see each option that make sense exposed as an extra option that we can verify for correctness prior passing to the underlying create external table statement.

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - Hi Venkat, thank you very much for sharing the design doc. It's very well written document, thank you for doing that! I do have one comment. The connector will provide one extra argument option - additional ­external-‐table-­options. I'm not sure that this a good idea. Some of the arguments might break the behavior and I'm not sure that users should have this kind of power. I would much more prefer to see each option that make sense exposed as an extra option that we can verify for correctness prior passing to the underlying create external table statement. Jarcec
          Venkat Ranganathan made changes -
          Remote Link This issue links to "Review board link (Web Link)" [ 12036 ]
          Hide
          Venkat Ranganathan added a comment -

          Thanks Jarcec. I created a review board request. Will add the link to this JIRA

          Show
          Venkat Ranganathan added a comment - Thanks Jarcec. I created a review board request. Will add the link to this JIRA
          Hide
          Jarek Jarcec Cecho added a comment -

          Hi Venkat,
          thank you very much for your effort, I'll definitely take a look. Considering that it's quite big patch, would you mind uploading it to the review board?

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - Hi Venkat, thank you very much for your effort, I'll definitely take a look. Considering that it's quite big patch, would you mind uploading it to the review board ? Jarcec
          Hide
          Venkat Ranganathan added a comment -

          The connector was tested against Netezza VMs 6 and 7

          Show
          Venkat Ranganathan added a comment - The connector was tested against Netezza VMs 6 and 7
          Venkat Ranganathan made changes -
          Link This issue is related to SQOOP-849 [ SQOOP-849 ]
          Venkat Ranganathan made changes -
          Link This issue is related to SQOOP-848 [ SQOOP-848 ]
          Venkat Ranganathan made changes -
          Link This issue is related to SQOOP-847 [ SQOOP-847 ]
          Venkat Ranganathan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Labels features
          Fix Version/s 1.4.3 [ 12322644 ]
          Hide
          Venkat Ranganathan added a comment -

          I have updated the patches for this

          Show
          Venkat Ranganathan added a comment - I have updated the patches for this
          Hide
          Venkat Ranganathan added a comment -

          Sorry for taking too long. I have updated the connector design approaches document and patch against trunk.

          Show
          Venkat Ranganathan added a comment - Sorry for taking too long. I have updated the connector design approaches document and patch against trunk.
          Venkat Ranganathan made changes -
          Field Original Value New Value
          Attachment Netezza Connector Doc.pdf [ 12570202 ]
          Attachment SQOOP-846.patch [ 12570203 ]
          Hide
          Jarek Jarcec Cecho added a comment -

          Hi Venkat,
          I believe that there is already closed source Netezza connector provided by Cloudera [1].

          Jarcec

          Links:
          1: https://ccp.cloudera.com/display/con/Cloudera+Connector+for+Netezza

          Show
          Jarek Jarcec Cecho added a comment - Hi Venkat, I believe that there is already closed source Netezza connector provided by Cloudera [1] . Jarcec Links: 1: https://ccp.cloudera.com/display/con/Cloudera+Connector+for+Netezza
          Hide
          Venkat Ranganathan added a comment -

          At a high level the connector can support both normal mode (export/import/update) and an optional direct mode to perform fast bulk transfers using Netezza specific facilities

          Show
          Venkat Ranganathan added a comment - At a high level the connector can support both normal mode (export/import/update) and an optional direct mode to perform fast bulk transfers using Netezza specific facilities
          Venkat Ranganathan created issue -

            People

            • Assignee:
              Venkat Ranganathan
              Reporter:
              Venkat Ranganathan
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 336h
                336h
                Remaining:
                Remaining Estimate - 336h
                336h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development