Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-937

Dont generate ORM files for Direct mode connectors

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.4
    • Component/s: None
    • Labels:
      None

      Description

      Not sure what the advantage of generating this and this confuses people

      1. SQOOP-937.patch
        14 kB
        Venkat Ranganathan

        Issue Links

          Activity

          Hide
          venkatnrangan Venkat Ranganathan added a comment -

          SQOOP generates an ORM file that represents a record in a table for a
          given connector implememntation. The generated class has methods to
          read field values off ResultSet, set bind values in PreparedStatements,
          handle LOB objects in a DB specific way (that the connector represents),
          parse input fields etc.

          This file is then compiled and archived using jar and used with the SQOOP job.

          This ORM instance is generated in all cases, and used to make sure that
          the data is read and processed in a more or less uniform way.

          Unfortunately, this generated class is not used by a class of connectors which
          manage the reading and processing of records themselves. In essence, the
          whole ORM class is unusable in these instances and is simply ignored. These
          are typically "direct" connectors which use a DB specific highspeed path.

          The generation of the ORM class in these cases causes confusion to users.

          This patch tries to solve this by

          1) Providing a capability for the connection managers to declare that
          they don't depend on the ORM jar file.
          2) Generating a dummy ORM jar file with explicit message during generation
          so that
          a) users are aware that the class generated is a dummy one
          b) there is a record that this jar file was generated explicitly
          c) we don't have to change a whole lot of the codebase to disable
          the generaration and loading of the jar file.

          This patch also adds one test.

          Show
          venkatnrangan Venkat Ranganathan added a comment - SQOOP generates an ORM file that represents a record in a table for a given connector implememntation. The generated class has methods to read field values off ResultSet, set bind values in PreparedStatements, handle LOB objects in a DB specific way (that the connector represents), parse input fields etc. This file is then compiled and archived using jar and used with the SQOOP job. This ORM instance is generated in all cases, and used to make sure that the data is read and processed in a more or less uniform way. Unfortunately, this generated class is not used by a class of connectors which manage the reading and processing of records themselves. In essence, the whole ORM class is unusable in these instances and is simply ignored. These are typically "direct" connectors which use a DB specific highspeed path. The generation of the ORM class in these cases causes confusion to users. This patch tries to solve this by 1) Providing a capability for the connection managers to declare that they don't depend on the ORM jar file. 2) Generating a dummy ORM jar file with explicit message during generation so that a) users are aware that the class generated is a dummy one b) there is a record that this jar file was generated explicitly c) we don't have to change a whole lot of the codebase to disable the generaration and loading of the jar file. This patch also adds one test.
          Hide
          venkatnrangan Venkat Ranganathan added a comment -

          Just to add, I have tested the DirectNetezzaManager after changing it to declare that is manages ORM and forcing the creation of the dummy class. After this is in, I can go through and update the connectors that can make use of this

          Show
          venkatnrangan Venkat Ranganathan added a comment - Just to add, I have tested the DirectNetezzaManager after changing it to declare that is manages ORM and forcing the creation of the dummy class. After this is in, I can go through and update the connectors that can make use of this
          Show
          jarcec Jarek Jarcec Cecho added a comment - The patch is in: https://git-wip-us.apache.org/repos/asf?p=sqoop.git;a=commit;h=f2ba7eaae25ac23c88651e1d14b9624aa5ded6ca Thank you Venkat! Jarcec
          Hide
          hudson Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop20 #596 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/596/)
          SQOOP-937: Dont generate ORM files for Direct mode connectors (Revision f2ba7eaae25ac23c88651e1d14b9624aa5ded6ca)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=f2ba7eaae25ac23c88651e1d14b9624aa5ded6ca
          Files :

          • src/java/org/apache/sqoop/manager/DirectNetezzaManager.java
          • src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
          • src/java/org/apache/sqoop/manager/ConnManager.java
          • src/java/org/apache/sqoop/tool/CodeGenTool.java
          • src/java/org/apache/sqoop/manager/ExportJobContext.java
          • src/java/org/apache/sqoop/mapreduce/ExportJobBase.java
          • src/test/com/cloudera/sqoop/orm/TestClassWriter.java
          • src/java/org/apache/sqoop/mapreduce/JobBase.java
          • src/java/org/apache/sqoop/orm/ClassWriter.java
          • src/java/org/apache/sqoop/manager/ImportJobContext.java
          Show
          hudson Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop20 #596 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/596/ ) SQOOP-937 : Dont generate ORM files for Direct mode connectors (Revision f2ba7eaae25ac23c88651e1d14b9624aa5ded6ca) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=f2ba7eaae25ac23c88651e1d14b9624aa5ded6ca Files : src/java/org/apache/sqoop/manager/DirectNetezzaManager.java src/java/org/apache/sqoop/mapreduce/ImportJobBase.java src/java/org/apache/sqoop/manager/ConnManager.java src/java/org/apache/sqoop/tool/CodeGenTool.java src/java/org/apache/sqoop/manager/ExportJobContext.java src/java/org/apache/sqoop/mapreduce/ExportJobBase.java src/test/com/cloudera/sqoop/orm/TestClassWriter.java src/java/org/apache/sqoop/mapreduce/JobBase.java src/java/org/apache/sqoop/orm/ClassWriter.java src/java/org/apache/sqoop/manager/ImportJobContext.java
          Hide
          hudson Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop100 #604 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/604/)
          SQOOP-937: Dont generate ORM files for Direct mode connectors (Revision f2ba7eaae25ac23c88651e1d14b9624aa5ded6ca)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=f2ba7eaae25ac23c88651e1d14b9624aa5ded6ca
          Files :

          • src/java/org/apache/sqoop/manager/ExportJobContext.java
          • src/java/org/apache/sqoop/manager/ImportJobContext.java
          • src/java/org/apache/sqoop/manager/DirectNetezzaManager.java
          • src/java/org/apache/sqoop/manager/ConnManager.java
          • src/java/org/apache/sqoop/mapreduce/JobBase.java
          • src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
          • src/java/org/apache/sqoop/mapreduce/ExportJobBase.java
          • src/java/org/apache/sqoop/tool/CodeGenTool.java
          • src/java/org/apache/sqoop/orm/ClassWriter.java
          • src/test/com/cloudera/sqoop/orm/TestClassWriter.java
          Show
          hudson Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop100 #604 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/604/ ) SQOOP-937 : Dont generate ORM files for Direct mode connectors (Revision f2ba7eaae25ac23c88651e1d14b9624aa5ded6ca) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=f2ba7eaae25ac23c88651e1d14b9624aa5ded6ca Files : src/java/org/apache/sqoop/manager/ExportJobContext.java src/java/org/apache/sqoop/manager/ImportJobContext.java src/java/org/apache/sqoop/manager/DirectNetezzaManager.java src/java/org/apache/sqoop/manager/ConnManager.java src/java/org/apache/sqoop/mapreduce/JobBase.java src/java/org/apache/sqoop/mapreduce/ImportJobBase.java src/java/org/apache/sqoop/mapreduce/ExportJobBase.java src/java/org/apache/sqoop/tool/CodeGenTool.java src/java/org/apache/sqoop/orm/ClassWriter.java src/test/com/cloudera/sqoop/orm/TestClassWriter.java
          Hide
          hudson Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop200 #589 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/589/)
          SQOOP-937: Dont generate ORM files for Direct mode connectors (Revision f2ba7eaae25ac23c88651e1d14b9624aa5ded6ca)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=f2ba7eaae25ac23c88651e1d14b9624aa5ded6ca
          Files :

          • src/java/org/apache/sqoop/mapreduce/ExportJobBase.java
          • src/java/org/apache/sqoop/orm/ClassWriter.java
          • src/java/org/apache/sqoop/mapreduce/JobBase.java
          • src/java/org/apache/sqoop/manager/ExportJobContext.java
          • src/test/com/cloudera/sqoop/orm/TestClassWriter.java
          • src/java/org/apache/sqoop/tool/CodeGenTool.java
          • src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
          • src/java/org/apache/sqoop/manager/ImportJobContext.java
          • src/java/org/apache/sqoop/manager/DirectNetezzaManager.java
          • src/java/org/apache/sqoop/manager/ConnManager.java
          Show
          hudson Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop200 #589 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/589/ ) SQOOP-937 : Dont generate ORM files for Direct mode connectors (Revision f2ba7eaae25ac23c88651e1d14b9624aa5ded6ca) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=f2ba7eaae25ac23c88651e1d14b9624aa5ded6ca Files : src/java/org/apache/sqoop/mapreduce/ExportJobBase.java src/java/org/apache/sqoop/orm/ClassWriter.java src/java/org/apache/sqoop/mapreduce/JobBase.java src/java/org/apache/sqoop/manager/ExportJobContext.java src/test/com/cloudera/sqoop/orm/TestClassWriter.java src/java/org/apache/sqoop/tool/CodeGenTool.java src/java/org/apache/sqoop/mapreduce/ImportJobBase.java src/java/org/apache/sqoop/manager/ImportJobContext.java src/java/org/apache/sqoop/manager/DirectNetezzaManager.java src/java/org/apache/sqoop/manager/ConnManager.java
          Hide
          hudson Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop23 #805 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/805/)
          SQOOP-937: Dont generate ORM files for Direct mode connectors (Revision f2ba7eaae25ac23c88651e1d14b9624aa5ded6ca)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=f2ba7eaae25ac23c88651e1d14b9624aa5ded6ca
          Files :

          • src/java/org/apache/sqoop/orm/ClassWriter.java
          • src/java/org/apache/sqoop/tool/CodeGenTool.java
          • src/java/org/apache/sqoop/manager/ExportJobContext.java
          • src/java/org/apache/sqoop/manager/ImportJobContext.java
          • src/test/com/cloudera/sqoop/orm/TestClassWriter.java
          • src/java/org/apache/sqoop/mapreduce/JobBase.java
          • src/java/org/apache/sqoop/manager/ConnManager.java
          • src/java/org/apache/sqoop/mapreduce/ExportJobBase.java
          • src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
          • src/java/org/apache/sqoop/manager/DirectNetezzaManager.java
          Show
          hudson Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop23 #805 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/805/ ) SQOOP-937 : Dont generate ORM files for Direct mode connectors (Revision f2ba7eaae25ac23c88651e1d14b9624aa5ded6ca) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=f2ba7eaae25ac23c88651e1d14b9624aa5ded6ca Files : src/java/org/apache/sqoop/orm/ClassWriter.java src/java/org/apache/sqoop/tool/CodeGenTool.java src/java/org/apache/sqoop/manager/ExportJobContext.java src/java/org/apache/sqoop/manager/ImportJobContext.java src/test/com/cloudera/sqoop/orm/TestClassWriter.java src/java/org/apache/sqoop/mapreduce/JobBase.java src/java/org/apache/sqoop/manager/ConnManager.java src/java/org/apache/sqoop/mapreduce/ExportJobBase.java src/java/org/apache/sqoop/mapreduce/ImportJobBase.java src/java/org/apache/sqoop/manager/DirectNetezzaManager.java

            People

            • Assignee:
              venkatnrangan Venkat Ranganathan
              Reporter:
              venkatnrangan Venkat Ranganathan
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development