Sqoop
  1. Sqoop
  2. SQOOP-1130

Sqoop2: Sqoop MapReduce Job Failed when max and min boundary are both null

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.99.2
    • Fix Version/s: 1.99.3
    • Component/s: connectors/generic
    • Labels:
      None
    • Environment:

      Mysql import data to HDFS

      Description

      It happens that all the values in the partition column are null. In this case, when Sqoop is partitioning data, it will find the max and min boundary are both null. Sqoop handles this case, but there is a bug.

      Currently, the Map task throw the following exception in this case:

      org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs during extractor run
      	at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:94)
      	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
      	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:396)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
      	at org.apache.hadoop.mapred.Child.main(Child.java:262)
      Caused by: org.apache.sqoop.common.SqoopException: GENERIC_JDBC_CONNECTOR_0002:Unable to execute the SQL statement
      	at org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
      	at org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:51)
      	at org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:31)
      	at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:89)
      	... 7 more
      Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'NULL' at line 1
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      	at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
      	at com.mysql.jdbc.Util.getInstance(Util.java:386)
      	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1054)
      	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4187)
      	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4119)
      	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2570)
      	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2731)
      	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2809)
      	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2758)
      	at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1612)
      	at org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:56)
      	... 10 more
      
      1. SQOOP-1130.1.patch
        0.9 kB
        Mengwei Ding

        Issue Links

          Activity

          Mengwei Ding created issue -
          Mengwei Ding made changes -
          Field Original Value New Value
          Description It happens that all the values in the partition column are null. In this case, when Sqoop is partitioning data, it will find the max and min boundary are both null. Sqoop handles this case, but there is a bug.

          Currently, the Map task throw the following exception in this case:

          org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs during extractor run
          at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:94)
          at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
          at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
          at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:396)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
          at org.apache.hadoop.mapred.Child.main(Child.java:262)
          Caused by: org.apache.sqoop.common.SqoopException: GENERIC_JDBC_CONNECTOR_0002:Unable to execute the SQL statement
          at org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
          at org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:51)
          at org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:31)
          at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:89)
          ... 7 more
          Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'NULL' at line 1
          at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
          at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
          at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
          at com.mysql.jdbc.Util.getInstance(Util.java:386)
          at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1054)
          at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4187)
          at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4119)
          at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2570)
          at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2731)
          at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2809)
          at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2758)
          at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1612)
          at org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:56)
          ... 10 more
          It happens that all the values in the partition column are null. In this case, when Sqoop is partitioning data, it will find the max and min boundary are both null. Sqoop handles this case, but there is a bug.

          Currently, the Map task throw the following exception in this case:

          {code}
          org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs during extractor run
          at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:94)
          at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
          at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
          at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:396)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
          at org.apache.hadoop.mapred.Child.main(Child.java:262)
          Caused by: org.apache.sqoop.common.SqoopException: GENERIC_JDBC_CONNECTOR_0002:Unable to execute the SQL statement
          at org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
          at org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:51)
          at org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:31)
          at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:89)
          ... 7 more
          Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'NULL' at line 1
          at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
          at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
          at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
          at com.mysql.jdbc.Util.getInstance(Util.java:386)
          at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1054)
          at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4187)
          at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4119)
          at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2570)
          at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2731)
          at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2809)
          at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2758)
          at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1612)
          at org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:56)
          ... 10 more
          {code}
          Mengwei Ding made changes -
          Attachment SQOOP-1130.1.patch [ 12591703 ]
          Hide
          Mengwei Ding added a comment -

          submit patch for review. It seems like the problem is a typo in SQL query.

          Show
          Mengwei Ding added a comment - submit patch for review. It seems like the problem is a typo in SQL query.
          Mengwei Ding made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Mengwei Ding made changes -
          Remote Link This issue links to "ReviewBoard (Web Link)" [ 12407 ]
          Hide
          ASF subversion and git services added a comment -

          Commit a3abffcd591f16a5f4f47fa43eeef7dcdec68cf7 in branch refs/heads/sqoop2 from Jarek Jarcec Cecho
          [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=a3abffc ]

          SQOOP-1130: Sqoop2: Sqoop MapReduce Job Failed when max and min boundary are both null

          (Mengwei Ding via Jarek Jarcec Cecho)

          Show
          ASF subversion and git services added a comment - Commit a3abffcd591f16a5f4f47fa43eeef7dcdec68cf7 in branch refs/heads/sqoop2 from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=a3abffc ] SQOOP-1130 : Sqoop2: Sqoop MapReduce Job Failed when max and min boundary are both null (Mengwei Ding via Jarek Jarcec Cecho)
          Hide
          Jarek Jarcec Cecho added a comment -

          Thank you for your contribution Mengwei Ding!

          Show
          Jarek Jarcec Cecho added a comment - Thank you for your contribution Mengwei Ding !
          Jarek Jarcec Cecho made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Sqoop2-hadoop200 #460 (See https://builds.apache.org/job/Sqoop2-hadoop200/460/)
          SQOOP-1130: Sqoop2: Sqoop MapReduce Job Failed when max and min boundary are both null (Revision a3abffcd591f16a5f4f47fa43eeef7dcdec68cf7)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a3abffcd591f16a5f4f47fa43eeef7dcdec68cf7
          Files :

          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartitioner.java
          Show
          Hudson added a comment - Integrated in Sqoop2-hadoop200 #460 (See https://builds.apache.org/job/Sqoop2-hadoop200/460/ ) SQOOP-1130 : Sqoop2: Sqoop MapReduce Job Failed when max and min boundary are both null (Revision a3abffcd591f16a5f4f47fa43eeef7dcdec68cf7) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a3abffcd591f16a5f4f47fa43eeef7dcdec68cf7 Files : connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartitioner.java
          Hide
          Hudson added a comment -

          Integrated in Sqoop2-hadoop100 #482 (See https://builds.apache.org/job/Sqoop2-hadoop100/482/)
          SQOOP-1130: Sqoop2: Sqoop MapReduce Job Failed when max and min boundary are both null (Revision a3abffcd591f16a5f4f47fa43eeef7dcdec68cf7)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a3abffcd591f16a5f4f47fa43eeef7dcdec68cf7
          Files :

          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartitioner.java
          Show
          Hudson added a comment - Integrated in Sqoop2-hadoop100 #482 (See https://builds.apache.org/job/Sqoop2-hadoop100/482/ ) SQOOP-1130 : Sqoop2: Sqoop MapReduce Job Failed when max and min boundary are both null (Revision a3abffcd591f16a5f4f47fa43eeef7dcdec68cf7) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a3abffcd591f16a5f4f47fa43eeef7dcdec68cf7 Files : connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartitioner.java
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          10m 49s 1 Mengwei Ding 10/Jul/13 19:48
          Patch Available Patch Available Resolved Resolved
          1d 3h 41m 1 Jarek Jarcec Cecho 11/Jul/13 23:30

            People

            • Assignee:
              Mengwei Ding
              Reporter:
              Mengwei Ding
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development