Sqoop
  1. Sqoop
  2. SQOOP-330

Free form query import with column transformation failed without obvious error message

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.0-incubating
    • Fix Version/s: 1.4.0-incubating
    • Component/s: tools
    • Labels:
      None

      Description

      Calling upstream version with free form query that contains column with transformation will fail on MS SQL server (I did not try other RDBMS):

      [root@hddev01 sqoop]# sqoop import --driver com.microsoft.sqlserver.jdbc.SQLServerDriver --connect jdbc:sqlserver://host\;databaseName=db --username jarcec --password password --as-textfile --num-mappers 10 --target-dir /user/root/table SELECT convert(varchar(50), md5, 2) FROM table where $CONDITIONS' --split-by id
      11/09/04 13:16:41 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
      11/09/04 13:16:41 INFO manager.SqlManager: Using default fetchSize of 1000
      11/09/04 13:16:41 INFO tool.CodeGenTool: Beginning code generation
      11/09/04 13:16:43 INFO manager.SqlManager: Executing SQL statement: SELECT convert(varchar(50), md5, 2) FROM table where (1 = 0)
      11/09/04 13:16:43 INFO manager.SqlManager: Executing SQL statement: SELECT convert(varchar(50), md5, 2) FROM table where (1 = 0)
      11/09/04 13:16:43 INFO orm.CompilationManager: HADOOP_HOME is /usr/lib/hadoop
      11/09/04 13:16:43 INFO orm.CompilationManager: Found hadoop core jar at: /usr/lib/hadoop/hadoop-core.jar
      /tmp/sqoop-root/compile/462603a17b4956b3bd9151dab43f4227/QueryResult.java:37: <identifier> expected
      private String ;
      ^
      /tmp/sqoop-root/compile/462603a17b4956b3bd9151dab43f4227/QueryResult.java:41: <identifier> expected
      public void set_(String ) {
      ^
      /tmp/sqoop-root/compile/462603a17b4956b3bd9151dab43f4227/QueryResult.java:42: <identifier> expected
      this. = ;
      ^
      /tmp/sqoop-root/compile/462603a17b4956b3bd9151dab43f4227/QueryResult.java:42: illegal start of expression
      this. = ;
      ^
      /tmp/sqoop-root/compile/462603a17b4956b3bd9151dab43f4227/QueryResult.java:44: <identifier> expected
      public QueryResult with_(String ) {
      ^
      /tmp/sqoop-root/compile/462603a17b4956b3bd9151dab43f4227/QueryResult.java:45: <identifier> expected
      this. = ;
      ^
      /tmp/sqoop-root/compile/462603a17b4956b3bd9151dab43f4227/QueryResult.java:45: illegal start of expression
      this. = ;
      ^
      /tmp/sqoop-root/compile/462603a17b4956b3bd9151dab43f4227/QueryResult.java:57: <identifier> expected
      equal = equal && (this. == null ? that. == null : this..equals(that.));
      ^
      /tmp/sqoop-root/compile/462603a17b4956b3bd9151dab43f4227/QueryResult.java:57: <identifier> expected
      equal = equal && (this. == null ? that. == null : this..equals(that.));
      ^
      /tmp/sqoop-root/compile/462603a17b4956b3bd9151dab43f4227/QueryResult.java:57: malformed floating point literal
      equal = equal && (this. == null ? that. == null : this..equals(that.));
      ^
      /tmp/sqoop-root/compile/462603a17b4956b3bd9151dab43f4227/QueryResult.java:57: <identifier> expected
      equal = equal && (this. == null ? that. == null : this..equals(that.));
      ^
      /tmp/sqoop-root/compile/462603a17b4956b3bd9151dab43f4227/QueryResult.java:57: ';' expected
      equal = equal && (this. == null ? that. == null : this..equals(that.));
      ^
      /tmp/sqoop-root/compile/462603a17b4956b3bd9151dab43f4227/QueryResult.java:62: <identifier> expected
      this. = JdbcWritableBridge.readString(1, __dbResults);
      ^
      /tmp/sqoop-root/compile/462603a17b4956b3bd9151dab43f4227/QueryResult.java:72: illegal start of expression
      JdbcWritableBridge.writeString(, 1 + __off, 12, __dbStmt);
      ^
      /tmp/sqoop-root/compile/462603a17b4956b3bd9151dab43f4227/QueryResult.java:77: <identifier> expected
      this. = null;
      ^
      /tmp/sqoop-root/compile/462603a17b4956b3bd9151dab43f4227/QueryResult.java:79: <identifier> expected
      this. = Text.readString(__dataIn);
      ^
      /tmp/sqoop-root/compile/462603a17b4956b3bd9151dab43f4227/QueryResult.java:83: <identifier> expected
      if (null == this.) {
      ^
      /tmp/sqoop-root/compile/462603a17b4956b3bd9151dab43f4227/QueryResult.java:87: illegal start of expression
      Text.writeString(__dataOut, );
      ^
      /tmp/sqoop-root/compile/462603a17b4956b3bd9151dab43f4227/QueryResult.java:103: illegal start of expression
      __sb.append(FieldFormatter.escapeAndEnclose(==null?"null":, delimiters));
      ^
      /tmp/sqoop-root/compile/462603a17b4956b3bd9151dab43f4227/QueryResult.java:103: illegal start of expression
      __sb.append(FieldFormatter.escapeAndEnclose(==null?"null":, delimiters));
      ^
      /tmp/sqoop-root/compile/462603a17b4956b3bd9151dab43f4227/QueryResult.java:163: <identifier> expected
      if (__cur_str.equals("null"))

      { this. = null; }

      else {
      ^
      /tmp/sqoop-root/compile/462603a17b4956b3bd9151dab43f4227/QueryResult.java:164: <identifier> expected
      this. = __cur_str;
      ^
      /tmp/sqoop-root/compile/462603a17b4956b3bd9151dab43f4227/QueryResult.java:176: <identifier> expected
      __sqoop$field_map.put("", this.);
      ^
      /tmp/sqoop-root/compile/462603a17b4956b3bd9151dab43f4227/QueryResult.java:182: <identifier> expected
      this. = (String) __fieldVal;
      ^
      24 errors
      11/09/04 13:16:43 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: Error returned by javac
      at com.cloudera.sqoop.orm.CompilationManager.compile(CompilationManager.java:205)
      at com.cloudera.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:83)
      at com.cloudera.sqoop.tool.ImportTool.importTable(ImportTool.java:367)
      at com.cloudera.sqoop.tool.ImportTool.run(ImportTool.java:453)
      at com.cloudera.sqoop.Sqoop.run(Sqoop.java:144)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:180)
      at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:219)
      at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:228)
      at com.cloudera.sqoop.Sqoop.main(Sqoop.java:237)

      1. SQOOP-330.patch
        1 kB
        Jarek Jarcec Cecho
      2. SQOOP-330.patch
        1 kB
        Jarek Jarcec Cecho

        Activity

        Hide
        Jarek Jarcec Cecho added a comment -

        Problem is that both getColumnName and getColumnLabel methods from class ResultSetMetaData are returning empty string and not null value.

        I tried to patch class SqlManager, method getColumnNamesForRawQuery to detect empty string as column name and create universal name in format RESULT

        {int}. Doing that ended on not clear error message about not existing column RESULT{int}

        .

        This patch is adding check for empty column name and will rise an exception if it will be detected.

        Show
        Jarek Jarcec Cecho added a comment - Problem is that both getColumnName and getColumnLabel methods from class ResultSetMetaData are returning empty string and not null value. I tried to patch class SqlManager, method getColumnNamesForRawQuery to detect empty string as column name and create universal name in format RESULT {int}. Doing that ended on not clear error message about not existing column RESULT {int} . This patch is adding check for empty column name and will rise an exception if it will be detected.
        Hide
        Arvind Prabhakar added a comment -

        Thanks for taking this up Jarek. The changes look good. <Nit> Could you please remove the following checkstyle violations from your change and update the patch:

        File src/java/com/cloudera/sqoop/orm/ClassWriter.java
        Error Description                                       Line
        Unused import - com.cloudera.sqoop.manager.SqlManager.  24
        Line is longer than 80 characters.                      1048
        Line is longer than 80 characters.                      1050
        Line matches the illegal pattern 'Trailing whitespace'. 1053
        

        You can run checkstyle yourself via ant checkstyle command.

        Show
        Arvind Prabhakar added a comment - Thanks for taking this up Jarek. The changes look good. <Nit> Could you please remove the following checkstyle violations from your change and update the patch: File src/java/com/cloudera/sqoop/orm/ClassWriter.java Error Description Line Unused import - com.cloudera.sqoop.manager.SqlManager. 24 Line is longer than 80 characters. 1048 Line is longer than 80 characters. 1050 Line matches the illegal pattern 'Trailing whitespace'. 1053 You can run checkstyle yourself via ant checkstyle command.
        Hide
        Jarek Jarcec Cecho added a comment -

        I've fixed the code style violations.

        Show
        Jarek Jarcec Cecho added a comment - I've fixed the code style violations.
        Hide
        Arvind Prabhakar added a comment -

        Patch committed. Thanks Jarek!

        Show
        Arvind Prabhakar added a comment - Patch committed. Thanks Jarek!
        Hide
        Hudson added a comment -

        Integrated in Sqoop-jdk-1.6 #20 (See https://builds.apache.org/job/Sqoop-jdk-1.6/20/)
        SQOOP-330. Descriptive error message when column name cannot be determined.

        (Jarek Jarcec Cecho via Arvind Prabhakar)

        arvind : http://svn.apache.org/viewvc/?view=rev&rev=1165969
        Files :

        • /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/orm/ClassWriter.java
        Show
        Hudson added a comment - Integrated in Sqoop-jdk-1.6 #20 (See https://builds.apache.org/job/Sqoop-jdk-1.6/20/ ) SQOOP-330 . Descriptive error message when column name cannot be determined. (Jarek Jarcec Cecho via Arvind Prabhakar) arvind : http://svn.apache.org/viewvc/?view=rev&rev=1165969 Files : /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/orm/ClassWriter.java

          People

          • Assignee:
            Jarek Jarcec Cecho
            Reporter:
            Jarek Jarcec Cecho
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development