Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-3934

sqoop import param '--null-string' result in null value become blank string in hive table

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: v2.6.0
    • Fix Version/s: v2.6.2
    • Component/s: Others
    • Labels:
      None

      Description

      when column value from jdbc is null, sqoop will write it into hive table with blank string.

      eg 

      jdbc:

      A | B

      1 | 1

      2 | 2

      a | null

       

      hive table:

      A | B

      1 | 1

      2 | 2

      a |

      because of this, when I use count(distinct B), it return 3, not 2, and it can lead to other problems

       

       

      String cmd = String.format(Locale.ROOT,
      "%s/bin/sqoop import" + generateSqoopConfigArgString()
      + "--connect \"%s\" --driver %s --username %s --password %s --query \"%s AND \\$CONDITIONS\" "
      + "--target-dir %s/%s --split-by %s --boundary-query \"%s\" --null-string '' "
      + "--fields-terminated-by '%s' --num-mappers %d",
      sqoopHome, connectionUrl, driverClass, jdbcUser, jdbcPass, selectSql, jobWorkingDir, hiveTable,
      splitColumn, bquery, filedDelimiter, mapperNum);
      

      the param '–null=string' should be '
      N' instead of blank string ''.

      I resolved this problem by replace the param. But it needs be configured in kylin.properties

       

       

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              wanghao312 wanghao
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: