[KYLIN-3934] sqoop import param '--null-string' result in null value become blank string in hive table

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: v2.6.0
• Fix Version/s: v2.6.2
• Component/s: Others
• Assignee: Unassigned
• Reporter: wanghao
• Created: 04/Apr/19 10:57
• Updated: 18/May/19 02:00
• Resolved: 13/May/19 07:47

Description

when column value from jdbc is null, sqoop will write it into hive table with blank string.

eg 

jdbc:

A | B

1 | 1

2 | 2

a | null

 

hive table:

A | B

1 | 1

2 | 2

a |

because of this, when I use count(distinct B), it return 3, not 2, and it can lead to other problems

 

 

String cmd = String.format(Locale.ROOT,
"%s/bin/sqoop import" + generateSqoopConfigArgString()
+ "--connect \"%s\" --driver %s --username %s --password %s --query \"%s AND \\$CONDITIONS\" "
+ "--target-dir %s/%s --split-by %s --boundary-query \"%s\" --null-string '' "
+ "--fields-terminated-by '%s' --num-mappers %d",
sqoopHome, connectionUrl, driverClass, jdbcUser, jdbcPass, selectSql, jobWorkingDir, hiveTable,
splitColumn, bquery, filedDelimiter, mapperNum);

the param '–null=string' should be '
N' instead of blank string ''.

I resolved this problem by replace the param. But it needs be configured in kylin.properties

 

 

 

Issue Links

• links to: GitHub Pull Request #587
• links to: GitHub Pull Request #635