Details
-
Bug
-
Status: Open
-
Critical
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
-
ubuntu, Sqoop 1.4.4-cdh5.1.2, java 8
Description
Sqoop command example:
sqoop import --connect "jdbc:mysql://127.0.0.1:3306/xxx" --username xxx --password xxx --table SampleTable --hcatalog-database xxx --create-hcatalog-table --hcatalog-storage-stanza 'stored as orc' --hcatalog-table sampletable
Call stack:
sqoop.Sqoop: Got exception running Sqoop: java.lang.NullPointerException java.lang.NullPointerException at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.createHCatTable(SqoopHCatUtilities.java:491) at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.configureHCat(SqoopHCatUtilities.java:293) at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.configureImportOutputFormat(SqoopHCatUtilities.java:657) at org.apache.sqoop.mapreduce.ImportJobBase.configureOutputFormat(ImportJobBase.java:98) at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:240) at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:614) at org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:118) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:413) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:506) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231) at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
This seems to be because source MySQL table contains a column name that has an upper case letter.
Sample table below. The column name, 'A', causes the exception.
DROP TABLE `SampleTable`; CREATE TABLE `SampleTable` ( `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT, `A` bigint(20) unsigned NOT NULL DEFAULT '0', PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Once column name is changed to 'a', it works properly:
DROP TABLE `SampleTable`; CREATE TABLE `SampleTable` ( `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT, `a` bigint(20) unsigned NOT NULL DEFAULT '0', PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
This is probably because LCKeyMap (and HCKeyMap) class in SqoopHCatUtilities.java line 160 - 177 are expecting HashMap to call put method for each element when putAll is invoked, which seem to have been the case in Java 6/7, but it doesn't behave that way in Java 8.
/** * A Map using String as key type that ignores case of its key and stores the * key in lower case. */ private static class LCKeyMap<V> extends HashMap<String, V> { private static final long serialVersionUID = -6751510232323094216L; @Override public V put(String key, V value) { return super.put(key.toLowerCase(), value); } @Override public V get(Object key) { return super.get(((String) key).toLowerCase()); } }