[SQOOP-3134] --class-name should override default Avro schema name

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.0, 3.0.0
• Assignee: Daniel Voros
• Reporter: Markus Kemper
• Created: 13/Feb/17 20:28
• Updated: 05/Apr/19 14:45
• Resolved: 05/Apr/19 14:16

Description

Please consider adding an option to configure the Avro schema output file name that is created with Sqoop (import + --as-avrodatafile), example cases below.

#################
# STEP 01 - Create Data
#################

export MYCONN=jdbc:mysql://mysql.cloudera.com:3306/db_coe
export MYUSER=sqoop
export MYPSWD=cloudera

sqoop list-tables --connect $MYCONN --username $MYUSER --password $MYPSWD
sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "drop table t1"
sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "create table t1 (c1 int, c2 date, c3 varchar(10))"
sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "insert into t1 values (1, current_date, 'some data')"
sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "select * from t1"

-----------------------------------------
| c1          | c2         | c3         | 
-----------------------------------------
| 1           | 2017-02-13 | some data  | 
-----------------------------------------

#################
# STEP 02 - Import + --table + --as-avrodatafile
#################

sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table t1 --target-dir /user/root/t1 --delete-target-dir --num-mappers 1 --as-avrodatafile 
ls -l ./*

Output:
17/02/13 12:14:52 INFO mapreduce.ImportJobBase: Transferred 413 bytes in 20.6988 seconds (19.9529 bytes/sec)
17/02/13 12:14:52 INFO mapreduce.ImportJobBase: Retrieved 1 records.
~~~~
-rw-r--r-- 1 root root   492 Feb 13 12:14 ./t1.avsc <==== want option to configure this file name
-rw-r--r-- 1 root root 12462 Feb 13 12:14 ./t1.java

#################
# STEP 03 - Import + --query + --as-avrodatafile
#################

sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --query "select * from t1 where \$CONDITIONS" --split-by c1 --target-dir /user/root/t1 --delete-target-dir --num-mappers 1 --as-avrodatafile 
ls -l ./*

Output:
17/02/13 12:16:58 INFO mapreduce.ImportJobBase: Transferred 448 bytes in 25.2757 seconds (17.7245 bytes/sec)
17/02/13 12:16:58 INFO mapreduce.ImportJobBase: Retrieved 1 records.
~~~~~
-rw-r--r-- 1 root root   527 Feb 13 12:16 ./AutoGeneratedSchema.avsc <==== want option to configure this file name
-rw-r--r-- 1 root root 12590 Feb 13 12:16 ./QueryResult.java

Issue Links

• is related to: SQOOP-2783 Query import with parquet fails on incompatible schema
• links to: GitHub Pull Request #78