Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
Not 100% sure that this is a defect but, it seems like one to me.
The Sqoop job help suggests that generic args are supported however I am not able to get them to work, see test case below
Sqoop Help
$ sqoop job -help Output: 17/01/17 08:13:12 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.8.3 usage: sqoop job [GENERIC-ARGS] [JOB-ARGS] [-- [<tool-name>] [TOOL-ARGS]]
Test Case
#################
# STEP 01 - Setup Table and Data
#################
export MYCONN=jdbc:mysql://mysql.cloudera.com:3306/db_coe
export MYUSER=sqoop
export MYPSWD=cloudera
sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "drop table t1"
sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "create table t1 (c1 int, c2 varchar(10))"
sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "insert into t1 values (1, 'some data')"
sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "select * from t1"
----------------------------
| c1 | c2 |
----------------------------
| 1 | some data |
----------------------------
#################
# STEP 02 - Import Data Baseline (default)
#################
sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table t1 --target-dir /user/root/t1 --delete-target-dir --num-mappers 1
yarn application -status application_1481911879790_0093 | head -6
17/01/02 07:03:46 INFO client.RMProxy: Connecting to ResourceManager at host1.cloudera.com/NN.NN.NNN.NNN:8032
Application Report :
Application-Id : application_1481911879790_0093
Application-Name : t1.jar
Application-Type : MAPREDUCE
User : root
Queue : root.users.root
#################
# STEP 03 - Import Data (-Dmapreduce.job.queuename=MY_SQOOP_QUEUE -Dmapreduce.job.name=MY_SQOOP_JOB)
#################
sqoop import -Dmapreduce.job.queuename=MY_SQOOP_QUEUE -Dmapreduce.job.name=MY_SQOOP_JOB --connect $MYCONN --username $MYUSER --password $MYPSWD --table t1 --target-dir /user/root/t1 --delete-target-dir --num-mappers 1
yarn application -status application_1481911879790_0094 | head -6
17/01/02 07:07:00 INFO client.RMProxy: Connecting to ResourceManager at host1.cloudera.com/NN.NN.NNN.NNN:8032
Application Report :
Application-Id : application_1481911879790_0094
Application-Name : MY_SQOOP_JOB
Application-Type : MAPREDUCE
User : root
Queue : root.MY_SQOOP_QUEUE
#################
# STEP 04 - Import Data using Sqoop1 Metastore Job (default)
#################
sqoop job --create my_sqoop_meta_job -- import --connect $MYCONN --username $MYUSER --password $MYPSWD --table t1 --target-dir /user/root/t1 --delete-target-dir --num-mappers 1
sqoop job --list
Available jobs:
my_sqoop_meta_job
sqoop job --exec my_sqoop_meta_job
yarn application -status application_1481911879790_0095 | head -6
17/01/02 07:16:13 INFO client.RMProxy: Connecting to ResourceManager at host1.cloudera.com/NN.NN.NNN.NNN:8032
Application Report :
Application-Id : application_1481911879790_0095
Application-Name : t1.jar
Application-Type : MAPREDUCE
User : root
Queue : root.users.root
#################
# STEP 06 - Review Sqoop Job Help
#################
$ sqoop job -help
Output:
17/01/17 08:13:12 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.8.3
usage: sqoop job [GENERIC-ARGS] [JOB-ARGS] [-- [<tool-name>] [TOOL-ARGS]]
Job management arguments:
--create <job-id> Create a new saved job
--delete <job-id> Delete a saved job
--exec <job-id> Run a saved job
--help Print usage instructions
--list List saved jobs
--meta-connect <jdbc-uri> Specify JDBC connect string for the
metastore
--show <job-id> Show the parameters for a saved job
--verbose Print more information while working
Generic Hadoop command-line arguments:
(must preceed any tool-specific arguments)
Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
#################
# STEP 07 - Import Data using Sqoop1 Metastore Job (-Dmapreduce.job.queuename=MY_SQOOP_QUEUE -Dmapreduce.job.name=MY_SQOOP_JOB)
#################
sqoop job -Dmapreduce.job.queuename=MY_SQOOP_META_QUEUE -Dmapreduce.job.name=MY_SQOOP_META_JOB --exec my_sqoop_meta_job
yarn application -status application_1481911879790_0096 | head -6
17/01/02 07:19:39 INFO client.RMProxy: Connecting to ResourceManager at host1.cloudera.com/NN.NN.NNN.NNN:8032
Application Report :
Application-Id : application_1481911879790_0096
Application-Name : t1.jar
Application-Type : MAPREDUCE
User : root
Queue : root.users.root