Uploaded image for project: 'Sqoop (Retired)'
  1. Sqoop (Retired)
  2. SQOOP-3134

--class-name should override default Avro schema name

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.5.0, 3.0.0
    • None
    • None

    Description

      Please consider adding an option to configure the Avro schema output file name that is created with Sqoop (import + --as-avrodatafile), example cases below.

      #################
      # STEP 01 - Create Data
      #################
      
      export MYCONN=jdbc:mysql://mysql.cloudera.com:3306/db_coe
      export MYUSER=sqoop
      export MYPSWD=cloudera
      
      sqoop list-tables --connect $MYCONN --username $MYUSER --password $MYPSWD
      sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "drop table t1"
      sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "create table t1 (c1 int, c2 date, c3 varchar(10))"
      sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "insert into t1 values (1, current_date, 'some data')"
      sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "select * from t1"
      
      -----------------------------------------
      | c1          | c2         | c3         | 
      -----------------------------------------
      | 1           | 2017-02-13 | some data  | 
      -----------------------------------------
      
      #################
      # STEP 02 - Import + --table + --as-avrodatafile
      #################
      
      sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table t1 --target-dir /user/root/t1 --delete-target-dir --num-mappers 1 --as-avrodatafile 
      ls -l ./*
      
      Output:
      17/02/13 12:14:52 INFO mapreduce.ImportJobBase: Transferred 413 bytes in 20.6988 seconds (19.9529 bytes/sec)
      17/02/13 12:14:52 INFO mapreduce.ImportJobBase: Retrieved 1 records.
      ~~~~
      -rw-r--r-- 1 root root   492 Feb 13 12:14 ./t1.avsc <==== want option to configure this file name
      -rw-r--r-- 1 root root 12462 Feb 13 12:14 ./t1.java
      
      #################
      # STEP 03 - Import + --query + --as-avrodatafile
      #################
      
      sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --query "select * from t1 where \$CONDITIONS" --split-by c1 --target-dir /user/root/t1 --delete-target-dir --num-mappers 1 --as-avrodatafile 
      ls -l ./*
      
      Output:
      17/02/13 12:16:58 INFO mapreduce.ImportJobBase: Transferred 448 bytes in 25.2757 seconds (17.7245 bytes/sec)
      17/02/13 12:16:58 INFO mapreduce.ImportJobBase: Retrieved 1 records.
      ~~~~~
      -rw-r--r-- 1 root root   527 Feb 13 12:16 ./AutoGeneratedSchema.avsc <==== want option to configure this file name
      -rw-r--r-- 1 root root 12590 Feb 13 12:16 ./QueryResult.java
      

      Attachments

        1. SQOOP-3134.1.patch
          0.9 kB
          Daniel Voros

        Issue Links

          Activity

            People

              dvoros Daniel Voros
              markuskemper@me.com Markus Kemper
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h

                  Slack

                    Issue deployment