Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-2103

Not able define Decimal(n,p) data type in map-column-hive option

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.5
    • Fix Version/s: 1.4.7
    • Component/s: None
    • Environment:

      Unix OS

      Description

      Not able to define any column as decimal(n,p) while sqooping data and creating hive table using --map-column-hive option

      I tried different syntax (with single quotes,double quotes,backslash..) but the command errors out.

      Below is the command and error:

      time sqoop import -D oraoop.disabled=true --options-file /home/sqoopuser/configfiles/c360_db.config --query "SELECT DLRS.*,'C360' as SOURCE,TO_CHAR(current_timestamp,'YYYY-MM-DD HH24:MI:SS.FF') AS LOAD_TS FROM DEALERS DLRS WHERE \$CONDITIONS" --split-by DEALER_NUMBER --create-hive-table --hive-table "c360dev.dealers_test" --hive-import --map-column-hive DEALER_NUMBER="VARCHAR(20)",DEALER_NAME="VARCHAR(50)",GEO_CODE="VARCHAR(50)",REGIONCODE=INT,REGIONNAME="VARCHAR(30)",DISTRICTCODE="VARCHAR(4)",21_INDICATOR="VARCHAR(1)",ADDRESS_TYPE="VARCHAR(25)",LINEONE="VARCHAR(50)",CITY="VARCHAR(25)",STATE="VARCHAR(2)",ZIPCODE="VARCHAR(5)",EFFECTIVE_DATE=TIMESTAMP,TERMINATION_CODE="VARCHAR(1)",PHONE_NUMBER="VARCHAR(25)",ENTITY_NAME="VARCHAR(50)",DEALERGROUP_NAME="VARCHAR(50)",DEALERGROUP_1COUNT=SMALLINT,DEALERGROUP_2COUNT=SMALLINT,DEALER_CLASSCODE="DECIMAL(22,7)",REGIONAREA_NAME="VARCHAR(50)",REGIONAREA_SHORTNAME="VARCHAR(50)",REGIONAREA_ABBREVATION="VARCHAR(5)",REGIONAREA_AREACCODE=SMALLINT,REGIONAREA_REGIONCODE=SMALLINT,PMACODE=SMALLINT,MARKETCODE=INT,MARKETNAME="VARCHAR(50)",MARKETTYPE="VARCHAR(50)",DUAL_INDICATOR="VARCHAR(5)",DEALERSHIP_TYPE="VARCHAR(5)",CREATION_DATE_TIMESTAMP=TIMESTAMP,CREATED_BY="VARCHAR(20)",LAST_UPDATE_DATE=TIMESTAMP,LAST_UPDATED_BY="VARCHAR(20)",TXM_DEALER="VARCHAR(50)",TFS_ECONTRACTING="VARCHAR(50)",YEARS_OF_PRESS_AWARDS=SMALLINT,IMAGE_USA="VARCHAR(50)",IMAGE_USA_EFFECTIVE_DATE=TIMESTAMP,TERMINATION_DATE=TIMESTAMP,3_DEALERNAME="VARCHAR(50)",IMAGE_USA_COMMENT="VARCHAR(100)",SINGLE_COMBINED_INDICATOR="VARCHAR(10)",1_URLNAME="VARCHAR(25)",1_URLVALUE="VARCHAR(100)",3_URLNAME="VARCHAR(25)",3_URLVALUE="VARCHAR(100)",2_URLNAME="VARCHAR(25)",2_URLVALUE="VARCHAR(100)" --hive-delims-replacement " " --null-string '
      N' --null-non-string '
      N' --target-dir "/user/hive/warehouse/c360dev.db/dealers_test" --verbose > dealers_test.log 2>&1

      real 0m0.876s
      user 0m1.062s
      sys 0m0.112s
      [sqoopuser@*****09 ~]$ more dealers_test.log
      Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
      Please set $ACCUMULO_HOME to the root of your Accumulo installation.
      15/02/12 15:19:26 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5-cdh5.3.1
      15/02/12 15:19:26 DEBUG tool.BaseSqoopTool: Enabled debug logging.
      15/02/12 15:19:26 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
      15/02/12 15:19:26 DEBUG sqoop.Sqoop: 1
      java.lang.ArrayIndexOutOfBoundsException: 1
      at org.apache.sqoop.SqoopOptions.parseColumnMapping(SqoopOptions.java:1200)
      at org.apache.sqoop.SqoopOptions.setMapColumnHive(SqoopOptions.java:1205)
      at org.apache.sqoop.tool.BaseSqoopTool.applyHiveOptions(BaseSqoopTool.java:1112)
      at org.apache.sqoop.tool.ImportTool.applyOptions(ImportTool.java:973)
      at org.apache.sqoop.tool.SqoopTool.parseArguments(SqoopTool.java:435)
      at org.apache.sqoop.Sqoop.run(Sqoop.java:131)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
      at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
      at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
      at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
      at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
      1

        Attachments

        1. SQOOP-2103.patch
          3 kB
          Abraham Elmahrek
        2. SQOOP-2103.patch
          3 kB
          Abraham Elmahrek

          Issue Links

            Activity

              People

              • Assignee:
                abec Abraham Elmahrek
                Reporter:
                AnithaKS Anitha
              • Votes:
                4 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: