Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-3356

SQL Syntax Error Raised when Double Quotes Added for "check-column"

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Blocker
    • Resolution: Unresolved
    • Affects Version/s: 1.4.6
    • Fix Version/s: None
    • Labels:
      None

      Description

      Goal: Implementing Sqoop Metastore to run daily incremental ETL.

      When executing the following Sqoop job:

      sqoop job -Dmapreduce.job.queuename=D_NO_SLA \
         --create yle0_gim_interaction_fact - import \
         --connect "jdbc:oracle:thin:@//xxxx.xxxx.com:0000/xxxx" \
         --username T0V0_DATALAKE_YLE0 \
         --password-file /dev/xxxxxxx/YLE0_OAT.password 
         --outdir ./sqoop/ \
         --table RYLEINFOMART_2.INTERACTION_FACT \
         --incremental append \
         --check-column 'GREATEST(CREATE_AUDIT_KEY, UPDATE_AUDIT_KEY)' \
         --last-value '53492217' \
         --split-by 'CREATE_AUDIT_KEY' -m 16 \
         --target-dir /dev/T0V0/hivedb/yle0/staging/yle0_gim_interaction_fact_staging/ \
         --fields-terminated-by '\001'

      Sqoop would automatically run the following query to update the max value:

      SELECT MAX("GREATEST(CREATE_AUDIT_KEY, UPDATE_AUDIT_KEY)") FROM "RYLEINFOMART_2"."INTERACTION_FACT"
      

      SQL syntax error was given since there shouldn't be double quotes around GREATEST() function.

      The proposal of using free form query is rejected since metastore cannot be implemented to automatically run the daily incremental job.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              seanmzx Sean Meng
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: