[SQOOP-1086] Running multiple incremental sqoop jobs in parallel resets the first sqoop job's --last-value

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Won't Fix
• Affects Version/s: 1.4.0-incubating
• Labels: import incremental job parallel

Environment

Ubuntu 12.04.2

• Assignee: Boglarka Egyed
• Reporter: Byron Foster
• Created: 17/Jun/13 15:31
• Updated: 28/Jul/16 09:37
• Resolved: 27/Jul/16 15:15

Description

I've created 2 jobs (different names) that pull from the same database(MSSQL), but 2 different tables.
They both use incremental append.

If I run the jobs in sequence, I got no issue and the meta store for both jobs remembers the --last-value per job.

If I run the jobs in parallel, when the 1st job finished the meta is updated with the --last-value correctly, but once the 2nd job finished the 1st job's meta --last-value is reset.

First Job

1. create the import job into the incremental table
   $ENV_SQOOP_HOME/bin/sqoop job -D mapred.job.name="Job 1" --create "import-events" – import --connect "$ENV_TRACKING_CONNECTION" --table "$TABLE1" --split-by "dtmDBDateTime" --target-dir "$OUTPUT1" --incremental append --check-column "dtmDBDateTime" --last-value "2012-01-01 00:00:00.000" --fields-terminated-by
   t --null-string '' --null-non-string '';

Second Job

1. create the import job into the table
   $ENV_SQOOP_HOME/bin/sqoop job -D mapred.job.name="Job 2" --create "import-impressions" – import --connect "$ENV_TRACKING_CONNECTION" --table "$TABLE2" --split-by "dtmDBDateTime" --target-dir "$OUTPUT2" --incremental append --check-column "dtmDBDateTime" --last-value "2012-01-01 00:00:00.000" --fields-terminated-by
   t --null-string '' --null-non-string '';

Sub-Tasks

• Update Sqoop1 Docs (warning about parallel jobs and "local" Sqoop1 Metastore): Open